Skip to content

Data Connect emulator: PGlite server never sets TCP_NODELAY, so each query costs ~100 ms on Linux (Nagle + delayed ACK) #11402

Open

Description

@bklynjones

[REQUIRED] Environment info

firebase-tools: 15.32.1 and 15.34.2 (src/emulator/dataconnect/pgliteServer.ts is unchanged between them); Data Connect emulator 3.4.21, PGlite 0.3.3 through pg-gateway 0.3.0-beta.4

Platform: Linux (GitHub Actions ubuntu-latest; any Linux with the kernel's default delayed-ACK behaviour). Not reproducible on macOS.

[REQUIRED] Test case

Any Data Connect project started with firebase emulators:start --only dataconnect on Linux. The emulator's built-in PGlite server (createPGServer in pgliteServer.ts) hands each socket from net.createServer to fromNodeSocket, which wraps it in web streams for pg-gateway's PostgresConnection. Nothing calls socket.setNoDelay(true), so Nagle's algorithm stays on for the server side of every connection.

[REQUIRED] Steps to reproduce

  1. On Linux, start the Data Connect emulator and note the Postgres port it prints.
  2. Run time psql "postgres://127.0.0.1:<postgresPort>/postgres" -c 'select 1' a few times: about 100 ms each.
  3. Make the kernel acknowledge loopback segments at once, which removes the delay Nagle waits on:
    sudo ip route replace local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 quickack 1
    sudo ip route replace local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 quickack 1
    
  4. Repeat step 2: a few milliseconds each.

[REQUIRED] Expected behavior

A query against the emulator's PGlite costs a few milliseconds on Linux, as it does on macOS.

[REQUIRED] Actual behavior

Each call that reaches PGlite costs 86 ms (a read) to 127 ms (a write) on Linux, against about 2 ms on macOS for the same emulator and schema. select 1 sent straight at the PGlite port with psql costs 116 ms; a request the emulator answers without touching Postgres costs 0.9 ms, so the time is between the emulator and PGlite.

The mechanism: PostgresConnection writes a reply as several small segments (row description, data row, command complete, ready for query). With Nagle on, the second segment waits for the ACK of the first, and Linux delays that ACK by up to 40 ms, so each reply waits through two or three delayed ACKs. macOS acknowledges differently, which is why it never shows there.

Measured on our suite of 6,909 calls through the emulator: 1,055 s on ubuntu-latest, 28 s on an M-series Mac. With the quickack routes above set before the emulator starts, the same suite takes 78 s on the runner (a read 3.4 ms, a write 4.7 ms), and a second suite fell from 155 s to 28 s. Per-layer timings are in BEJ-OPS-LLC/a440-mm#493.

Suggested fix: one line in fromNodeSocket in src/emulator/dataconnect/pgliteServer.ts, before the streams are created:

socket.setNoDelay(true);

pg-gateway's own Node adapter could do the same, but the emulator owns the net.Socket here, so this is the smallest change, and it would speed up every Linux developer machine and CI runner that uses the Data Connect emulator.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ Proxy