Skip to content

Firestore emulator sends GOAWAY too_many_pings to the iOS SDK after ~90 s (keepalive enforcement stricter than production) #11238

Open

Description

@KsiazeKochel

[REQUIRED] Environment info

firebase-tools: 15.31.0 (cloud-firestore-emulator-v1.22.0.jar, OpenJDK 21.0.12)

Platform: macOS 27 on Apple silicon; Xcode 27.0; iOS Simulator 27.0 and 18.6; Firebase Apple SDK 12.19.2 (Swift Package Manager, grpc-binary 1.69.1)

[REQUIRED] Test case

An iOS app connected to the Firestore emulator with Firestore.firestore().useEmulator(withHost: "127.0.0.1", port: <port>), one snapshot listener open, idle for about 75 seconds, then a single setData on one document.

[REQUIRED] Steps to reproduce

  1. firebase emulators:start --only firestore
  2. Run an iOS app on the simulator that connects to the emulator and opens one snapshot listener.
  3. Leave the app idle for about 75 seconds.
  4. Write one document and measure the time until the completion handler fires.

[REQUIRED] Expected behavior

The write completes within about a second, as it does against production Firestore.

[REQUIRED] Actual behavior

About 90 seconds after the app opens its connection, the emulator closes the HTTP/2 connection with GOAWAY (ENHANCE_YOUR_CALM, debug data too_many_pings). The SDK reconnects only about 25 seconds later, so a write issued in that window completes 25 to 30 seconds late. Tests that expect a write to finish within 30 seconds fail once the app has been connected for ~90 s.

Evidence:

  • HTTP/2 frame log of the emulator (java.util.logging at FINEST for io.grpc.netty.NettyServerHandler):
    OUTBOUND GO_AWAY: lastStreamId=5 errorCode=11 length=14 bytes=746f6f5f6d616e795f70696e6773 (hex for too_many_pings), followed by OUTBOUND RST_STREAM for stream 3 (listener) and stream 5 (write).
  • lsof every 3 s: the app's socket to the emulator is ESTABLISHED for 94 s, then CLOSE_WAIT for about 25 s, then CLOSED.
  • sample of the app process during the stall: the Firestore RPC queue is idle in Datastore::PollGrpcQueue / poll. jstack of the emulator: Netty and gRPC threads idle, no lock held. Nothing is blocked on either side.
  • Plain idle followed by one write gives the same result, so it is not caused by work in the app.

Likely cause: the Apple SDK sets GRPC_ARG_KEEPALIVE_TIME_MS to 30 s (Firestore/core/src/remote/grpc_connection.cc, around line 307 in 12.19.2). The emulator's gRPC server appears to use the grpc-java KeepAliveEnforcer defaults (permitKeepAliveTime 5 minutes, two ping strikes), so the third client ping at ~90 s triggers too_many_pings. Production Firestore accepts the SDK's 30-second keepalive, so the emulator behaves differently from production. firebase emulators:start --help and the emulator jar's options show no setting to change this.

Suggested fix: configure the emulator's Netty server to accept the keepalive interval used by the official SDKs, for example NettyServerBuilder.permitKeepAliveTime(10, TimeUnit.SECONDS) (and permitKeepAliveWithoutCalls(true) if the SDKs ping without active calls), or expose it as an emulator option.

Workaround we use: emulator tests wait up to 120 s for a write to be acknowledged, and the app treats a write that exceeds its own timeout as "result unknown" instead of failed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ Proxy