Skip to content

fix(bigtable): scope session pool maintenance loops to the pool, not the client - #20444

Merged
mutianf merged 2 commits into
googleapis:mainfrom
mutianf:bigtable-session-pool-maint-loop-leak
Sep 1, 2026
Merged

mutianf merged 2 commits into
googleapis:mainfrom
mutianf:bigtable-session-pool-maint-loop-leak

Conversation

@mutianf

@mutianf mutianf commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Problem

SessionPoolImpl.Start(ctx) launched three background maintenance loops — AFE prune, WaitServerClose sweep, slow metrics — that selected on the ctx handed in by the caller. The only caller passed the session Client's process-lifetime BackgroundCtx, while Close() cancels the pool's own poolCtx, a separate tree rooted at context.Background(). Cancellation only flows along derivation edges, so poolCancel() had nothing to say to those loops.

This survived review because on the client-shutdown path the loops do die — via the client's ctx, not the pool's. session.TableCache TTL eviction has no such backstop: it calls pool.Close() and nothing else. Every evicted pool leaked three goroutines, and since each loop closure captures p, each leaked goroutine pinned an entire SessionPoolImpl (metrics ring buffers, sessionList) on the heap. With a read and a write pool per table that's six goroutines and two pinned pools per evicted table, growing without bound over a long-lived process.

Affects the accelerator daemon and the native Go client alike, since both share this code.

Fix

  • Start() takes no ctx. The loops hang off a new pool-owned maintCtx (child of poolCtx).
  • Close() cancels maintCtx in Phase 1, ahead of Phase 4's poolCancel, so no prune/sweep/metrics tick fires against a pool that's mid-teardown. poolCtx can't be reused for this: Phase 2 derives the 30s session-close RPC context from it, so it has to stay alive through teardown while the loops must stop immediately.
  • The three near-identical loop bodies collapse into one startMaintenanceLoop helper.

Close cancels the loops but deliberately does not join them. They park on a select, so cancel releases the pool immediately; joining would only close a microsecond-wide window where an already-running tick body finishes after Close returns. Not worth blocking for, because Close runs inline on the TableCache sweeper (sweepOnce closes evicted handles serially) — a wedged tick body would stall eviction for every table, which is strictly worse than the leak being fixed.

Verification

A/B reproduction against pre- and post-fix builds, counting live maintenance frames via runtime.Stack:

Unit, driving the real eviction chain (TableCache.sweepOnce → TableHandle.Close → sessionTable.Close → releaseSessionPool → SessionPoolImpl.Close), 6 tables × 2 pools × 3 loops × 4 cycles:

after-evict c0 c1 c2 c3
pre-fix +36 +72 +108 +144
post-fix 0 0 0 0

Worth noting: pre-fix, livePools still drained to 0 — eviction did reach releaseSessionPool/pool.Close() and the goroutines survived it anyway. That isolates the bug to ctx wiring.

Accelerator daemon against a live backend, 5-minute cache TTL, 3 cycles, frames counted via pprof: pre-fix 24 → 48 → 72, never reclaiming; post-fix 24 → 0 every cycle.

Cost per leaked pool is ~14 KB (three 4 KB goroutine stacks + ~10 KB pinned pool), so this presents as unbounded goroutine growth rather than visibly rising RSS — RSS was noise-dominated in the daemon run.

Regression tests added in session_pool_lifecycle_test.go: TestPoolClose_UnwindsMaintenanceLoops (3 Start/Close cycles, frame-counted so it's immune to unrelated goroutine noise, and asserts the count rises after Start so it can't pass vacuously) and TestPoolStart_AfterClose_LoopsExitImmediately.

gofmt, go vet, and the full internal/transport + internal/session suites pass under -race.

…the client

SessionPoolImpl.Start(ctx) launched three background maintenance loops
(AFE prune, WaitServerClose sweep, slow metrics) that selected on the ctx
handed in by the caller. The only caller passed the session Client's
process-lifetime BackgroundCtx, while Close() cancels the pool's own
poolCtx -- a separate tree rooted at context.Background(). Nothing ever
cancelled the loops on pool teardown.

That went unnoticed because on client shutdown the loops do die, just via
the client's ctx rather than the pool's. session.TableCache TTL eviction
has no such backstop: it calls pool.Close() only. Every evicted pool
therefore leaked three goroutines, and because each loop closure captures
p, each leaked goroutine pinned an entire SessionPoolImpl (metrics ring
buffers, sessionList) on the heap. With a read and a write pool per table
that is six goroutines and two pinned pools per evicted table, growing
without bound over a long-lived process.

Fix: Start() takes no ctx. The loops hang off a new pool-owned maintCtx
(child of poolCtx) that Close cancels in Phase 1, ahead of Phase 4's
poolCancel, so no prune/sweep/metrics tick fires against a pool that is
mid-teardown. The three near-identical loop bodies collapse into one
startMaintenanceLoop helper.

Close cancels the loops but deliberately does not join them: they park on
a select, so cancel releases the pool immediately, and Close runs inline
on the TableCache sweeper (sweepOnce closes evicted handles serially), so
a wedged tick body must not be able to stall eviction for every table.

Verified by A/B reproduction against pre- and post-fix builds, counting
live maintenance frames via runtime.Stack:

  - unit, driving the real TableCache eviction chain (6 tables x 2 pools
    x 3 loops x 4 cycles): pre-fix +36/+72/+108/+144 monotonic; post-fix
    0 after every cycle. Notably livePools still drained to 0 pre-fix,
    i.e. eviction did reach releaseSessionPool/pool.Close and the
    goroutines survived it anyway.
  - accelerator daemon against a live backend with a 5 minute cache TTL:
    pre-fix 24 -> 48 -> 72 never reclaiming; post-fix 24 -> 0 each cycle.

Cost per leaked pool is ~14 KB (three 4 KB goroutine stacks plus ~10 KB
of pinned pool), so this shows up as unbounded goroutine growth rather
than visibly rising RSS.

Change-Id: I03569860b9f9a26e16ca9d50665ddccc4b79423c
@mutianf
mutianf requested review from a team as code owners September 1, 2026 03:17
@product-auto-label product-auto-label Bot added the api: bigtable Issues related to the Bigtable API. label Sep 1, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request fixes a goroutine leak in the session pool by decoupling background maintenance loops from the client-scoped background context. A new pool-scoped maintCtx is introduced to manage these loops, ensuring they are cancelled during pool teardown. The Start method has been updated to no longer accept an external context, and regression tests have been added to verify proper unwinding of the loops. Feedback on the changes includes correcting an inaccurate comment in the new test file which incorrectly states that Close waits on a non-existent p.maint field.

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
@mutianf
mutianf merged commit 7457e6e into googleapis:main Sep 1, 2026
20 of 21 checks passed
@mutianf
mutianf deleted the bigtable-session-pool-maint-loop-leak branch September 1, 2026 16:31
haphungw pushed a commit that referenced this pull request Sep 2, 2026
🤖 I have created a release *beep* *boop*
---


##
[1.54.0](bigtable/v1.53.0...bigtable/v1.54.0)
(2026-09-02)


### Features

* **bigtable:** Serve grpc.health.v1.Health from the accelerator daemon
([#20446](#20446))
([5fba38a](5fba38a))
* Update API sources and regenerate
([#20459](#20459))
([5426e32](5426e32))


### Bug Fixes

* **bigtable:** Measure real scheduling delay in pacemaker_delays
([#20445](#20445))
([e1304d6](e1304d6))
* **bigtable:** Scope session pool maintenance loops to the pool, not
the client
([#20444](#20444))
([7457e6e](7457e6e))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: bigtable Issues related to the Bigtable API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

⌂ Proxy