Repository navigation
fix(bigtable): scope session pool maintenance loops to the pool, not the client - #20444
Merged
mutianf merged 2 commits intoSep 1, 2026
Merged
Conversation
…the client
SessionPoolImpl.Start(ctx) launched three background maintenance loops
(AFE prune, WaitServerClose sweep, slow metrics) that selected on the ctx
handed in by the caller. The only caller passed the session Client's
process-lifetime BackgroundCtx, while Close() cancels the pool's own
poolCtx -- a separate tree rooted at context.Background(). Nothing ever
cancelled the loops on pool teardown.
That went unnoticed because on client shutdown the loops do die, just via
the client's ctx rather than the pool's. session.TableCache TTL eviction
has no such backstop: it calls pool.Close() only. Every evicted pool
therefore leaked three goroutines, and because each loop closure captures
p, each leaked goroutine pinned an entire SessionPoolImpl (metrics ring
buffers, sessionList) on the heap. With a read and a write pool per table
that is six goroutines and two pinned pools per evicted table, growing
without bound over a long-lived process.
Fix: Start() takes no ctx. The loops hang off a new pool-owned maintCtx
(child of poolCtx) that Close cancels in Phase 1, ahead of Phase 4's
poolCancel, so no prune/sweep/metrics tick fires against a pool that is
mid-teardown. The three near-identical loop bodies collapse into one
startMaintenanceLoop helper.
Close cancels the loops but deliberately does not join them: they park on
a select, so cancel releases the pool immediately, and Close runs inline
on the TableCache sweeper (sweepOnce closes evicted handles serially), so
a wedged tick body must not be able to stall eviction for every table.
Verified by A/B reproduction against pre- and post-fix builds, counting
live maintenance frames via runtime.Stack:
- unit, driving the real TableCache eviction chain (6 tables x 2 pools
x 3 loops x 4 cycles): pre-fix +36/+72/+108/+144 monotonic; post-fix
0 after every cycle. Notably livePools still drained to 0 pre-fix,
i.e. eviction did reach releaseSessionPool/pool.Close and the
goroutines survived it anyway.
- accelerator daemon against a live backend with a 5 minute cache TTL:
pre-fix 24 -> 48 -> 72 never reclaiming; post-fix 24 -> 0 each cycle.
Cost per leaked pool is ~14 KB (three 4 KB goroutine stacks plus ~10 KB
of pinned pool), so this shows up as unbounded goroutine growth rather
than visibly rising RSS.
Change-Id: I03569860b9f9a26e16ca9d50665ddccc4b79423c
Contributor
There was a problem hiding this comment.
Code Review
This pull request fixes a goroutine leak in the session pool by decoupling background maintenance loops from the client-scoped background context. A new pool-scoped maintCtx is introduced to manage these loops, ensuring they are cancelled during pool teardown. The Start method has been updated to no longer accept an external context, and regression tests have been added to verify proper unwinding of the loops. Feedback on the changes includes correcting an inaccurate comment in the new test file which incorrectly states that Close waits on a non-existent p.maint field.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
sushanb
approved these changes
Sep 1, 2026
haphungw
pushed a commit
that referenced
this pull request
Sep 2, 2026
🤖 I have created a release *beep* *boop* --- ## [1.54.0](bigtable/v1.53.0...bigtable/v1.54.0) (2026-09-02) ### Features * **bigtable:** Serve grpc.health.v1.Health from the accelerator daemon ([#20446](#20446)) ([5fba38a](5fba38a)) * Update API sources and regenerate ([#20459](#20459)) ([5426e32](5426e32)) ### Bug Fixes * **bigtable:** Measure real scheduling delay in pacemaker_delays ([#20445](#20445)) ([e1304d6](e1304d6)) * **bigtable:** Scope session pool maintenance loops to the pool, not the client ([#20444](#20444)) ([7457e6e](7457e6e)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
SessionPoolImpl.Start(ctx)launched three background maintenance loops — AFE prune,WaitServerClosesweep, slow metrics — that selected on thectxhanded in by the caller. The only caller passed the sessionClient's process-lifetimeBackgroundCtx, whileClose()cancels the pool's ownpoolCtx, a separate tree rooted atcontext.Background(). Cancellation only flows along derivation edges, sopoolCancel()had nothing to say to those loops.This survived review because on the client-shutdown path the loops do die — via the client's ctx, not the pool's.
session.TableCacheTTL eviction has no such backstop: it callspool.Close()and nothing else. Every evicted pool leaked three goroutines, and since each loop closure capturesp, each leaked goroutine pinned an entireSessionPoolImpl(metrics ring buffers,sessionList) on the heap. With a read and a write pool per table that's six goroutines and two pinned pools per evicted table, growing without bound over a long-lived process.Affects the accelerator daemon and the native Go client alike, since both share this code.
Fix
Start()takes no ctx. The loops hang off a new pool-ownedmaintCtx(child ofpoolCtx).Close()cancelsmaintCtxin Phase 1, ahead of Phase 4'spoolCancel, so no prune/sweep/metrics tick fires against a pool that's mid-teardown.poolCtxcan't be reused for this: Phase 2 derives the 30s session-close RPC context from it, so it has to stay alive through teardown while the loops must stop immediately.startMaintenanceLoophelper.Closecancels the loops but deliberately does not join them. They park on a select, so cancel releases the pool immediately; joining would only close a microsecond-wide window where an already-running tick body finishes afterClosereturns. Not worth blocking for, becauseCloseruns inline on theTableCachesweeper (sweepOncecloses evicted handles serially) — a wedged tick body would stall eviction for every table, which is strictly worse than the leak being fixed.Verification
A/B reproduction against pre- and post-fix builds, counting live maintenance frames via
runtime.Stack:Unit, driving the real eviction chain (
TableCache.sweepOnce → TableHandle.Close → sessionTable.Close → releaseSessionPool → SessionPoolImpl.Close), 6 tables × 2 pools × 3 loops × 4 cycles:Worth noting: pre-fix,
livePoolsstill drained to 0 — eviction did reachreleaseSessionPool/pool.Close()and the goroutines survived it anyway. That isolates the bug to ctx wiring.Accelerator daemon against a live backend, 5-minute cache TTL, 3 cycles, frames counted via pprof: pre-fix
24 → 48 → 72, never reclaiming; post-fix24 → 0every cycle.Cost per leaked pool is ~14 KB (three 4 KB goroutine stacks + ~10 KB pinned pool), so this presents as unbounded goroutine growth rather than visibly rising RSS — RSS was noise-dominated in the daemon run.
Regression tests added in
session_pool_lifecycle_test.go:TestPoolClose_UnwindsMaintenanceLoops(3 Start/Close cycles, frame-counted so it's immune to unrelated goroutine noise, and asserts the count rises after Start so it can't pass vacuously) andTestPoolStart_AfterClose_LoopsExitImmediately.gofmt,go vet, and the fullinternal/transport+internal/sessionsuites pass under-race.