Repository navigation
fix(stovepipe): key build-slot ownership by request ID - #966
Draft
sbalabanov wants to merge 1 commit into
Draft
sbalabanov wants to merge 1 commit into
sbalabanov wants to merge 1 commit into
Conversation
sbalabanov
force-pushed
the
sergeyb/sp-oct-003-owner-keyed-slots
branch
2 times, most recently
from
October 27, 2026 23:39
c404a3a to
beeea87
Compare
sbalabanov
marked this pull request as ready for review
October 27, 2026 23:41
sbalabanov
force-pushed
the
sergeyb/sp-oct-003-owner-keyed-slots
branch
from
October 28, 2026 17:20
beeea87 to
285089f
Compare
Track build slots in in_flight_request_ids, the set of requests holding a slot, instead of the queue's anonymous in_flight_count. The set lives in a new column next to in_flight_count, which is retained unused for schema compatibility. A redelivery that finds its own claim resumes the admit instead of holding behind it, or, when a newer head arrived, releases the slot before being superseded, so a crash between the claim and the processing write no longer wedges the queue. Release removes by ID and is a no-op when absent, so a retried outcome write cannot free another request's slot, and the DLQ releases for any nonterminal request. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
sbalabanov
force-pushed
the
sergeyb/sp-oct-003-owner-keyed-slots
branch
from
October 28, 2026 17:27
285089f to
5718039
Compare
sbalabanov
marked this pull request as draft
October 28, 2026 18:24
sbalabanov
added a commit
that referenced
this pull request
Oct 30, 2026
## Why? `process` admits the latest head with two ordered CAS writes on two rows: it increments the anonymous `queue.in_flight_count` first, then moves the request accepted→processing. A crash between them leaves the count raised with no owner. Holds never consume retry budget and the DLQ reconciler releases only `processing` requests, so the queue wedges permanently. 1. Queue at `max_concurrent = 1`; request R is accepted and is the latest head. 2. `process` claims the slot (`in_flight_count = 1`) and crashes before writing `processing`. 3. Every redelivery re-reads `accepted`, finds the gate closed by its own claim, and holds. 4. Nothing admits again, and the best-effort unwind releases never ran because the process crashed. ## What? - `admitLatestHead` gates and derives the strategy as before, then CASes the request accepted→processing first and increments `in_flight_count` second. A version conflict on the increment reloads and retries without re-gating or coalescing, because the admit is already durable. A lost request CAS acks without touching the counter. - Removes `process`'s `releaseBuildSlot` helper and both unwind call sites. - Trade-off: the count now drifts low rather than high. A crash or hard failure between the two writes, or two heads that both pass the gate in the same window, over-admits by one. The existing zero clamp in the `buildsignal` and DLQ release paths brings the count back to the true value as the queue drains. The count can go high only if a delivery stalls between the two writes past the visibility timeout and wakes after its redelivery has finished the validation; the RFC records this residual and how to narrow or close it. - `doc/rfc/stovepipe/steps/process.md`: step 7 reordered; the redelivery and `in_flight_count integrity` sections now describe the fail-open order. Two `dlq.go`/`buildsignal.go` doc comments that described the old claim-first order are corrected (comments only). - This change does not repair counts that leaked before it, so a queue that is already wedged needs a one-time manual reset of `in_flight_count`. Alternative to #966, which keys slots by request ID in a new `in_flight_request_ids` column and is the exact fix; this PR needs no schema change. Merge one or the other, not both. ## Test Plan - `make fmt && make lint && make check-tidy && make check-gazelle && make check-mocks` - `./tool/bazel test //stovepipe/... //service/stovepipe/... --test_output=errors` - `./tool/bazel build //test/e2e/stovepipe:go_default_test //test/integration/stovepipe/...` Audit: SP-OCT-003 and SP-OCT-005 in doc/audit/stovepipe-fable-v2.md (Sol v2 #1/#2, Opus v2 #1/#8). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: sergeyb <sergeyb@uber.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why?
Admission claims a build slot by incrementing an anonymous
in_flight_countand then moves the request accepted→processing in a second write. If the second write never lands, the count stays raised and the redelivery is held behind its own claim. Holds never dead-letter, and the DLQ releases onlyprocessingrequests, so nothing repairs it. Release has the same problem in reverse: after a failed outcome write, the retry releases a second time and takes another request's slot.max_concurrent = 1; request R is accepted and becomes the latest head.processclaims the slot (in_flight_count = 1), then crashes before writingprocessing.What?
Queue.InFlightCountbecomesInFlightRequestIDs, the set of requests holding a slot, stored in a new MySQL columnin_flight_request_idsadded next toin_flight_count, which is retained unused for schema compatibility. A claim is idempotent per request ID. A redelivery that finds its own ID resumes the processing write. If a newer head arrived meanwhile, it releases the slot before being superseded, so no slot is spent on a stale head and no sweeper is needed. Release removes by ID and writes nothing when the ID is absent. That makes the buildsignal and DLQ retries safe and lets the DLQ release for any nonterminal request. A claim left by a failed admit stays for the redelivery to resume or the DLQ to release.processreleases it only when it loses the admit to a terminal transition; a duplicate admission keeps the slot. The process RFC, related step docs and the storage README (with an upgrade note) describe the owner set.Test Plan
make fmt && make lint && make check-tidy && make check-gazelle && make check-mocks./tool/bazel test //stovepipe/... //service/stovepipe/... //test/integration/stovepipe/extension/storage/... --test_output=errorsAudit: SP-OCT-003 and SP-OCT-005 in doc/audit/stovepipe-fable-v2.md (Sol v2 #1/#2, Opus v2 #1/#8).
🤖 Generated with Claude Code