tx: detect cycles in forEachPageInternal#1194
Conversation
forEachPageInternal walks a branch page's children without tracking pages it has already entered, so a corrupted db whose branch references one of its own ancestors drives the goroutine stack to overflow. Thread a visited set through the recursion and bail on a revisit; fn still fires on the back- edge so verifyPageReachable's "multiple references" diagnostic is emitted once per cycle. Related to etcd-io#701 / etcd-io#581. Signed-off-by: Iván Salazar <ivangio.salazar@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: ivangsm The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
recursivelyCheckPageKeyOrderInternal walks branch children without tracking pages it has already entered, so a corrupted db whose branch references one of its own ancestors drives the goroutine stack to overflow even after forEachPageInternal (etcd-io#1194) stops looping. Thread a visited set through the recursion, emit a single "page cycle detected" error on revisit, and return so the outer Check continues to surface other diagnostics. Related to etcd-io#701 / etcd-io#581. Signed-off-by: Iván Salazar <ivangio.salazar@gmail.com>
| // cannot drive this recursion to stack overflow. fn still runs above, | ||
| // so verifyPageReachable's "multiple references" diagnostic fires. | ||
| if _, ok := visited[pgid]; ok { | ||
| return |
There was a problem hiding this comment.
the question is what we're going to do here, reading from a corrupted db should definitely be surfaced.
I (personally) would rather side with a panic than changing the function interface to an error.
There was a problem hiding this comment.
Good push — I audited the forEachPage callers and you're right that the soft-return is a blind spot for one of them:
tx_check.go:148(Check): the fn isverifyPageReachable, which already emitspage N: multiple referencesto the error channel on duplicate pgids (tx_check.go:163-165), andtx.checkwraps its body indefer recover()that converts panics intopanicked{}errors on the same channel (tx_check.go:39-43). Surfacing is fine here under either strategy.bucket.go:621(Bucket.Stats): the fn just accumulates counters and recursively re-enters sub-buckets viasubStats.Add(b.openBucket(...).Stats()). On a revisit it silently inflatesKeyN/LeafPageN/BranchPageNand double-enters sub-buckets. No recover on this path. With the current soft-return,bbolt statson a cycle-corrupted db prints wrong numbers with no indication — exactly the silent behavior you're flagging.
A few ways forward, in increasing cost:
- Panic on cycle (your suggestion). One line in
forEachPageInternal.Checkstays clean via the existing recover;Stats/bbolt statscrash loudly with the offending pgid. Simplest, matches your preference. - Panic +
db.Logger().Errorffirst. Same as (1), plus a log entry via the existingLoggerinterface so the cycle is recorded even if some caller recovers the panic. Marginal, but ~free. - Add
Corrupted booltoBucketStats(additive, non-breaking).forEachPagewould route the cycle through a sink thatStatsreads, so the CLI could print a prominent warning alongside the (inflated) numbers instead of crashing. Strictly the most graceful UX, but it's more scope than the bug this PR is fixing — probably wants its own issue/PR.
My default would be (1), possibly (2). Happy to go straight to (3) in a follow-up if you'd rather keep this PR tight. Which direction do you want?
Summary
tx.forEachPageInternalwalks a branch page's children without tracking pages it has already entered, so a corrupted db whose branch references one of its own ancestors drives the goroutine stack to overflow. This follows up on the same class of bug that #1193 addresses in the surgeonXRaywalker and that @tjungblu called out as a follow-up in #701.This change threads a
visitedset throughforEachPageInternaland bails on a revisit.fnstill fires on the back-edge before we bail, soverifyPageReachable's "multiple references" diagnostic is emitted once per cycle — the walker just stops descending instead of recursing forever.Related to #701 / #581. Two other recursive walkers in the check path (
recursivelyCheckPageKeyOrderInternaland the cursor used byrecursivelyCheckBucket) still loop on corrupted pages; they can be addressed as separate follow-ups.Test plan
TestTx_forEachPage_CycleTerminatesthat usessurgeon.CopyPageto rewrite a leaf with its branch ancestor's content (so the leaf's child list references itself), then callstx.forEachPageand asserts the walk is bounded.main(stack overflow after ~4M recursive frames).TEST_FREELIST_TYPE=arrayandTEST_FREELIST_TYPE=hashmapon./+./internal/tests/...+./internal/surgeon/...(Stats / Check / RecursivelyCheckPages / new cycle test).