backport: assumeutxo M4 — evo snapshot format v3 and LLMQ reconstruction - #53
Open
PastaPastaPasta wants to merge 29 commits into
Open
backport: assumeutxo M4 — evo snapshot format v3 and LLMQ reconstruction#53PastaPastaPasta wants to merge 29 commits into
PastaPastaPasta wants to merge 29 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
4 tasks
PastaPastaPasta
force-pushed
the
assumeutxo/m3-background-completion
branch
from
July 12, 2026 14:21
32d5966 to
4f9d716
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m4-evo-snapshot
branch
from
July 12, 2026 14:29
52a0371 to
ed610de
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m3-background-completion
branch
from
July 12, 2026 16:00
4f9d716 to
f08bce5
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m4-evo-snapshot
branch
from
July 12, 2026 16:01
ed610de to
609b604
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m3-background-completion
branch
from
July 13, 2026 22:45
f08bce5 to
a6e49d6
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m4-evo-snapshot
branch
from
July 13, 2026 22:45
609b604 to
11d1fb7
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m3-background-completion
branch
from
July 14, 2026 05:10
a6e49d6 to
eb757a0
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m4-evo-snapshot
branch
2 times, most recently
from
July 14, 2026 17:01
022d3e6 to
590e34b
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m3-background-completion
branch
from
July 14, 2026 17:01
eb757a0 to
2bc543c
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m4-evo-snapshot
branch
from
July 20, 2026 16:59
590e34b to
9e0f8d7
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m3-background-completion
branch
from
July 20, 2026 16:59
2bc543c to
5b3d0d7
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m4-evo-snapshot
branch
2 times, most recently
from
August 1, 2026 19:55
e0377b0 to
9baa44d
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m3-background-completion
branch
from
August 1, 2026 22:04
569913c to
05723b8
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m4-evo-snapshot
branch
from
August 1, 2026 22:04
9baa44d to
ab27f8b
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m3-background-completion
branch
from
August 1, 2026 22:27
05723b8 to
667611e
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m4-evo-snapshot
branch
from
August 1, 2026 22:27
ab27f8b to
2a968e5
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m3-background-completion
branch
from
August 1, 2026 22:44
667611e to
328d224
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m4-evo-snapshot
branch
2 times, most recently
from
August 1, 2026 23:09
ea58a7f to
ed41e4e
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m3-background-completion
branch
from
August 1, 2026 23:35
328d224 to
f9f8fa6
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m4-evo-snapshot
branch
4 times, most recently
from
August 2, 2026 03:10
336b093 to
269d77e
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m3-background-completion
branch
from
August 2, 2026 18:42
f9f8fa6 to
63de2c1
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m4-evo-snapshot
branch
from
August 2, 2026 18:42
269d77e to
02ced9e
Compare
Disable DKG participation and quorum signing until snapshot background validation completes. Enforce the refusal at CreateSigShare, the actual share-production boundary, so direct RPC, async, and queued signing paths cannot bypass it. The quorum sign RPC now returns a clear JSON-RPC error for both submit modes, and masternode status exposes the disabled participation state. Add unit coverage for the shared production-gate predicate across snapshot activation.
A WriteDerived failure means independently derived block data disagrees with the copy already recorded in EvoDB. That is local state corruption (or a cross-chainstate divergence bug), never evidence about the block being processed. Previously the mismatch surfaced as BLOCK_CONSENSUS: the block was persistently marked BLOCK_FAILED_VALID (surviving restart and forking the node off the network) and the relaying peer was handed a 100-point misbehavior score via BlockChecked, which background validation also triggers. Instead, follow the existing EvoDbInconsistencyMessage convention: request node shutdown via AbortNode and fail validation with M_ERROR, which neither marks the block invalid nor punishes peers. The credit-pool and MNHF sites abort at the throw site because miner and RPC callers never pass through a validation-state catch; a typed EvoDbInconsistencyError lets the four block-path catch blocks that would otherwise swallow it into BLOCK_CONSENSUS reclassify it as M_ERROR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ethrow CMNHFManager::GetForBlock re-throws internal ProcessBlock failures as a plain runtime_error, which would let a downstream generic catch misreport an EvoDB mismatch (M_ERROR) as a consensus failure. Re-throw typed when the validation state carries M_ERROR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reads and writes outside a BeginTransaction scope previously always bound to the NORMAL identity. While a snapshot chainstate is active, transaction-less consumers (RPC, mempool, miner, P2P serving) could not see snapshot-chain records still pending in the SNAPSHOT root overlay, and transaction-less writes dirtied the wrong overlay. CEvoDB now tracks a default identity which snapshot activation (ActivateSnapshot, ActivateExistingSnapshot) sets to SNAPSHOT and ResetChainstates resets; snapshot completion must reset it to NORMAL when marker promotion lands. Also make the GetListForBlockInternal fallback loud: a missing list diff for a DIP3-active block is pending-elsewhere or corrupt data, never the pre-DIP3 genesis of the masternode list, so throw instead of silently caching an empty list and clobbering m_initial_snapshot_index. The thrown message carries the IsBlockDataUnavailableError sentinel and BuildSimplifiedMNListDiff converts it into a serve failure, so peers requesting such history are not penalized. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GetListForBlock can now throw when list data is unavailable (missing diff for a DIP3-active block). CActiveMasternodeManager::UpdatedBlockTip and GovernanceSigner's trigger creation run on the scheduler thread, where an uncaught exception terminates the node; catch it there and skip the update instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hen renaming chainstates
Discard Dash snapshot lifecycle markers only after the invalid snapshot directory rename succeeds. If the rename fails, preserving the markers keeps the existing restart recovery state recognizable while the upstream rename error is propagated to the fatal shutdown message.
Keep Dash’s mock shutdown callback while asserting the expected fatal diagnostic. The default callback reaches StartShutdown(), whose unit-test guard aborts the process before this Dash test can complete.
a733dd7 Remove unused function `reliesOnAssumedValid` (Suhas Daftuar) d4a11ab Cache block index entry corresponding to assumeutxo snapshot base blockhash (Suhas Daftuar) 3556b85 Move CheckBlockIndex() from Chainstate to ChainstateManager (Suhas Daftuar) 0ce805b Documentation improvements for assumeutxo (Ryan Ofsky) 768690b Fix initialization of setBlockIndexCandidates when working with multiple chainstates (Suhas Daftuar) d43a1f1 Tighten requirements for adding elements to setBlockIndexCandidates (Suhas Daftuar) d0d40ea Move block-storage-related logic to ChainstateManager (Suhas Daftuar) 3cfc753 test: Clear block index flags when testing snapshots (Suhas Daftuar) 272fbc3 Update CheckBlockIndex invariants for chains based on an assumeutxo snapshot (Suhas Daftuar) 10c0571 Add wrapper for adding entries to a chainstate's block index candidates (Suhas Daftuar) 471da5f Move block-arrival information / preciousblock counters to ChainstateManager (Suhas Daftuar) 1cfc887 Remove CChain dependency in node/blockstorage (Suhas Daftuar) fe86a7c Explicitly track maximum block height stored in undo files (Suhas Daftuar) Pull request description: This PR proposes a clean up of the relationship between block storage and the chainstate objects, by moving the decision of whether to store a block on disk to something that is not chainstate-specific. Philosophically, the decision of whether to store a block on disk is related to validation rules that do not require any UTXO state; for anti-DoS reasons we were using some chainstate-specific heuristics, and those have been reworked here to achieve the proposed separation. This PR also fixes a bug in how a chainstate's `setBlockIndexCandidates` was being initialized; it should always have all the HAVE_DATA block index entries that have more work than the chain tip. During startup, we were not fully populating `setBlockIndexCandidates` in some scenarios involving multiple chainstates. Further, this PR establishes a concept that whenever we have 2 chainstates, that we always know the snapshotted chain's base block and the base block's hash must be an element of our block index. Given that, we can establish a new invariant that the background validation chainstate only needs to consider blocks leading to that snapshotted block entry as potential candidates for its tip. As a followup I would imagine that when writing net_processing logic to download blocks for the background chainstate, that we would use this concept to only download blocks towards the snapshotted entry as well. ACKs for top commit: achow101: ACK a733dd7 jamesob: reACK a733dd7 ([`jamesob/ackr/27746.5.sdaftuar.rework_validation_logic`](https://github.com/jamesob/bitcoin/tree/ackr/27746.5.sdaftuar.rework_validation_logic)) Sjors: Code review ACK a733dd7. ryanofsky: Code review ACK a733dd7. Just suggested changes since the last review. There are various small things that could be followed up on, but I think this is ready for merge. Tree-SHA512: 9ec17746f22b9c27082743ee581b8adceb2bd322fceafa507b428bdcc3ffb8b4c6601fc61cc7bb1161f890c3d38503e8b49474da7b5ab1b1f38bda7aa8668675
Preserve ChainLock candidate exclusions in the new admission wrapper and keep Dash background-notification and EvoDB fixtures consistent with the tightened multi-chainstate candidate invariants.
…or discarded Snapshot marker promotion and discard end the dual-chainstate run, so transaction-less EvoDB access must resolve against the NORMAL identity again. Resolves the TODO left on SetDefaultIdentity in the M2 base. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
quorum_members_reconstruct_from_seeded_state_only forced DIP0003/v19/v20 active through const_cast on a TestChain100Setup chain that was mined with those forks inactive. The v20 quorum hash modifier then probed a pre-DIP3 coinbase for a chainlock and tripped GetTxPayload's payload-type assertion in --enable-debug builds. The abort longjmps past fixture teardown, so mocktime, gArgs, and worker threads leaked and poisoned every later case in the same test_dash process. Mine the fixture chain with the required activation heights already in force instead (new SnapshotActivationChainSetup with its own deterministic checkpoint), and restore const_cast-mutated consensus params through a scope guard so a failed assertion in one case can no longer leak mutated params into cases that run after it.
PastaPastaPasta
force-pushed
the
assumeutxo/m3-background-completion
branch
from
August 3, 2026 18:42
0868bc1 to
266aaea
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m4-evo-snapshot
branch
from
August 3, 2026 18:42
96a793c to
abb7a0c
Compare
PastaPastaPasta
force-pushed
the
assumeutxo/m3-background-completion
branch
5 times, most recently
from
August 8, 2026 02:10
14994f4 to
e3a8989
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue being fixed or feature implemented
Upstream UTXO-only snapshots can't validate a DIP3-active chain like Dash's:
masternode lists, LLMQ quorums/rotation, the credit pool, and EHF state all
need to be captured and reconstructible from a snapshot, independently
verifiable against the chain (not merely trusted). This is milestone 4/7, the
largest design unit of the series (10 review rounds; design docs D1/D2). The
execution plan and per-unit ledger (M4 section) are tracked outside the repo
in a private gist, not checked in: https://gist.github.com/PastaPastaPasta/aa52b1f89fb74a0566ba3b5e15afab6a
loadtxoutsetis deliberately still unexposed after this PR — it needs theformat landed here, but the RPC itself arrives in M5.
PR 4/7 in the stacked series — base is M3 (
assumeutxo/m3-background-completion).This PR implements the user-locked design decision (2026-07-11): the evo
snapshot carries the FULL quorum member-reconstruction history, diff-encoded
from the base MN list, plus per-work-block score modifiers/CbTx signatures —
chosen over a smaller verification-only format because both rotated and
non-rotated
GetHashModifierneed CbTx data unavailable pre-completion, andreconstructing the active non-rotated set needs MN lists ~24 cycles (~576
blocks) to ~2,304 blocks back, depending on quorum type.
What was done?
evo_hash:CEvoSnapshotv3 replacesfull historical lists with one canonical reverse diff chain from the base
list through every required LLMQ work block (predecessor/target hashes,
height, registration counter, canonical target hash, bounded canonical
CDeterministicMNListDiffper link; exact(llmq_type, work_block_hash, modifier)tuples for every commitment/rotation reference). A pure horizonenumerator covers non-rotated active+safety cycles and rotated H-C..H-4C
(worst default span 2,312 blocks; 104 on regtest), with a params-derived
link count cap (192 max with currently enabled LLMQs) and a shared
786,432-operation decode budget. Five review rounds.
dumptxoutsetbuilds the v3section under the coins-cursor
cs_mainpin; populate validates ancestry,exact history/modifier coverage, work-block relationships, evo_hash, and
CbTx roots, then atomically seeds reconstructed lists, modifiers,
commitments, rotation snapshots, credit pool, and MNHF state under the
snapshot EvoDB identity. A modifier-mismatch signal is contained at quorum,
connect/disconnect, direct-validity, VerifyDB, and replay boundaries and
routes to the controlled invalid-snapshot/shutdown path. 3 review rounds.
lifecycle dirs before the EvoDB wipe/detection; an
assumeutxoprune lockprotects the base block until deferred CbTx/evo checks complete. Round 2
review found a released prune lock resurrected by
DisconnectTiprewind —fixed via
BlockManager::DeletePruneLock.broken by extracting
evo/snapshot_types.hand moving snapshotload/seed/completion orchestration into
evo/snapshot_load.cpp(note forfuture backports: upstream hunks touching
PopulateAndValidateSnapshot/MaybeCompleteSnapshotValidationinvalidation.cppmust be retargetedthere).
cross-checking, and rotated/non-rotated
GetAllQuorumMembersreconstruction with block data unavailable, using an independent manager
oracle; removing a seed trips the fail-closed NORMAL-state barrier
(also covers
ScanQuorumsand recovered-signature verification).How Has This Been Tested?
make check: exit 0, 0 failures (rerun on the final post-refactor tree).feature_assumeutxo_dash,feature_dip3×2,feature_llmq_signing×2,feature_llmq_rotation,feature_llmq_chainlocks,feature_mnehf,feature_asset_locks,feature_init,feature_reindex,feature_pruning— all passed.Breaking Changes
Snapshot format is new/versioned (v3) and unreleased; no compatibility
surface exists yet.
loadtxoutsetremains unexposed to end users until M5.Checklist: