Skip to content

backport: assumeutxo M4 — evo snapshot format v3 and LLMQ reconstruction - #53

Open
PastaPastaPasta wants to merge 29 commits into
assumeutxo/m3-background-completionfrom
assumeutxo/m4-evo-snapshot
Open

backport: assumeutxo M4 — evo snapshot format v3 and LLMQ reconstruction#53
PastaPastaPasta wants to merge 29 commits into
assumeutxo/m3-background-completionfrom
assumeutxo/m4-evo-snapshot

Conversation

@PastaPastaPasta

@PastaPastaPasta PastaPastaPasta commented Jul 12, 2026

Copy link
Copy Markdown
Owner

Issue being fixed or feature implemented

Upstream UTXO-only snapshots can't validate a DIP3-active chain like Dash's:
masternode lists, LLMQ quorums/rotation, the credit pool, and EHF state all
need to be captured and reconstructible from a snapshot, independently
verifiable against the chain (not merely trusted). This is milestone 4/7, the
largest design unit of the series (10 review rounds; design docs D1/D2). The
execution plan and per-unit ledger (M4 section) are tracked outside the repo
in a private gist, not checked in: https://gist.github.com/PastaPastaPasta/aa52b1f89fb74a0566ba3b5e15afab6a
loadtxoutset is deliberately still unexposed after this PR — it needs the
format landed here, but the RPC itself arrives in M5.

PR 4/7 in the stacked series — base is M3 (assumeutxo/m3-background-completion).

This PR implements the user-locked design decision (2026-07-11): the evo
snapshot carries the FULL quorum member-reconstruction history, diff-encoded
from the base MN list, plus per-work-block score modifiers/CbTx signatures —
chosen over a smaller verification-only format because both rotated and
non-rotated GetHashModifier need CbTx data unavailable pre-completion, and
reconstructing the active non-rotated set needs MN lists ~24 cycles (~576
blocks) to ~2,304 blocks back, depending on quorum type.

What was done?

  • B4.1 — evo snapshot format v3 + evo_hash: CEvoSnapshot v3 replaces
    full historical lists with one canonical reverse diff chain from the base
    list through every required LLMQ work block (predecessor/target hashes,
    height, registration counter, canonical target hash, bounded canonical
    CDeterministicMNListDiff per link; exact (llmq_type, work_block_hash, modifier) tuples for every commitment/rotation reference). A pure horizon
    enumerator covers non-rotated active+safety cycles and rotated H-C..H-4C
    (worst default span 2,312 blocks; 104 on regtest), with a params-derived
    link count cap (192 max with currently enabled LLMQs) and a shared
    786,432-operation decode budget. Five review rounds.
  • B4.2 — dump emission + populate/seed v3: dumptxoutset builds the v3
    section under the coins-cursor cs_main pin; populate validates ancestry,
    exact history/modifier coverage, work-block relationships, evo_hash, and
    CbTx roots, then atomically seeds reconstructed lists, modifiers,
    commitments, rotation snapshots, credit pool, and MNHF state under the
    snapshot EvoDB identity. A modifier-mismatch signal is contained at quorum,
    connect/disconnect, direct-validity, VerifyDB, and replay boundaries and
    routes to the controlled invalid-snapshot/shutdown path. 3 review rounds.
  • B6 — init-order integration: reindex modes remove all snapshot
    lifecycle dirs before the EvoDB wipe/detection; an assumeutxo prune lock
    protects the base block until deferred CbTx/evo checks complete. Round 2
    review found a released prune lock resurrected by DisconnectTip rewind —
    fixed via BlockManager::DeletePruneLock.
  • Circular-deps refactor: 6 new cycles introduced by the M4 work were
    broken by extracting evo/snapshot_types.h and moving snapshot
    load/seed/completion orchestration into evo/snapshot_load.cpp (note for
    future backports: upstream hunks touching PopulateAndValidateSnapshot /
    MaybeCompleteSnapshotValidation in validation.cpp must be retargeted
    there).
  • Tests prove diff determinism, cumulative bounded decode, exact modifier
    cross-checking, and rotated/non-rotated GetAllQuorumMembers
    reconstruction with block data unavailable, using an independent manager
    oracle; removing a seed trips the fail-closed NORMAL-state barrier
    (also covers ScanQuorums and recovered-signature verification).

How Has This Been Tested?

  • make check: exit 0, 0 failures (rerun on the final post-refactor tree).
  • Functional battery (12): feature_assumeutxo_dash, feature_dip3 ×2,
    feature_llmq_signing ×2, feature_llmq_rotation,
    feature_llmq_chainlocks, feature_mnehf, feature_asset_locks,
    feature_init, feature_reindex, feature_pruning — all passed.
  • Lint: circular-dependencies PASS, whitespace PASS; python lint deferred to CI.
  • Milestone gate PASS on 2026-07-11.

Breaking Changes

Snapshot format is new/versioned (v3) and unreleased; no compatibility
surface exists yet. loadtxoutset remains unexposed to end users until M5.

Checklist:

  • I have performed a self-review of my own code
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have made corresponding changes to the documentation (docs rewrite lands in M7/B8)
  • I have assigned this pull request to a milestone

@coderabbitai

coderabbitai Bot commented Jul 12, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 289296d7-38c6-49e8-9490-2373773a3349

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch assumeutxo/m4-evo-snapshot

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch from 32d5966 to 4f9d716 Compare July 12, 2026 14:21
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m4-evo-snapshot branch from 52a0371 to ed610de Compare July 12, 2026 14:29
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch from 4f9d716 to f08bce5 Compare July 12, 2026 16:00
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m4-evo-snapshot branch from ed610de to 609b604 Compare July 12, 2026 16:01
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch from f08bce5 to a6e49d6 Compare July 13, 2026 22:45
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m4-evo-snapshot branch from 609b604 to 11d1fb7 Compare July 13, 2026 22:45
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch from a6e49d6 to eb757a0 Compare July 14, 2026 05:10
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m4-evo-snapshot branch 2 times, most recently from 022d3e6 to 590e34b Compare July 14, 2026 17:01
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch from eb757a0 to 2bc543c Compare July 14, 2026 17:01
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m4-evo-snapshot branch from 590e34b to 9e0f8d7 Compare July 20, 2026 16:59
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch from 2bc543c to 5b3d0d7 Compare July 20, 2026 16:59
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m4-evo-snapshot branch 2 times, most recently from e0377b0 to 9baa44d Compare August 1, 2026 19:55
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch from 569913c to 05723b8 Compare August 1, 2026 22:04
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m4-evo-snapshot branch from 9baa44d to ab27f8b Compare August 1, 2026 22:04
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch from 05723b8 to 667611e Compare August 1, 2026 22:27
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m4-evo-snapshot branch from ab27f8b to 2a968e5 Compare August 1, 2026 22:27
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch from 667611e to 328d224 Compare August 1, 2026 22:44
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m4-evo-snapshot branch 2 times, most recently from ea58a7f to ed41e4e Compare August 1, 2026 23:09
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch from 328d224 to f9f8fa6 Compare August 1, 2026 23:35
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m4-evo-snapshot branch 4 times, most recently from 336b093 to 269d77e Compare August 2, 2026 03:10
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch from f9f8fa6 to 63de2c1 Compare August 2, 2026 18:42
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m4-evo-snapshot branch from 269d77e to 02ced9e Compare August 2, 2026 18:42
PastaPastaPasta and others added 23 commits August 3, 2026 13:33
Disable DKG participation and quorum signing until snapshot background validation completes. Enforce the refusal at CreateSigShare, the actual share-production boundary, so direct RPC, async, and queued signing paths cannot bypass it.

The quorum sign RPC now returns a clear JSON-RPC error for both submit modes, and masternode status exposes the disabled participation state. Add unit coverage for the shared production-gate predicate across snapshot activation.
A WriteDerived failure means independently derived block data disagrees with the copy already recorded in EvoDB. That is local state corruption (or a cross-chainstate divergence bug), never evidence about the block being processed. Previously the mismatch surfaced as BLOCK_CONSENSUS: the block was persistently marked BLOCK_FAILED_VALID (surviving restart and forking the node off the network) and the relaying peer was handed a 100-point misbehavior score via BlockChecked, which background validation also triggers.

Instead, follow the existing EvoDbInconsistencyMessage convention: request node shutdown via AbortNode and fail validation with M_ERROR, which neither marks the block invalid nor punishes peers. The credit-pool and MNHF sites abort at the throw site because miner and RPC callers never pass through a validation-state catch; a typed EvoDbInconsistencyError lets the four block-path catch blocks that would otherwise swallow it into BLOCK_CONSENSUS reclassify it as M_ERROR.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ethrow

CMNHFManager::GetForBlock re-throws internal ProcessBlock failures as a plain runtime_error, which would let a downstream generic catch misreport an EvoDB mismatch (M_ERROR) as a consensus failure. Re-throw typed when the validation state carries M_ERROR.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reads and writes outside a BeginTransaction scope previously always bound to the NORMAL identity. While a snapshot chainstate is active, transaction-less consumers (RPC, mempool, miner, P2P serving) could not see snapshot-chain records still pending in the SNAPSHOT root overlay, and transaction-less writes dirtied the wrong overlay. CEvoDB now tracks a default identity which snapshot activation (ActivateSnapshot, ActivateExistingSnapshot) sets to SNAPSHOT and ResetChainstates resets; snapshot completion must reset it to NORMAL when marker promotion lands.

Also make the GetListForBlockInternal fallback loud: a missing list diff for a DIP3-active block is pending-elsewhere or corrupt data, never the pre-DIP3 genesis of the masternode list, so throw instead of silently caching an empty list and clobbering m_initial_snapshot_index. The thrown message carries the IsBlockDataUnavailableError sentinel and BuildSimplifiedMNListDiff converts it into a serve failure, so peers requesting such history are not penalized.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GetListForBlock can now throw when list data is unavailable (missing diff for a DIP3-active block). CActiveMasternodeManager::UpdatedBlockTip and GovernanceSigner's trigger creation run on the scheduler thread, where an uncaught exception terminates the node; catch it there and skip the update instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Discard Dash snapshot lifecycle markers only after the invalid snapshot directory rename succeeds. If the rename fails, preserving the markers keeps the existing restart recovery state recognizable while the upstream rename error is propagated to the fatal shutdown message.
Keep Dash’s mock shutdown callback while asserting the expected fatal diagnostic. The default callback reaches StartShutdown(), whose unit-test guard aborts the process before this Dash test can complete.
a733dd7 Remove unused function `reliesOnAssumedValid` (Suhas Daftuar)
d4a11ab Cache block index entry corresponding to assumeutxo snapshot base blockhash (Suhas Daftuar)
3556b85 Move CheckBlockIndex() from Chainstate to ChainstateManager (Suhas Daftuar)
0ce805b Documentation improvements for assumeutxo (Ryan Ofsky)
768690b Fix initialization of setBlockIndexCandidates when working with multiple chainstates (Suhas Daftuar)
d43a1f1 Tighten requirements for adding elements to setBlockIndexCandidates (Suhas Daftuar)
d0d40ea Move block-storage-related logic to ChainstateManager (Suhas Daftuar)
3cfc753 test: Clear block index flags when testing snapshots (Suhas Daftuar)
272fbc3 Update CheckBlockIndex invariants for chains based on an assumeutxo snapshot (Suhas Daftuar)
10c0571 Add wrapper for adding entries to a chainstate's block index candidates (Suhas Daftuar)
471da5f Move block-arrival information / preciousblock counters to ChainstateManager (Suhas Daftuar)
1cfc887 Remove CChain dependency in node/blockstorage (Suhas Daftuar)
fe86a7c Explicitly track maximum block height stored in undo files (Suhas Daftuar)

Pull request description:

  This PR proposes a clean up of the relationship between block storage and the chainstate objects, by moving the decision of whether to store a block on disk to something that is not chainstate-specific.  Philosophically, the decision of whether to store a block on disk is related to validation rules that do not require any UTXO state; for anti-DoS reasons we were using some chainstate-specific heuristics, and those have been reworked here to achieve the proposed separation.

  This PR also fixes a bug in how a chainstate's `setBlockIndexCandidates` was being initialized; it should always have all the HAVE_DATA block index entries that have more work than the chain tip.  During startup, we were not fully populating `setBlockIndexCandidates` in some scenarios involving multiple chainstates.

  Further, this PR establishes a concept that whenever we have 2 chainstates, that we always know the snapshotted chain's base block and the base block's hash must be an element of our block index. Given that, we can establish a new invariant that the background validation chainstate only needs to consider blocks leading to that snapshotted block entry as potential candidates for its tip. As a followup I would imagine that when writing net_processing logic to download blocks for the background chainstate, that we would use this concept to only download blocks towards the snapshotted entry as well.

ACKs for top commit:
  achow101:
    ACK a733dd7
  jamesob:
    reACK a733dd7 ([`jamesob/ackr/27746.5.sdaftuar.rework_validation_logic`](https://github.com/jamesob/bitcoin/tree/ackr/27746.5.sdaftuar.rework_validation_logic))
  Sjors:
    Code review ACK a733dd7.
  ryanofsky:
    Code review ACK a733dd7. Just suggested changes since the last review. There are various small things that could be followed up on, but I think this is ready for merge.

Tree-SHA512: 9ec17746f22b9c27082743ee581b8adceb2bd322fceafa507b428bdcc3ffb8b4c6601fc61cc7bb1161f890c3d38503e8b49474da7b5ab1b1f38bda7aa8668675
Preserve ChainLock candidate exclusions in the new admission wrapper and keep Dash background-notification and EvoDB fixtures consistent with the tightened multi-chainstate candidate invariants.
…or discarded

Snapshot marker promotion and discard end the dual-chainstate run, so transaction-less EvoDB access must resolve against the NORMAL identity again. Resolves the TODO left on SetDefaultIdentity in the M2 base.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
quorum_members_reconstruct_from_seeded_state_only forced DIP0003/v19/v20 active through const_cast on a TestChain100Setup chain that was mined with those forks inactive. The v20 quorum hash modifier then probed a pre-DIP3 coinbase for a chainlock and tripped GetTxPayload's payload-type assertion in --enable-debug builds. The abort longjmps past fixture teardown, so mocktime, gArgs, and worker threads leaked and poisoned every later case in the same test_dash process.

Mine the fixture chain with the required activation heights already in force instead (new SnapshotActivationChainSetup with its own deterministic checkpoint), and restore const_cast-mutated consensus params through a scope guard so a failed assertion in one case can no longer leak mutated params into cases that run after it.
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch from 0868bc1 to 266aaea Compare August 3, 2026 18:42
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m4-evo-snapshot branch from 96a793c to abb7a0c Compare August 3, 2026 18:42
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch 5 times, most recently from 14994f4 to e3a8989 Compare August 8, 2026 02:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants