diff --git a/doc/design/assumeutxo.md b/doc/design/assumeutxo.md index 74aff7d392f8..9846f7f26a0b 100644 --- a/doc/design/assumeutxo.md +++ b/doc/design/assumeutxo.md @@ -3,9 +3,9 @@ Assumeutxo is a feature that allows fast bootstrapping of a validating dashd instance with a very similar security model to assumevalid. -The RPC commands `dumptxoutset` and `loadtxoutset` are used to respectively generate -and load UTXO snapshots. The utility script `./contrib/devtools/utxo_snapshot.sh` may -be of use. +The RPC commands `dumptxoutset` and `loadtxoutset` (yet to be merged) are used to +respectively generate and load UTXO snapshots. The utility script +`./contrib/devtools/utxo_snapshot.sh` may be of use. ## General background @@ -17,14 +17,9 @@ be of use. - A new block index `nStatus` flag is introduced, `BLOCK_ASSUMED_VALID`, to mark block index entries that are required to be assumed-valid by a chainstate created - from a UTXO snapshot. This flag is mostly used as a way to modify certain + from a UTXO snapshot. This flag is used as a way to modify certain CheckBlockIndex() logic to account for index entries that are pending validation by a - chainstate running asynchronously in the background. We also use this flag to control - which index entries are added to setBlockIndexCandidates during LoadBlockIndex(). - -- Indexing implementations via BaseIndex can no longer assume that indexation happens - sequentially, since background validation chainstates can submit BlockConnected - events out of order with the active chain. + chainstate running asynchronously in the background. - The concept of UTXO snapshots is treated as an implementation detail that lives behind the ChainstateManager interface. The external presentation of the changes @@ -76,9 +71,15 @@ original chainstate remains in use as active. Once the snapshot chainstate is loaded and validated, it is promoted to active chainstate and a sync to tip begins. A new chainstate directory is created in the -datadir for the snapshot chainstate called `chainstate_snapshot`. When this directory -is present in the datadir, the snapshot chainstate will be detected and loaded as -active on node startup (via `DetectSnapshotChainstate()`). +datadir for the snapshot chainstate called `chainstate_snapshot`. + +When this directory is present in the datadir, the snapshot chainstate will be detected +and loaded as active on node startup (via `DetectSnapshotChainstate()`). + +A special file is created within that directory, `base_blockhash`, which contains the +serialized `uint256` of the base block of the snapshot. This is used to reinitialize +the snapshot chainstate on subsequent inits. Otherwise, the directory is a normal +leveldb database. | | | | ---------- | ----------- | @@ -88,7 +89,7 @@ active on node startup (via `DetectSnapshotChainstate()`). The snapshot begins to sync to tip from its base block, technically in parallel with the original chainstate, but it is given priority during block download and is allocated most of the cache (see `MaybeRebalanceCaches()` and usages) as our chief -consideration is getting to network tip. +goal is getting to network tip. **Failure consideration:** if shutdown happens at any point during this phase, both chainstates will be detected during the next init and the process will resume. @@ -107,33 +108,36 @@ sequentially. ### Background chainstate hits snapshot base block Once the tip of the background chainstate hits the base block of the snapshot -chainstate, we stop use of the background chainstate by setting `m_stop_use` (not yet -committed - see bitcoin#15606), in `CompleteSnapshotValidation()`, which is checked in -`ActivateBestChain()`). We hash the background chainstate's UTXO set contents and -ensure it matches the compiled value in `CMainParams::m_assumeutxo_data`. - -The background chainstate data lingers on disk until shutdown, when in -`ChainstateManager::Reset()`, the background chainstate is cleaned up with -`ValidatedSnapshotShutdownCleanup()`, which renames the `chainstate_[hash]` datadir as -`chainstate`. +chainstate, we stop use of the background chainstate by setting `m_disabled`, in +`MaybeCompleteSnapshotValidation()`, which is checked in `ActivateBestChain()`. We hash the +background chainstate's UTXO set contents and ensure it matches the compiled value in +`CMainParams::m_assumeutxo_data`. In Dash, completion additionally compares the +deterministic masternode-list hash the background chainstate derived at the base block +against the hash recorded at snapshot activation, and the EvoDB best-block markers +against both chainstates' coins tips; any divergence fails completion with +`EVO_STATE_MISMATCH` and quarantines the snapshot exactly like a UTXO hash mismatch. | | | | ---------- | ----------- | -| number of chainstates | 2 (ibd has `m_stop_use=true`) | +| number of chainstates | 2 (ibd has `m_disabled=true`) | | active chainstate | snapshot | -**Failure consideration:** if dashd unexpectedly halts after `m_stop_use` is set on -the background chainstate but before `CompleteSnapshotValidation()` can finish, the -need to complete snapshot validation will be detected on subsequent init by -`ChainstateManager::CheckForUncleanShutdown()`. +The background chainstate data lingers on disk until the program is restarted. ### Dashd restarts sometime after snapshot validation has completed -When dashd initializes again, what began as the snapshot chainstate is now -indistinguishable from a chainstate that has been built from the traditional IBD -process, and will be initialized as such. +After a shutdown and subsequent restart, `LoadChainstate()` cleans up the background +chainstate with `ValidatedSnapshotCleanup()`, which renames the `chainstate_snapshot` +datadir as `chainstate` and removes the now unnecessary background chainstate data. | | | | ---------- | ----------- | | number of chainstates | 1 | -| active chainstate | ibd | +| active chainstate | ibd (was snapshot, but is now fully validated) | + +What began as the snapshot chainstate is now indistinguishable from a chainstate that +has been built from the traditional IBD process, and will be initialized as such. + +A file will be left in `chainstate/base_blockhash`, which indicates that the +chainstate, even though now fully validated, was originally started from a snapshot +with the corresponding base blockhash. diff --git a/src/bench/load_external.cpp b/src/bench/load_external.cpp index ec97be45ff1e..e11766929430 100644 --- a/src/bench/load_external.cpp +++ b/src/bench/load_external.cpp @@ -48,14 +48,13 @@ static void LoadExternalBlockFile(benchmark::Bench& bench) fclose(file); } - Chainstate& chainstate{testing_setup->m_node.chainman->ActiveChainstate()}; std::multimap blocks_with_unknown_parent; FlatFilePos pos; bench.run([&] { // "rb" is "binary, O_RDONLY", positioned to the start of the file. // The file will be closed by LoadExternalBlockFile(). FILE* file{fsbridge::fopen(blkfile, "rb")}; - chainstate.LoadExternalBlockFile(file, &pos, &blocks_with_unknown_parent); + testing_setup->m_node.chainman->LoadExternalBlockFile(file, &pos, &blocks_with_unknown_parent); }); fs::remove(blkfile); } diff --git a/src/chain.h b/src/chain.h index b4022f2a95d4..f7f1d24fb8a4 100644 --- a/src/chain.h +++ b/src/chain.h @@ -98,10 +98,10 @@ enum BlockStatus : uint32_t { BLOCK_VALID_TRANSACTIONS = 3, //! Outputs do not overspend inputs, no double spends, coinbase output ok, no immature coinbase spends, BIP30. - //! Implies all parents are also at least CHAIN. + //! Implies all parents are either at least VALID_CHAIN, or are ASSUMED_VALID BLOCK_VALID_CHAIN = 4, - //! Scripts & signatures ok. Implies all parents are also at least SCRIPTS. + //! Scripts & signatures ok. Implies all parents are either at least VALID_SCRIPTS, or are ASSUMED_VALID. BLOCK_VALID_SCRIPTS = 5, //! All validity bits. @@ -119,10 +119,18 @@ enum BlockStatus : uint32_t { BLOCK_CONFLICT_CHAINLOCK = 128, //!< conflicts with chainlock system /** - * If set, this indicates that the block index entry is assumed-valid. - * Certain diagnostics will be skipped in e.g. CheckBlockIndex(). - * It almost certainly means that the block's full validation is pending - * on a background chainstate. See `doc/design/assumeutxo.md`. + * If ASSUMED_VALID is set, it means that this block has not been validated + * and has validity status less than VALID_SCRIPTS. Also that it may have + * descendant blocks with VALID_SCRIPTS set, because they can be validated + * based on an assumeutxo snapshot. + * + * When an assumeutxo snapshot is loaded, the ASSUMED_VALID flag is added to + * unvalidated blocks at the snapshot height and below. Then, as the background + * validation progresses, and these blocks are validated, the ASSUMED_VALID + * flags are removed. See `doc/design/assumeutxo.md` for details. + * + * This flag is only used to implement checks in CheckBlockIndex() and + * should not be used elsewhere. */ BLOCK_ASSUMED_VALID = 256, }; diff --git a/src/evo/chainhelper.cpp b/src/evo/chainhelper.cpp index acffa3cd2d4a..9412607e93a0 100644 --- a/src/evo/chainhelper.cpp +++ b/src/evo/chainhelper.cpp @@ -7,14 +7,17 @@ #include #include #include +#include #include #include #include +#include #include #include #include #include #include +#include CChainstateHelper::CChainstateHelper(CEvoDB& evodb, CDeterministicMNManager& dmnman, const CMasternodeSync& mn_sync, llmq::CInstantSendManager& isman, llmq::CQuorumBlockProcessor& qblockman, @@ -23,6 +26,7 @@ CChainstateHelper::CChainstateHelper(CEvoDB& evodb, CDeterministicMNManager& dmn const llmq::CQuorumManager& qman) : isman{isman}, mn_sync{mn_sync}, + m_dmnman{dmnman}, credit_pool_manager{std::make_unique(evodb, chainman)}, m_chainlocks{chainlocks}, ehf_manager{std::make_unique(evodb, chainman)}, @@ -60,6 +64,11 @@ bool CChainstateHelper::HasChainLock(int nHeight, const uint256& blockHash) cons int32_t CChainstateHelper::GetBestChainLockHeight() const { return m_chainlocks.GetBestChainLockHeight(); } +uint256 CChainstateHelper::GetDeterministicMNListHash(const CBlockIndex* pindex) const +{ + return SerializeHash(m_dmnman.GetListForBlock(Assert(pindex))); +} + /** Passthrough functions to CCreditPoolManager */ CCreditPool CChainstateHelper::GetCreditPool(const CBlockIndex* const pindex) { diff --git a/src/evo/chainhelper.h b/src/evo/chainhelper.h index f68c48bd26bf..eac183777ba1 100644 --- a/src/evo/chainhelper.h +++ b/src/evo/chainhelper.h @@ -42,6 +42,7 @@ class CChainstateHelper private: llmq::CInstantSendManager& isman; const CMasternodeSync& mn_sync; + CDeterministicMNManager& m_dmnman; public: const std::unique_ptr credit_pool_manager; @@ -69,6 +70,9 @@ class CChainstateHelper bool HasChainLock(int nHeight, const uint256& blockHash) const; int32_t GetBestChainLockHeight() const; + /** Return a canonical hash of the deterministic MN list derived at a block. */ + uint256 GetDeterministicMNListHash(const CBlockIndex* pindex) const; + /** Passthrough functions to CCreditPoolManager */ CCreditPool GetCreditPool(const CBlockIndex* const pindex); diff --git a/src/evo/deterministicmns.cpp b/src/evo/deterministicmns.cpp index b25f4d01a514..c66d87afdb62 100644 --- a/src/evo/deterministicmns.cpp +++ b/src/evo/deterministicmns.cpp @@ -814,8 +814,8 @@ CDeterministicMNList CDeterministicMNManager::GetListForBlockInternal(gsl::not_n // cached chain) bootstraps an empty list here and rebuilds via // ProcessBlock from that point on. throw BlockDataUnavailableError(strprintf( - "CDeterministicMNManager::%s -- masternode list diff for block %s is not available (pruned or below an unvalidated snapshot base)", - __func__, pindex->GetBlockHash().ToString())); + "CDeterministicMNManager::%s -- masternode list diff for block %s %s", + __func__, pindex->GetBlockHash().ToString(), BLOCK_DATA_UNAVAILABLE_SUFFIX)); } // no snapshot and no diff on disk means that it's the initial snapshot m_initial_snapshot_index = pindex; diff --git a/src/evo/deterministicmns.h b/src/evo/deterministicmns.h index 68d48f00badb..3790eb2f4f9a 100644 --- a/src/evo/deterministicmns.h +++ b/src/evo/deterministicmns.h @@ -26,6 +26,7 @@ #include #include #include +#include #include #include #include @@ -685,12 +686,21 @@ struct MNListUpdates CDeterministicMNListDiff diff; }; +/** Sentinel suffix carried by serving-failure messages that report locally + * missing block data (pruned, or below an unvalidated snapshot base) rather + * than peer misbehavior. Every producer must append this verbatim so that + * IsBlockDataUnavailableError() keeps recognizing the condition; matching on + * this constant is what keeps such requests from penalizing the peer. */ +inline constexpr std::string_view BLOCK_DATA_UNAVAILABLE_SUFFIX{ + "is not available (pruned or below an unvalidated snapshot base)"}; + /** Thrown when the masternode list for a block cannot be reconstructed because * the data is not on this node yet (pruned, or below an unvalidated snapshot * base, or pending in another chainstate's unflushed EvoDB overlay). Distinct * from the plain std::runtime_error that CDeterministicMNList::ApplyDiff * raises for genuine local corruption, which must never be swallowed. - * The message carries the sentinel matched by IsBlockDataUnavailableError(). */ + * The message carries BLOCK_DATA_UNAVAILABLE_SUFFIX, matched by + * IsBlockDataUnavailableError(). */ class BlockDataUnavailableError : public std::runtime_error { public: @@ -741,6 +751,12 @@ class CDeterministicMNManager }; CDeterministicMNList GetListAtChainTip() EXCLUSIVE_LOCKS_REQUIRED(!cs); + void SetListForBlockForTesting(const CDeterministicMNList& list) EXCLUSIVE_LOCKS_REQUIRED(!cs) + { + LOCK(cs); + mnListsCache.insert_or_assign(list.GetBlockHash(), list); + } + // Test if given TX is a ProRegTx which also contains the collateral at index n static bool IsProTxWithCollateral(const CTransactionRef& tx, uint32_t n); diff --git a/src/evo/evodb.cpp b/src/evo/evodb.cpp index f9f0aa255924..88787fe1e8ea 100644 --- a/src/evo/evodb.cpp +++ b/src/evo/evodb.cpp @@ -6,6 +6,8 @@ #include +#include + CEvoDBScopedCommitter::CEvoDBScopedCommitter(CEvoDB& _evoDB, EvoDbIdentity identity) : evoDB{_evoDB}, identity{identity} @@ -90,13 +92,13 @@ void CEvoDB::RollbackCurTransaction(EvoDbIdentity identity) active_transaction.reset(); } -bool CEvoDB::CommitRootTransaction(EvoDbIdentity identity) +bool CEvoDB::CommitRootTransaction(EvoDbIdentity identity, bool sync) { LOCK(cs); auto& context = GetContext(identity); assert(context.cur_transaction.IsClean()); context.root_transaction.Commit(); - bool ret = db->WriteBatch(context.root_batch); + bool ret = db->WriteBatch(context.root_batch, sync); context.root_batch.Clear(); return ret; } @@ -146,5 +148,93 @@ void CEvoDB::EraseSnapshotMarkers() LOCK(cs); auto& transaction = GetContext(GetCurrentIdentity()).cur_transaction; transaction.Erase(std::make_pair(EVODB_BEST_BLOCK, uint8_t{1})); + transaction.Erase(EVODB_SNAPSHOT_MNLIST_HASH); + transaction.Erase(EVODB_BACKGROUND_MNLIST_HASH); transaction.Erase(EVODB_DUAL_CHAINSTATE); } + +void CEvoDB::WriteSnapshotBaseMNListHash(const uint256& hash) +{ + Write(EVODB_SNAPSHOT_MNLIST_HASH, hash); +} + +bool CEvoDB::ReadSnapshotBaseMNListHash(uint256& hash) +{ + // Lifecycle markers are read at rest (completion after both identities' + // sync commits, or startup recovery). Read the raw DB so the result cannot + // depend on which transaction-less default identity happens to be current. + LOCK(cs); + return db->Read(EVODB_SNAPSHOT_MNLIST_HASH, hash); +} + +void CEvoDB::WriteBackgroundMNListHash(const uint256& block_hash, const uint256& mn_list_hash) +{ + Write(EVODB_BACKGROUND_MNLIST_HASH, std::make_pair(block_hash, mn_list_hash)); +} + +bool CEvoDB::ReadBackgroundMNListHash(uint256& block_hash, uint256& mn_list_hash) +{ + // See ReadSnapshotBaseMNListHash: at-rest raw read, identity-independent. + LOCK(cs); + std::pair value; + if (!db->Read(EVODB_BACKGROUND_MNLIST_HASH, value)) return false; + std::tie(block_hash, mn_list_hash) = value; + return true; +} + +bool CEvoDB::PromoteSnapshotMarkers(const uint256& expected_snapshot_tip) +{ + LOCK(cs); + assert(!active_transaction.has_value()); + for (const auto& [_, context] : transaction_contexts) { + if (!context) continue; + assert(context->cur_transaction.IsClean()); + assert(context->root_transaction.IsClean()); + } + + const auto snapshot_key = std::make_pair(EVODB_BEST_BLOCK, uint8_t{1}); + uint256 snapshot_tip; + if (!db->Read(snapshot_key, snapshot_tip)) { + uint256 normal_tip; + const bool already_promoted = db->Read(EVODB_BEST_BLOCK, normal_tip) && normal_tip == expected_snapshot_tip && + !db->Exists(EVODB_DUAL_CHAINSTATE) && !db->Exists(EVODB_SNAPSHOT_MNLIST_HASH) && + !db->Exists(EVODB_BACKGROUND_MNLIST_HASH); + if (already_promoted) m_default_identity = EvoDbIdentity::NORMAL; + return already_promoted; + } + if (snapshot_tip != expected_snapshot_tip) return false; + + CDBBatch batch{*db}; + batch.Write(EVODB_BEST_BLOCK, snapshot_tip); + batch.Erase(snapshot_key); + batch.Erase(EVODB_SNAPSHOT_MNLIST_HASH); + batch.Erase(EVODB_BACKGROUND_MNLIST_HASH); + batch.Erase(EVODB_DUAL_CHAINSTATE); + if (!db->WriteBatch(batch, /*fSync=*/true)) return false; + // The dual-chainstate run is over: the promoted state is the NORMAL + // identity, so transaction-less access must resolve there again. + m_default_identity = EvoDbIdentity::NORMAL; + return true; +} + +bool CEvoDB::DiscardSnapshotMarkers() +{ + LOCK(cs); + assert(!active_transaction.has_value()); + for (const auto& [_, context] : transaction_contexts) { + if (!context) continue; + assert(context->cur_transaction.IsClean()); + assert(context->root_transaction.IsClean()); + } + + CDBBatch batch{*db}; + batch.Erase(std::make_pair(EVODB_BEST_BLOCK, uint8_t{1})); + batch.Erase(EVODB_SNAPSHOT_MNLIST_HASH); + batch.Erase(EVODB_BACKGROUND_MNLIST_HASH); + batch.Erase(EVODB_DUAL_CHAINSTATE); + if (!db->WriteBatch(batch, /*fSync=*/true)) return false; + // The snapshot chainstate is gone; transaction-less access must resolve + // against the NORMAL identity again. + m_default_identity = EvoDbIdentity::NORMAL; + return true; +} diff --git a/src/evo/evodb.h b/src/evo/evodb.h index 40e03f4724e0..40d37b48229d 100644 --- a/src/evo/evodb.h +++ b/src/evo/evodb.h @@ -29,9 +29,9 @@ static const std::string EVODB_BEST_BLOCK = "b_b4"; // with this legacy marker. That pair is the background chainstate's own coins // and marker, so downgrading mid-snapshot safely reverts to background IBD. static const std::string EVODB_DUAL_CHAINSTATE = "b_dcs"; +static const std::string EVODB_SNAPSHOT_MNLIST_HASH = "b_dcs_mn"; +static const std::string EVODB_BACKGROUND_MNLIST_HASH = "b_dcs_bg_mn"; -// TODO(assumeutxo): snapshot completion must promote the SNAPSHOT marker to -// the legacy key when chainstate_snapshot is renamed over chainstate. enum class EvoDbIdentity { NORMAL, SNAPSHOT, @@ -221,14 +221,13 @@ class CEvoDB return result; } - bool CommitRootTransaction(EvoDbIdentity identity = EvoDbIdentity::NORMAL) EXCLUSIVE_LOCKS_REQUIRED(!cs); + bool CommitRootTransaction(EvoDbIdentity identity = EvoDbIdentity::NORMAL, bool sync = false) EXCLUSIVE_LOCKS_REQUIRED(!cs); bool IsEmpty() { return db->IsEmpty(); } //! Set the identity used by reads/writes outside any transaction. Must - //! track the active chainstate: snapshot activation sets SNAPSHOT. - //! TODO(assumeutxo): snapshot completion (marker promotion) must reset - //! this to NORMAL. + //! track the active chainstate: snapshot activation sets SNAPSHOT; + //! PromoteSnapshotMarkers/DiscardSnapshotMarkers reset it to NORMAL. void SetDefaultIdentity(EvoDbIdentity identity) EXCLUSIVE_LOCKS_REQUIRED(!cs) { LOCK(cs); @@ -240,12 +239,27 @@ class CEvoDB void WriteBestBlock(EvoDbIdentity identity, const uint256& hash) EXCLUSIVE_LOCKS_REQUIRED(!cs); void WriteDualChainstateMarker() EXCLUSIVE_LOCKS_REQUIRED(!cs); bool HasDualChainstateMarker() EXCLUSIVE_LOCKS_REQUIRED(!cs); - //! Undo WriteBestBlock(SNAPSHOT) and WriteDualChainstateMarker(). Needed - //! when snapshot activation is abandoned after those markers were already - //! committed: a stale dual-chainstate marker turns supported legacy - //! bootstrapping into "unavailable history", and a stale SNAPSHOT marker - //! would let a later snapshot directory pass ActivateExistingSnapshot(). + //! Undo every snapshot lifecycle marker (SNAPSHOT best block, base and + //! background MN-list hashes, dual-chainstate marker). Needed when snapshot + //! activation is abandoned after those markers were already committed: a + //! stale dual-chainstate marker turns supported legacy bootstrapping into + //! "unavailable history", and a stale SNAPSHOT marker would let a later + //! snapshot directory pass ActivateExistingSnapshot(). void EraseSnapshotMarkers() EXCLUSIVE_LOCKS_REQUIRED(!cs); + void WriteSnapshotBaseMNListHash(const uint256& hash) EXCLUSIVE_LOCKS_REQUIRED(!cs); + bool ReadSnapshotBaseMNListHash(uint256& hash) EXCLUSIVE_LOCKS_REQUIRED(!cs); + void WriteBackgroundMNListHash(const uint256& block_hash, const uint256& mn_list_hash) EXCLUSIVE_LOCKS_REQUIRED(!cs); + bool ReadBackgroundMNListHash(uint256& block_hash, uint256& mn_list_hash) EXCLUSIVE_LOCKS_REQUIRED(!cs); + + /** + * Atomically promote the surviving snapshot marker to the legacy NORMAL key + * and remove all dual-chainstate metadata. Both identity transaction trees + * must already be fully committed by the caller. + */ + bool PromoteSnapshotMarkers(const uint256& expected_snapshot_tip) EXCLUSIVE_LOCKS_REQUIRED(!cs); + + /** Remove snapshot metadata after rejecting a snapshot, preserving NORMAL. */ + bool DiscardSnapshotMarkers() EXCLUSIVE_LOCKS_REQUIRED(!cs); bool VerifyBestBlock(const uint256& hash) EXCLUSIVE_LOCKS_REQUIRED(!cs) { return VerifyBestBlock(EvoDbIdentity::NORMAL, hash); } void WriteBestBlock(const uint256& hash) EXCLUSIVE_LOCKS_REQUIRED(!cs) { WriteBestBlock(EvoDbIdentity::NORMAL, hash); } diff --git a/src/evo/smldiff.cpp b/src/evo/smldiff.cpp index b0de56d2c964..4aa210dcb7d1 100644 --- a/src/evo/smldiff.cpp +++ b/src/evo/smldiff.cpp @@ -188,8 +188,8 @@ bool BuildSimplifiedMNListDiff(CDeterministicMNManager& dmnman, const Chainstate // node has not validated the base yet. Only the target block is read // from disk (for cbTx and its merkle tree). if (!(blockIndex->nStatus & BLOCK_HAVE_DATA)) { - errorRet = strprintf("block data for block %s is not available (pruned or below an unvalidated snapshot base)", - blockIndex->GetBlockHash().ToString()); + errorRet = strprintf("block data for block %s %s", + blockIndex->GetBlockHash().ToString(), BLOCK_DATA_UNAVAILABLE_SUFFIX); return false; } @@ -247,5 +247,5 @@ bool BuildSimplifiedMNListDiff(CDeterministicMNManager& dmnman, const Chainstate bool IsBlockDataUnavailableError(const std::string& error) { - return error.find("is not available (pruned or below an unvalidated snapshot base)") != std::string::npos; + return error.find(BLOCK_DATA_UNAVAILABLE_SUFFIX) != std::string::npos; } diff --git a/src/evo/specialtxman.cpp b/src/evo/specialtxman.cpp index 527024de4e1d..055ea7e0b0be 100644 --- a/src/evo/specialtxman.cpp +++ b/src/evo/specialtxman.cpp @@ -745,6 +745,15 @@ bool CSpecialTxProcessor::ProcessSpecialTxsInBlock(Chainstate& chainstate, const return false; } } + if (!fJustCheck) { + // Persist the list produced by this chainstate's own connection of + // the snapshot base block (no-op for every other block). Snapshot + // activation may populate the shared MN-list cache with seeded + // state, so completion must not reconstruct this value through that + // cache. Before DIP3 activates, mn_list is the independently + // computed empty list. + chainstate.RecordBackgroundMNListHash(pindex, mn_list); + } int64_t nTime6 = GetTimeMicros(); nTimeDMN += nTime6 - nTime5; diff --git a/src/init.cpp b/src/init.cpp index 2cb11a7687a0..3607311e002c 100644 --- a/src/init.cpp +++ b/src/init.cpp @@ -2024,7 +2024,7 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info) } } - if (status == node::ChainstateLoadStatus::FAILURE_INCOMPATIBLE_DB) { + if (status == node::ChainstateLoadStatus::FAILURE_FATAL || status == node::ChainstateLoadStatus::FAILURE_INCOMPATIBLE_DB) { return InitError(error); } diff --git a/src/llmq/blockprocessor.cpp b/src/llmq/blockprocessor.cpp index ce2d24057ea2..82df24a77947 100644 --- a/src/llmq/blockprocessor.cpp +++ b/src/llmq/blockprocessor.cpp @@ -369,6 +369,12 @@ bool CQuorumBlockProcessor::ProcessCommitment(Chainstate& chainstate, int nHeigh !SerializedEqual(stored_commitment, std::make_pair(qc, blockHash))) { // Preserve the existing duplicate-commitment result while allowing an // exact block re-derivation to proceed through all validation below. + // Note: a commitment retained by UndoBlock for another chainstate's + // benefit would hit this path if this chain later re-mined the same + // qc in a different block. That needs a disconnect of a block shared + // with the other chainstate, which background validation (advancing + // only toward the snapshot base along the snapshot chain) never does; + // revisit if background reorgs ever become possible. return state.Invalid(BlockValidationResult::BLOCK_CONSENSUS, "bad-qc-dup"); } diff --git a/src/llmq/snapshot.cpp b/src/llmq/snapshot.cpp index cd0a4e6bf124..51d3a44007b8 100644 --- a/src/llmq/snapshot.cpp +++ b/src/llmq/snapshot.cpp @@ -5,6 +5,7 @@ #include #include +#include #include #include #include @@ -23,8 +24,7 @@ bool CheckBlockDataAvailable(gsl::not_null pindex, std::stri EXCLUSIVE_LOCKS_REQUIRED(::cs_main) { if (pindex->nStatus & BLOCK_HAVE_DATA) return true; - error = strprintf("block data for block %s is not available (pruned or below an unvalidated snapshot base)", - pindex->GetBlockHash().ToString()); + error = strprintf("block data for block %s %s", pindex->GetBlockHash().ToString(), BLOCK_DATA_UNAVAILABLE_SUFFIX); return false; } diff --git a/src/node/blockstorage.cpp b/src/node/blockstorage.cpp index 2a1253bf56b5..8676bd846a7b 100644 --- a/src/node/blockstorage.cpp +++ b/src/node/blockstorage.cpp @@ -641,7 +641,7 @@ fs::path GetBlockPosFilename(const FlatFilePos& pos) return BlockFileSeq().FileName(pos); } -bool BlockManager::FindBlockPos(FlatFilePos& pos, unsigned int nAddSize, unsigned int nHeight, CChain& active_chain, uint64_t nTime, bool fKnown) +bool BlockManager::FindBlockPos(FlatFilePos& pos, unsigned int nAddSize, unsigned int nHeight, uint64_t nTime, bool fKnown) { LOCK(cs_LastBlockFile); @@ -667,7 +667,7 @@ bool BlockManager::FindBlockPos(FlatFilePos& pos, unsigned int nAddSize, unsigne // when the undo file is keeping up with the block file, we want to flush it explicitly // when it is lagging behind (more blocks arrive than are being connected), we let the // undo block write case handle it - finalize_undo = (m_blockfile_info[nFile].nHeightLast == (unsigned int)active_chain.Tip()->nHeight); + finalize_undo = (m_blockfile_info[nFile].nHeightLast == m_undo_height_in_last_blockfile); nFile++; if (m_blockfile_info.size() <= nFile) { m_blockfile_info.resize(nFile + 1); @@ -683,6 +683,7 @@ bool BlockManager::FindBlockPos(FlatFilePos& pos, unsigned int nAddSize, unsigne } FlushBlockFile(!fKnown, finalize_undo); m_last_blockfile = nFile; + m_undo_height_in_last_blockfile = 0; // No undo data yet in the new file, so reset our undo-height tracking. } m_blockfile_info[nFile].AddBlock(nHeight, nTime); @@ -771,8 +772,9 @@ bool BlockManager::WriteUndoDataForBlock(const CBlockUndo& blockundo, BlockValid // the FindBlockPos function if (_pos.nFile < m_last_blockfile && static_cast(block.nHeight) == m_blockfile_info[_pos.nFile].nHeightLast) { FlushUndoFile(_pos.nFile, true); + } else if (_pos.nFile == m_last_blockfile && static_cast(block.nHeight) > m_undo_height_in_last_blockfile) { + m_undo_height_in_last_blockfile = block.nHeight; } - // update nUndoPos in block index block.nUndoPos = _pos.nPos; block.nStatus |= BLOCK_HAVE_UNDO; @@ -826,7 +828,7 @@ bool ReadBlockFromDisk(CBlock& block, const CBlockIndex* pindex, const Consensus return true; } -FlatFilePos BlockManager::SaveBlockToDisk(const CBlock& block, int nHeight, CChain& active_chain, const FlatFilePos* dbp) +FlatFilePos BlockManager::SaveBlockToDisk(const CBlock& block, int nHeight, const FlatFilePos* dbp) { unsigned int nBlockSize = ::GetSerializeSize(block, CLIENT_VERSION); FlatFilePos blockPos; @@ -839,7 +841,7 @@ FlatFilePos BlockManager::SaveBlockToDisk(const CBlock& block, int nHeight, CCha // we add BLOCK_SERIALIZATION_HEADER_SIZE only for new blocks since they will have the serialization header added when written to disk. nBlockSize += static_cast(BLOCK_SERIALIZATION_HEADER_SIZE); } - if (!FindBlockPos(blockPos, nBlockSize, nHeight, active_chain, block.GetBlockTime(), position_known)) { + if (!FindBlockPos(blockPos, nBlockSize, nHeight, block.GetBlockTime(), position_known)) { error("%s: FindBlockPos failed", __func__); return FlatFilePos(); } @@ -889,7 +891,7 @@ void ThreadImport(ChainstateManager& chainman, std::vector vImportFile break; // This error is logged in OpenBlockFile } LogPrintf("Reindexing block file blk%05u.dat...\n", (unsigned int)nFile); - chainman.ActiveChainstate().LoadExternalBlockFile(file, &pos, &blocks_with_unknown_parent); + chainman.LoadExternalBlockFile(file, &pos, &blocks_with_unknown_parent); if (ShutdownRequested()) { LogPrintf("Shutdown requested. Exit %s\n", __func__); return; @@ -908,7 +910,7 @@ void ThreadImport(ChainstateManager& chainman, std::vector vImportFile FILE *file = fsbridge::fopen(path, "rb"); if (file) { LogPrintf("Importing blocks file %s...\n", fs::PathToString(path)); - chainman.ActiveChainstate().LoadExternalBlockFile(file); + chainman.LoadExternalBlockFile(file); if (ShutdownRequested()) { LogPrintf("Shutdown requested. Exit %s\n", __func__); return; diff --git a/src/node/blockstorage.h b/src/node/blockstorage.h index 84801e7d6a98..66eb07f4b4e8 100644 --- a/src/node/blockstorage.h +++ b/src/node/blockstorage.h @@ -25,7 +25,6 @@ class ArgsManager; class BlockValidationState; class CBlock; class CBlockUndo; -class CChain; class CChainParams; class Chainstate; class ChainstateManager; @@ -102,7 +101,7 @@ class BlockManager EXCLUSIVE_LOCKS_REQUIRED(cs_main); void FlushBlockFile(bool fFinalize = false, bool finalize_undo = false); void FlushUndoFile(int block_file, bool finalize = false); - bool FindBlockPos(FlatFilePos& pos, unsigned int nAddSize, unsigned int nHeight, CChain& active_chain, uint64_t nTime, bool fKnown); + bool FindBlockPos(FlatFilePos& pos, unsigned int nAddSize, unsigned int nHeight, uint64_t nTime, bool fKnown); bool FindUndoPos(BlockValidationState& state, int nFile, FlatFilePos& pos, unsigned int nAddSize); /* Calculate the block/rev files to delete based on height specified by user with RPC command pruneblockchain */ @@ -128,6 +127,19 @@ class BlockManager RecursiveMutex cs_LastBlockFile; std::vector m_blockfile_info; int m_last_blockfile = 0; + + // Track the height of the highest block in m_last_blockfile whose undo + // data has been written. Block data is written to block files in download + // order, but is written to undo files in validation order, which is + // usually in order by height. To avoid wasting disk space, undo files will + // be trimmed whenever the corresponding block file is finalized and + // the height of the highest block written to the block file equals the + // height of the highest block written to the undo file. This is a + // heuristic and can sometimes preemptively trim undo files that will write + // more data later, and sometimes fail to trim undo files that can't have + // more data written later. + unsigned int m_undo_height_in_last_blockfile = 0; + /** Global flag to indicate we should check to see if there are * block/undo files that should be deleted. Set on startup * or if we allocate more file space when we're in prune mode @@ -195,7 +207,7 @@ class BlockManager EXCLUSIVE_LOCKS_REQUIRED(::cs_main); /** Store block on disk. If dbp is not nullptr, then it provides the known position of the block within a block file on disk. */ - FlatFilePos SaveBlockToDisk(const CBlock& block, int nHeight, CChain& active_chain, const FlatFilePos* dbp); + FlatFilePos SaveBlockToDisk(const CBlock& block, int nHeight, const FlatFilePos* dbp); /** Calculate the amount of disk space the block & undo files currently use */ uint64_t CalculateCurrentUsage(); diff --git a/src/node/chainstate.cpp b/src/node/chainstate.cpp index b8a83cab1a10..aa9d85976ceb 100644 --- a/src/node/chainstate.cpp +++ b/src/node/chainstate.cpp @@ -18,6 +18,7 @@ #include #include #include +#include #include #include @@ -35,65 +36,117 @@ #include namespace node { -ChainstateLoadResult LoadChainstate(ChainstateManager& chainman, const CacheSizes& cache_sizes, - const ChainstateLoadOptions& options, std::unique_ptr& evodb, - std::unique_ptr& dmnman, std::unique_ptr& llmq_ctx, - std::unique_ptr& chain_helper) +static bool RecoverSnapshotCleanup(CEvoDB& evodb, const fs::path& data_dir, bilingual_str& error) { - assert(options.mn_metaman); - assert(options.sporkman); - assert(options.chainlocks); - assert(options.mn_sync); - - const bool to_wipe_data = options.reindex || options.reindex_chainstate; - auto is_coinsview_empty = [&](Chainstate* chainstate) EXCLUSIVE_LOCKS_REQUIRED(::cs_main) { - return to_wipe_data || chainstate->CoinsTip().GetBestBlock().IsNull(); - }; - - if (!hashAssumeValid.IsNull()) { - LogPrintf("Assuming ancestors of block %s have valid signatures.\n", hashAssumeValid.GetHex()); - } else { - LogPrintf("Validating signatures for all blocks.\n"); - } - LogPrintf("Setting nMinimumChainWork=%s\n", nMinimumChainWork.GetHex()); - if (nMinimumChainWork < UintToArith256(chainman.GetConsensus().nMinimumChainWork)) { - LogPrintf("Warning: nMinimumChainWork set below default value of %s\n", chainman.GetConsensus().nMinimumChainWork.GetHex()); - } - if (nPruneTarget == std::numeric_limits::max()) { - LogPrintf("Block pruning enabled. Use RPC call pruneblockchain(height) to manually prune block and undo files.\n"); - } else if (nPruneTarget) { - LogPrintf("Prune configured to target %u MiB on disk for block and undo files.\n", nPruneTarget / 1024 / 1024); + const fs::path normal{data_dir / "chainstate"}; + const fs::path snapshot{data_dir / "chainstate_snapshot"}; + const fs::path to_delete{data_dir / "chainstate_todelete"}; + const fs::path invalid{data_dir / "chainstate_snapshot_INVALID"}; + + uint256 snapshot_tip; + const bool has_snapshot_tip{evodb.ReadBestBlock(EvoDbIdentity::SNAPSHOT, snapshot_tip)}; + uint256 ignored; + const bool has_metadata{has_snapshot_tip || evodb.HasDualChainstateMarker() || + evodb.ReadSnapshotBaseMNListHash(ignored)}; + + // The first validated-cleanup rename completed. Roll it back so the normal + // cleanup path can deterministically retry both renames. + if (fs::exists(to_delete) && fs::exists(snapshot) && !fs::exists(normal)) { + LogPrintf("[snapshot] rolling back interrupted background chainstate rename\n"); + try { + fs::rename(to_delete, normal); + DirectoryCommit(data_dir); + } catch (const fs::filesystem_error& e) { + error = strprintf(_("Failed to recover interrupted snapshot cleanup: %s"), e.what()); + return false; + } + return true; } - LOCK(cs_main); + // Both validated-cleanup renames completed. The NORMAL coins directory now + // contains the snapshot tip, while lifecycle markers still describe the dual + // state. Match those durable facts before promoting the markers. + if (fs::exists(normal) && !fs::exists(snapshot) && has_metadata) { + uint256 coins_tip; + try { + CCoinsViewDB coins_db{normal, /*nCacheSize=*/1 << 20, /*fMemory=*/false, /*fWipe=*/false}; + coins_tip = coins_db.GetBestBlock(); + } catch (const std::exception& e) { + error = strprintf(_("Failed to inspect interrupted snapshot cleanup: %s"), e.what()); + return false; + } + // Caution: this condition can also match a crash between an + // invalid-snapshot rename and its marker discard when validation + // failed with the background tip AT the base block (HASH_MISMATCH / + // EVO_STATE_MISMATCH): the background coins tip and the SNAPSHOT + // marker both hold the base hash. Taking the promote branch there is + // deliberate and equivalent: PromoteSnapshotMarkers rewrites the + // NORMAL best-block with the value it already has and erases the same + // marker set DiscardSnapshotMarkers would. Keep the two functions' + // side effects equivalent under that overlap, or disambiguate here. + if (has_snapshot_tip && coins_tip == snapshot_tip) { + LogPrintf("[snapshot] completing interrupted snapshot marker promotion\n"); + if (!evodb.PromoteSnapshotMarkers(snapshot_tip)) { + error = _("Failed to finish interrupted snapshot marker promotion."); + return false; + } + if (fs::exists(to_delete)) { + try { + fs::remove_all(to_delete); + DirectoryCommit(data_dir); + } catch (const fs::filesystem_error& e) { + LogPrintf("[snapshot] unable to remove recovered background chainstate directory: %s\n", e.what()); + } + } + return true; + } - evodb.reset(); - // TODO: pass DbWrapperParams as options instead multiple params - evodb = std::make_unique(util::DbWrapperParams{.path = options.data_dir, .memory = options.dash_dbs_in_memory, .wipe = to_wipe_data}); + // Invalid snapshot rename completed, but the synced marker discard did + // not. A stale _INVALID directory cannot mask a completed valid swap: + // the coins/snapshot-tip match above always takes precedence. + if (fs::exists(invalid)) { + LogPrintf("[snapshot] completing interrupted invalid-snapshot marker cleanup\n"); + if (!evodb.DiscardSnapshotMarkers()) { + error = _("Failed to finish interrupted invalid snapshot cleanup."); + return false; + } + return true; + } - dmnman.reset(); - dmnman = std::make_unique(*evodb, *options.mn_metaman); + error = has_snapshot_tip + ? _("Interrupted snapshot cleanup has inconsistent coins and EvoDB tips.") + : _("Interrupted snapshot cleanup is missing its snapshot EvoDB marker."); + return false; + } - chainman.m_total_coinstip_cache = cache_sizes.coins; - chainman.m_total_coinsdb_cache = cache_sizes.coins_db; + // Marker promotion can become durable before deletion of the old background + // directory. Treat that retry as success and finish the nonessential deletion. + if (!has_metadata && fs::exists(normal) && !fs::exists(snapshot) && fs::exists(to_delete)) { + try { + fs::remove_all(to_delete); + DirectoryCommit(data_dir); + } catch (const fs::filesystem_error& e) { + LogPrintf("[snapshot] unable to remove promoted background chainstate directory: %s\n", e.what()); + } + } - // Load the fully validated chainstate. - chainman.InitializeChainstate(options.mempool, *evodb, chain_helper); + // No metadata is otherwise the already-promoted/no-snapshot state and needs no work. + return true; +} - // Wiping the shared EvoDB above erased the SNAPSHOT best-block marker that - // ActivateExistingSnapshot() requires, so a persisted snapshot chainstate can - // no longer be revived. Discard it here rather than letting startup fail with - // advice ("reindex") the user has just followed, which would never recover. - if (to_wipe_data && !DeleteSnapshotChainstateFromDisk()) { - return {ChainstateLoadStatus::FAILURE, - _("Failed to remove the snapshot chainstate directory. Remove it manually before restarting.")}; - } +// Complete initialization of chainstates after the initial call has been made +// to ChainstateManager::InitializeChainstate(). +static ChainstateLoadResult CompleteChainstateInitialization(ChainstateManager& chainman, const CacheSizes& cache_sizes, + const ChainstateLoadOptions& options, CEvoDB& evodb, + std::unique_ptr& dmnman, + std::unique_ptr& llmq_ctx, + std::unique_ptr& chain_helper) + EXCLUSIVE_LOCKS_REQUIRED(::cs_main) +{ + const bool to_wipe_data = options.reindex || options.reindex_chainstate; - // Load a chain created from a UTXO snapshot, if any exist. - bilingual_str snapshot_error; - if (!chainman.DetectSnapshotChainstate(options.mempool, snapshot_error)) { - return {ChainstateLoadStatus::FAILURE, snapshot_error}; - } + dmnman.reset(); + dmnman = std::make_unique(evodb, *options.mn_metaman); auto& pblocktree{chainman.m_blockman.m_block_tree_db}; // new CBlockTreeDB tries to delete the existing file, which @@ -103,7 +156,7 @@ ChainstateLoadResult LoadChainstate(ChainstateManager& chainman, const CacheSize // Initialize llmq_ctx and connection to mempool llmq_ctx.reset(); - llmq_ctx = std::make_unique(*dmnman, *evodb, *options.sporkman, chainman, + llmq_ctx = std::make_unique(*dmnman, evodb, *options.sporkman, chainman, util::DbWrapperParams{.path = options.data_dir, .memory = options.dash_dbs_in_memory, .wipe = to_wipe_data}, options.bls_threads, options.worker_count, options.max_recsigs_age); if (options.mempool) { @@ -112,7 +165,7 @@ ChainstateLoadResult LoadChainstate(ChainstateManager& chainman, const CacheSize // Initialize chain_helper chain_helper.reset(); - chain_helper = std::make_unique(*evodb, *dmnman, *options.mn_sync, *(llmq_ctx->isman), *(llmq_ctx->quorum_block_processor), + chain_helper = std::make_unique(evodb, *dmnman, *options.mn_sync, *(llmq_ctx->isman), *(llmq_ctx->quorum_block_processor), *(llmq_ctx->qsnapman), chainman, chainman.GetConsensus(), *options.chainlocks, *(llmq_ctx->qman)); @@ -161,6 +214,13 @@ ChainstateLoadResult LoadChainstate(ChainstateManager& chainman, const CacheSize return {ChainstateLoadStatus::FAILURE, _("Error initializing block database")}; } + auto is_coinsview_empty = [&](Chainstate* chainstate) EXCLUSIVE_LOCKS_REQUIRED(::cs_main) { + return options.reindex || options.reindex_chainstate || chainstate->CoinsTip().GetBestBlock().IsNull(); + }; + + assert(chainman.m_total_coinstip_cache > 0); + assert(chainman.m_total_coinsdb_cache > 0); + // Conservative value which is arbitrarily chosen, as it will ultimately be changed // by a call to `chainman.MaybeRebalanceCaches()`. We just need to make sure // that the sum of the two caches (40%) does not exceed the allowable amount @@ -208,7 +268,7 @@ ChainstateLoadResult LoadChainstate(ChainstateManager& chainman, const CacheSize // immediately, so leaving a non-active identity's EvoDB writes in the // in-memory overlay would let a crash strand the coins DB ahead of // that identity's best-block marker. - if (!evodb->CommitRootTransaction(chainstate->EvoDbIdentity())) { + if (!evodb.CommitRootTransaction(chainstate->EvoDbIdentity())) { return {ChainstateLoadStatus::FAILURE, _("Failed to commit Evo database")}; } @@ -238,6 +298,132 @@ ChainstateLoadResult LoadChainstate(ChainstateManager& chainman, const CacheSize return {ChainstateLoadStatus::SUCCESS, {}}; } +ChainstateLoadResult LoadChainstate(ChainstateManager& chainman, const CacheSizes& cache_sizes, + const ChainstateLoadOptions& options, std::unique_ptr& evodb, + std::unique_ptr& dmnman, std::unique_ptr& llmq_ctx, + std::unique_ptr& chain_helper) +{ + assert(options.mn_metaman); + assert(options.sporkman); + assert(options.chainlocks); + assert(options.mn_sync); + + if (!hashAssumeValid.IsNull()) { + LogPrintf("Assuming ancestors of block %s have valid signatures.\n", hashAssumeValid.GetHex()); + } else { + LogPrintf("Validating signatures for all blocks.\n"); + } + LogPrintf("Setting nMinimumChainWork=%s\n", nMinimumChainWork.GetHex()); + if (nMinimumChainWork < UintToArith256(chainman.GetConsensus().nMinimumChainWork)) { + LogPrintf("Warning: nMinimumChainWork set below default value of %s\n", chainman.GetConsensus().nMinimumChainWork.GetHex()); + } + if (nPruneTarget == std::numeric_limits::max()) { + LogPrintf("Block pruning enabled. Use RPC call pruneblockchain(height) to manually prune block and undo files.\n"); + } else if (nPruneTarget) { + LogPrintf("Prune configured to target %u MiB on disk for block and undo files.\n", nPruneTarget / 1024 / 1024); + } + + LOCK(cs_main); + + evodb.reset(); + // TODO: pass DbWrapperParams as options instead multiple params + evodb = std::make_unique(util::DbWrapperParams{ + .path = options.data_dir, + .memory = options.dash_dbs_in_memory, + .wipe = options.reindex || options.reindex_chainstate}); + if (!options.dash_dbs_in_memory && !options.reindex && !options.reindex_chainstate) { + bilingual_str recovery_error; + if (!RecoverSnapshotCleanup(*evodb, options.data_dir, recovery_error)) { + return {ChainstateLoadStatus::FAILURE, recovery_error}; + } + } + chainman.m_total_coinstip_cache = cache_sizes.coins; + chainman.m_total_coinsdb_cache = cache_sizes.coins_db; + + // Load the fully validated chainstate. + chainman.InitializeChainstate(options.mempool, *evodb, chain_helper); + + // Wiping the shared EvoDB above erased the SNAPSHOT best-block marker that + // ActivateExistingSnapshot() requires, so a persisted snapshot chainstate can + // no longer be revived. Discard it here rather than letting startup fail with + // advice ("reindex") the user has just followed, which would never recover. + if ((options.reindex || options.reindex_chainstate) && !DeleteSnapshotChainstateFromDisk()) { + return {ChainstateLoadStatus::FAILURE, + _("Failed to remove the snapshot chainstate directory. Remove it manually before restarting.")}; + } + + // Load a chain created from a UTXO snapshot, if any exist. + bilingual_str snapshot_error; + if (!chainman.DetectSnapshotChainstate(options.mempool, snapshot_error)) { + return {ChainstateLoadStatus::FAILURE, snapshot_error}; + } + + auto [init_status, init_error] = CompleteChainstateInitialization(chainman, cache_sizes, options, *evodb, dmnman, + llmq_ctx, chain_helper); + if (init_status != ChainstateLoadStatus::SUCCESS) { + return {init_status, init_error}; + } + + // If a snapshot chainstate was fully validated by a background chainstate during + // the last run, detect it here and clean up the now-unneeded background + // chainstate. + // + // Why is this cleanup done here (on subsequent restart) and not just when the + // snapshot is actually validated? Because this entails unusual + // filesystem operations to move leveldb data directories around, and that seems + // too risky to do in the middle of normal runtime. + const auto snapshot_completion = chainman.MaybeCompleteSnapshotValidation(); + + if (snapshot_completion == SnapshotCompletionResult::SKIPPED) { + // Do nothing; expected case. + } else if (snapshot_completion == SnapshotCompletionResult::SUCCESS) { + LogPrintf("[snapshot] cleaning up unneeded background chainstate, then reinitializing\n"); + // The mempool holds raw pointers to dmnman and llmq_ctx->isman, so it has to + // let go of them before either manager is destroyed. + if (options.mempool) { + options.mempool->DisconnectManagers(); + } + chain_helper.reset(); + llmq_ctx.reset(); + dmnman.reset(); + if (!chainman.ValidatedSnapshotCleanup()) { + return {ChainstateLoadStatus::FAILURE_FATAL, Untranslated("Background chainstate cleanup failed unexpectedly.")}; + } + + // Because ValidatedSnapshotCleanup() has torn down chainstates with + // ChainstateManager::ResetChainstates(), reinitialize them here without + // duplicating the blockindex work above. + assert(chainman.GetAll().empty()); + assert(!chainman.IsSnapshotActive()); + assert(!chainman.IsSnapshotValidated()); + + chainman.InitializeChainstate(options.mempool, *evodb, chain_helper); + + // A reload of the block index is required to recompute setBlockIndexCandidates + // for the fully validated chainstate. + chainman.ActiveChainstate().ClearBlockIndexCandidates(); + + std::tie(init_status, init_error) = CompleteChainstateInitialization(chainman, cache_sizes, options, *evodb, + dmnman, llmq_ctx, chain_helper); + if (init_status != ChainstateLoadStatus::SUCCESS) { + return {init_status, init_error}; + } + } else { + // The UTXO hashing inside completion takes minutes and aborts with + // STATS_FAILED when shutdown is requested mid-way. That says nothing + // about the snapshot, so report the interruption instead of telling + // the user to discard a perfectly good snapshot. + if (options.check_interrupt && options.check_interrupt()) { + return {ChainstateLoadStatus::INTERRUPTED, {}}; + } + return {ChainstateLoadStatus::FAILURE, _( + "UTXO snapshot failed to validate. " + "Restart to resume normal initial block download, or try loading a different snapshot.")}; + } + + return {ChainstateLoadStatus::SUCCESS, {}}; +} + ChainstateLoadResult VerifyLoadedChainstate(ChainstateManager& chainman, const ChainstateLoadOptions& options, CEvoDB& evodb, std::function notify_bls_state) { diff --git a/src/node/chainstate.h b/src/node/chainstate.h index 3f66cddd2ddd..98c0ea9f7511 100644 --- a/src/node/chainstate.h +++ b/src/node/chainstate.h @@ -59,7 +59,13 @@ struct ChainstateLoadOptions { //! case, and treat other cases as errors. More complex applications may want to //! try reindexing in the generic failure case, and pass an interrupt callback //! and exit cleanly in the interrupted case. -enum class ChainstateLoadStatus { SUCCESS, FAILURE, FAILURE_INCOMPATIBLE_DB, INTERRUPTED }; +enum class ChainstateLoadStatus { + SUCCESS, + FAILURE, //!< Generic failure which reindexing may fix + FAILURE_FATAL, //!< Fatal error which should not prompt to reindex + FAILURE_INCOMPATIBLE_DB, + INTERRUPTED, +}; //! Chainstate load status code and optional error string. using ChainstateLoadResult = std::tuple; diff --git a/src/node/utxo_snapshot.cpp b/src/node/utxo_snapshot.cpp index 9aadf1f237b9..9a9e4332ea83 100644 --- a/src/node/utxo_snapshot.cpp +++ b/src/node/utxo_snapshot.cpp @@ -8,6 +8,7 @@ #include #include #include +#include #include #include @@ -34,11 +35,18 @@ bool WriteSnapshotBaseBlockhash(Chainstate& snapshot_chainstate) } afile << *snapshot_chainstate.m_from_snapshot_blockhash; + if (!FileCommit(afile.Get())) { + LogPrintf("[snapshot] failed to sync base blockhash file %s\n", + fs::PathToString(write_to)); + return false; + } + if (afile.fclose() != 0) { LogPrintf("[snapshot] failed to close base blockhash file %s after writing\n", fs::PathToString(write_to)); return false; } + DirectoryCommit(*chaindir); return true; } diff --git a/src/test/blockmanager_tests.cpp b/src/test/blockmanager_tests.cpp index da5fee7c1876..f70cbc2c2ba3 100644 --- a/src/test/blockmanager_tests.cpp +++ b/src/test/blockmanager_tests.cpp @@ -26,22 +26,21 @@ BOOST_AUTO_TEST_CASE(blockmanager_find_block_pos) .chainparams = *params, }; BlockManager blockman{blockman_opts}; - CChain chain {}; // simulate adding a genesis block normally - BOOST_CHECK_EQUAL(blockman.SaveBlockToDisk(params->GenesisBlock(), 0, chain, nullptr).nPos, BLOCK_SERIALIZATION_HEADER_SIZE); + BOOST_CHECK_EQUAL(blockman.SaveBlockToDisk(params->GenesisBlock(), 0, nullptr).nPos, BLOCK_SERIALIZATION_HEADER_SIZE); // simulate what happens during reindex // simulate a well-formed genesis block being found at offset 8 in the blk00000.dat file // the block is found at offset 8 because there is an 8 byte serialization header // consisting of 4 magic bytes + 4 length bytes before each block in a well-formed blk file. FlatFilePos pos{0, BLOCK_SERIALIZATION_HEADER_SIZE}; - BOOST_CHECK_EQUAL(blockman.SaveBlockToDisk(params->GenesisBlock(), 0, chain, &pos).nPos, BLOCK_SERIALIZATION_HEADER_SIZE); + BOOST_CHECK_EQUAL(blockman.SaveBlockToDisk(params->GenesisBlock(), 0, &pos).nPos, BLOCK_SERIALIZATION_HEADER_SIZE); // now simulate what happens after reindex for the first new block processed // the actual block contents don't matter, just that it's a block. // verify that the write position is at offset 0x12d. // this is a check to make sure that https://github.com/bitcoin/bitcoin/issues/21379 does not recur // 8 bytes (for serialization header) + 285 (for serialized genesis block) = 293 // add another 8 bytes for the second block's serialization header and we get 293 + 8 = 301 - FlatFilePos actual{blockman.SaveBlockToDisk(params->GenesisBlock(), 1, chain, nullptr)}; + FlatFilePos actual{blockman.SaveBlockToDisk(params->GenesisBlock(), 1, nullptr)}; BOOST_CHECK_EQUAL(actual.nPos, BLOCK_SERIALIZATION_HEADER_SIZE + ::GetSerializeSize(params->GenesisBlock(), CLIENT_VERSION) + BLOCK_SERIALIZATION_HEADER_SIZE); } diff --git a/src/test/coinstatsindex_tests.cpp b/src/test/coinstatsindex_tests.cpp index cdf24f975f38..de9a01bb1a91 100644 --- a/src/test/coinstatsindex_tests.cpp +++ b/src/test/coinstatsindex_tests.cpp @@ -97,7 +97,7 @@ BOOST_FIXTURE_TEST_CASE(coinstatsindex_unclean_shutdown, TestChain100Setup) LOCK(cs_main); BlockValidationState state; BOOST_CHECK(CheckBlock(block, state, params.GetConsensus())); - BOOST_CHECK(chainstate.AcceptBlock(new_block, state, &new_block_index, true, nullptr, nullptr)); + BOOST_CHECK(m_node.chainman->AcceptBlock(new_block, state, &new_block_index, true, nullptr, nullptr)); CCoinsViewCache view(&chainstate.CoinsTip()); BOOST_CHECK(chainstate.ConnectBlock(block, state, new_block_index, view)); } diff --git a/src/test/evo_db_tests.cpp b/src/test/evo_db_tests.cpp index 4d8ffa14821d..617c47869f32 100644 --- a/src/test/evo_db_tests.cpp +++ b/src/test/evo_db_tests.cpp @@ -238,6 +238,7 @@ BOOST_AUTO_TEST_CASE(snapshot_markers_can_be_discarded) WriteMarker(db, EvoDbIdentity::SNAPSHOT, BlockHash(40)); { auto tx = db.BeginTransaction(EvoDbIdentity::SNAPSHOT); + db.WriteSnapshotBaseMNListHash(BlockHash(4)); db.WriteDualChainstateMarker(); tx->Commit(); } @@ -258,7 +259,52 @@ BOOST_AUTO_TEST_CASE(snapshot_markers_can_be_discarded) CEvoDB reopened{util::DbWrapperParams{.path = path, .memory = false, .wipe = false}}; uint256 hash; BOOST_CHECK(!reopened.ReadBestBlock(EvoDbIdentity::SNAPSHOT, hash)); + BOOST_CHECK(!reopened.ReadSnapshotBaseMNListHash(hash)); BOOST_CHECK(!reopened.HasDualChainstateMarker()); } +BOOST_AUTO_TEST_CASE(snapshot_marker_promotion_and_discard) +{ + CEvoDB db{util::DbWrapperParams{.path = m_args.GetDataDirBase() / "evodb_promotion", .memory = true, .wipe = true}}; + const uint256 normal_tip = BlockHash(30); + const uint256 snapshot_tip = BlockHash(300); + const uint256 mn_list_hash = BlockHash(3); + + WriteMarker(db, EvoDbIdentity::NORMAL, normal_tip); + { + auto tx = db.BeginTransaction(EvoDbIdentity::SNAPSHOT); + db.WriteBestBlock(EvoDbIdentity::SNAPSHOT, snapshot_tip); + db.WriteSnapshotBaseMNListHash(mn_list_hash); + db.WriteDualChainstateMarker(); + tx->Commit(); + } + BOOST_REQUIRE(db.CommitRootTransaction(EvoDbIdentity::NORMAL)); + BOOST_REQUIRE(db.CommitRootTransaction(EvoDbIdentity::SNAPSHOT)); + + BOOST_REQUIRE(db.PromoteSnapshotMarkers(snapshot_tip)); + // Promotion is idempotent across a restart after the synced batch lands. + BOOST_REQUIRE(db.PromoteSnapshotMarkers(snapshot_tip)); + BOOST_CHECK(db.VerifyBestBlock(EvoDbIdentity::NORMAL, snapshot_tip)); + uint256 value; + BOOST_CHECK(!db.ReadBestBlock(EvoDbIdentity::SNAPSHOT, value)); + BOOST_CHECK(!db.ReadSnapshotBaseMNListHash(value)); + BOOST_CHECK(!db.HasDualChainstateMarker()); + + WriteMarker(db, EvoDbIdentity::SNAPSHOT, BlockHash(301)); + { + auto tx = db.BeginTransaction(EvoDbIdentity::SNAPSHOT); + db.WriteSnapshotBaseMNListHash(BlockHash(4)); + db.WriteDualChainstateMarker(); + tx->Commit(); + } + BOOST_REQUIRE(db.CommitRootTransaction(EvoDbIdentity::SNAPSHOT)); + BOOST_REQUIRE(db.DiscardSnapshotMarkers()); + // Discard is likewise safe to retry. + BOOST_REQUIRE(db.DiscardSnapshotMarkers()); + BOOST_CHECK(db.VerifyBestBlock(EvoDbIdentity::NORMAL, snapshot_tip)); + BOOST_CHECK(!db.ReadBestBlock(EvoDbIdentity::SNAPSHOT, value)); + BOOST_CHECK(!db.ReadSnapshotBaseMNListHash(value)); + BOOST_CHECK(!db.HasDualChainstateMarker()); +} + BOOST_AUTO_TEST_SUITE_END() diff --git a/src/test/fuzz/load_external_block_file.cpp b/src/test/fuzz/load_external_block_file.cpp index a2f8694ee618..ca7ae63db7d8 100644 --- a/src/test/fuzz/load_external_block_file.cpp +++ b/src/test/fuzz/load_external_block_file.cpp @@ -35,9 +35,9 @@ FUZZ_TARGET(load_external_block_file, .init = initialize_load_external_block_fil // Corresponds to the -reindex case (track orphan blocks across files). FlatFilePos flat_file_pos; std::multimap blocks_with_unknown_parent; - g_setup->m_node.chainman->ActiveChainstate().LoadExternalBlockFile(fuzzed_block_file, &flat_file_pos, &blocks_with_unknown_parent); + g_setup->m_node.chainman->LoadExternalBlockFile(fuzzed_block_file, &flat_file_pos, &blocks_with_unknown_parent); } else { // Corresponds to the -loadblock= case (orphan blocks aren't tracked across files). - g_setup->m_node.chainman->ActiveChainstate().LoadExternalBlockFile(fuzzed_block_file); + g_setup->m_node.chainman->LoadExternalBlockFile(fuzzed_block_file); } } diff --git a/src/test/util/chainstate.h b/src/test/util/chainstate.h index c5ccd666a4ba..6b589b597243 100644 --- a/src/test/util/chainstate.h +++ b/src/test/util/chainstate.h @@ -71,6 +71,7 @@ CreateAndActivateUTXOSnapshot( // This is a stripped-down version of node::LoadChainstate which // preserves the block index. LOCK(::cs_main); + CBlockIndex *orig_tip = node.chainman->ActiveChainstate().m_chain.Tip(); uint256 gen_hash = node.chainman->ActiveChainstate().m_chain[0]->GetBlockHash(); node.chainman->ResetChainstates(); node.chainman->InitializeChainstate( @@ -84,6 +85,22 @@ CreateAndActivateUTXOSnapshot( chain.setBlockIndexCandidates.insert(node.chainman->m_blockman.LookupBlockIndex(gen_hash)); chain.LoadChainTip(); node.chainman->MaybeRebalanceCaches(); + + // Reset the HAVE_DATA flags below the snapshot height, simulating + // never-having-downloaded them in the first place. + // TODO: perhaps we could improve this by using pruning to delete + // these blocks instead + CBlockIndex *pindex = orig_tip; + while (pindex && pindex != chain.m_chain.Tip()) { + pindex->nStatus &= ~BLOCK_HAVE_DATA; + pindex->nStatus &= ~BLOCK_HAVE_UNDO; + // We have to set the ASSUMED_VALID flag, because otherwise it + // would not be possible to have a block index entry without HAVE_DATA + // and with nTx > 0 (since we aren't setting the pruned flag); + // see CheckBlockIndex(). + pindex->nStatus |= BLOCK_ASSUMED_VALID; + pindex = pindex->pprev; + } } BlockValidationState state; if (!node.chainman->ActiveChainstate().ActivateBestChain(state)) { diff --git a/src/test/validation_block_tests.cpp b/src/test/validation_block_tests.cpp index 046ea0589777..555c87c09b96 100644 --- a/src/test/validation_block_tests.cpp +++ b/src/test/validation_block_tests.cpp @@ -227,7 +227,7 @@ BOOST_AUTO_TEST_CASE(checkblock_accept_known_hash) BlockValidationState state; CBlockIndex* pindex = nullptr; bool newblock = false; - BOOST_REQUIRE(m_node.chainman->ActiveChainstate().AcceptBlock( + BOOST_REQUIRE(m_node.chainman->AcceptBlock( good, state, &pindex, /*fRequested=*/true, /*dbp=*/nullptr, &newblock, &hash)); BOOST_REQUIRE(state.IsValid()); diff --git a/src/test/validation_chainstate_tests.cpp b/src/test/validation_chainstate_tests.cpp index b88eb377e26b..51a8e690877a 100644 --- a/src/test/validation_chainstate_tests.cpp +++ b/src/test/validation_chainstate_tests.cpp @@ -6,7 +6,9 @@ #include #include #include +#include #include +#include #include #include #include @@ -16,11 +18,31 @@ #include #include #include +#include #include +#include #include +namespace { + +class TipEventCounter final : public CValidationInterface +{ +public: + int block_connected{0}; + int updated_tip{0}; + int mn_list_changed{0}; + int chainstate_flushed{0}; + + void BlockConnected(const std::shared_ptr&, const CBlockIndex*) override { ++block_connected; } + void UpdatedBlockTip(const CBlockIndex*, const CBlockIndex*, bool) override { ++updated_tip; } + void NotifyMasternodeListChanged(bool, const CDeterministicMNList&, const CDeterministicMNListDiff&) override { ++mn_list_changed; } + void ChainStateFlushed(const CBlockLocator&) override { ++chainstate_flushed; } +}; + +} // namespace + BOOST_FIXTURE_TEST_SUITE(validation_chainstate_tests, ChainTestingSetup) //! Test resizing coins-related Chainstate caches during runtime. @@ -80,6 +102,13 @@ BOOST_FIXTURE_TEST_CASE(chainstate_update_tip, TestChain100Setup) // After adding some blocks to the tip, best block should have changed. BOOST_CHECK(::g_best_block != curr_tip); + // Grab block 1 from disk; we'll add it to the background chain later. + std::shared_ptr pblockone = std::make_shared(); + { + LOCK(::cs_main); + BOOST_REQUIRE(node::ReadBlockFromDisk(*pblockone, chainman.ActiveChain()[1], Params().GetConsensus())); + } + BOOST_REQUIRE(CreateAndActivateUTXOSnapshot( this, NoMalleation, /*reset_chainstate=*/ true)); @@ -107,11 +136,7 @@ BOOST_FIXTURE_TEST_CASE(chainstate_update_tip, TestChain100Setup) assert(false); }()}; - // Create a block to append to the validation chain. - std::vector noTxns; - CScript scriptPubKey = CScript() << ToByteVector(coinbaseKey.GetPubKey()) << OP_CHECKSIG; - CBlock validation_block = this->CreateBlock(noTxns, scriptPubKey, background_cs); - auto pblock = std::make_shared(validation_block); + // Append the first block to the background chain. BlockValidationState state; CBlockIndex* pindex = nullptr; const CChainParams& chainparams = Params(); @@ -121,22 +146,39 @@ BOOST_FIXTURE_TEST_CASE(chainstate_update_tip, TestChain100Setup) // once it is changed to support multiple chainstates. { LOCK(::cs_main); - bool checked = CheckBlock(*pblock, state, chainparams.GetConsensus()); + bool checked = CheckBlock(*pblockone, state, chainparams.GetConsensus()); BOOST_CHECK(checked); - bool accepted = background_cs.AcceptBlock( - pblock, state, &pindex, true, nullptr, &newblock); + bool accepted = chainman.AcceptBlock( + pblockone, state, &pindex, true, nullptr, &newblock); BOOST_CHECK(accepted); } - // UpdateTip is called here - bool block_added = background_cs.ActivateBestChain(state, pblock); + + SyncWithValidationInterfaceQueue(); + TipEventCounter event_counter; + RegisterValidationInterface(&event_counter); + int ui_mn_list_changed{0}; + auto ui_connection = uiInterface.NotifyMasternodeListChanged_connect( + [&](const CDeterministicMNList&, const CBlockIndex*) { ++ui_mn_list_changed; }); + + // UpdateTip is called here. + bool block_added = background_cs.ActivateBestChain(state, pblockone); + WITH_LOCK(::cs_main, background_cs.ForceFlushStateToDisk()); + SyncWithValidationInterfaceQueue(); + ui_connection.disconnect(); + UnregisterValidationInterface(&event_counter); // Ensure tip is as expected - BOOST_CHECK_EQUAL(background_cs.m_chain.Tip()->GetBlockHash(), validation_block.GetHash()); + BOOST_CHECK_EQUAL(background_cs.m_chain.Tip()->GetBlockHash(), pblockone->GetHash()); // g_best_block should be unchanged after adding a block to the background // validation chain. BOOST_CHECK(block_added); BOOST_CHECK_EQUAL(curr_tip, ::g_best_block); + BOOST_CHECK_EQUAL(event_counter.block_connected, 0); + BOOST_CHECK_EQUAL(event_counter.updated_tip, 0); + BOOST_CHECK_EQUAL(event_counter.mn_list_changed, 0); + BOOST_CHECK_EQUAL(event_counter.chainstate_flushed, 0); + BOOST_CHECK_EQUAL(ui_mn_list_changed, 0); } //! A chain whose V19 activation sits above the assumeutxo height, so the @@ -172,6 +214,14 @@ BOOST_FIXTURE_TEST_CASE(chainstate_connectblock_bls_scheme, V19AboveSnapshotSetu mineBlocks(9); BOOST_REQUIRE(CreateAndActivateUTXOSnapshot(this, NoMalleation, /*reset_chainstate=*/true)); BOOST_REQUIRE(WITH_LOCK(::cs_main, return chainman.IsSnapshotActive())); + + // The background chainstate was reset to genesis before activation, so + // the base MN list was not derivable and no lifecycle marker may have + // been captured: deriving one would fabricate an empty list and poison + // the shared list cache for the background chainstate's later + // re-validation of the base region. + uint256 stale_hash; + BOOST_CHECK(!m_node.evodb->ReadSnapshotBaseMNListHash(stale_hash)); mineBlocks(V19_HEIGHT - WITH_LOCK(::cs_main, return chainman.ActiveHeight())); BOOST_REQUIRE(!bls::bls_legacy_scheme.load()); @@ -192,7 +242,7 @@ BOOST_FIXTURE_TEST_CASE(chainstate_connectblock_bls_scheme, V19AboveSnapshotSetu CBlockIndex* pindex = nullptr; bool newblock = false; BOOST_REQUIRE(CheckBlock(*pblock, state, Params().GetConsensus())); - BOOST_REQUIRE(background_cs.AcceptBlock(pblock, state, &pindex, true, nullptr, &newblock)); + BOOST_REQUIRE(m_node.chainman->AcceptBlock(pblock, state, &pindex, true, nullptr, &newblock)); } BOOST_REQUIRE(background_cs.ActivateBestChain(state, pblock)); diff --git a/src/test/validation_chainstatemanager_tests.cpp b/src/test/validation_chainstatemanager_tests.cpp index 111c15cb43f8..33af548a6510 100644 --- a/src/test/validation_chainstatemanager_tests.cpp +++ b/src/test/validation_chainstatemanager_tests.cpp @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -28,33 +29,17 @@ #include #include #include -#include #include #include -#include #include using node::SnapshotMetadata; namespace { -class TipEventCounter final : public CValidationInterface -{ -public: - int block_connected{0}; - int updated_tip{0}; - int mn_list_changed{0}; - int chainstate_flushed{0}; - - void BlockConnected(const std::shared_ptr&, const CBlockIndex*) override { ++block_connected; } - void UpdatedBlockTip(const CBlockIndex*, const CBlockIndex*, bool) override { ++updated_tip; } - void NotifyMasternodeListChanged(bool, const CDeterministicMNList&, const CDeterministicMNListDiff&) override { ++mn_list_changed; } - void ChainStateFlushed(const CBlockLocator&) override { ++chainstate_flushed; } -}; - void SeedSnapshotMarker(CEvoDB& evodb, const uint256& hash) { auto tx = evodb.BeginTransaction(EvoDbIdentity::SNAPSHOT); @@ -118,7 +103,10 @@ BOOST_AUTO_TEST_CASE(chainstatemanager) WITH_LOCK(::cs_main, c1.InitCoinsCache(1 << 23)); DashChainstateSetup(manager, m_node, /*llmq_dbs_in_memory=*/true, /*llmq_dbs_wipe=*/false); + BOOST_REQUIRE(c1.LoadGenesisBlock()); + BlockValidationState val_state; + BOOST_CHECK(c1.ActivateBestChain(val_state, nullptr)); BOOST_CHECK(!manager.IsSnapshotActive()); BOOST_CHECK(!manager.IsSnapshotActiveAndUnvalidated()); @@ -130,7 +118,7 @@ BOOST_AUTO_TEST_CASE(chainstatemanager) auto& active_chain = WITH_LOCK(manager.GetMutex(), return manager.ActiveChain()); BOOST_CHECK_EQUAL(&active_chain, &c1.m_chain); - BOOST_CHECK_EQUAL(WITH_LOCK(manager.GetMutex(), return manager.ActiveHeight()), -1); + BOOST_CHECK_EQUAL(WITH_LOCK(manager.GetMutex(), return manager.ActiveHeight()), 0); auto active_tip = WITH_LOCK(manager.GetMutex(), return manager.ActiveTip()); auto exp_tip = c1.m_chain.Tip(); @@ -142,7 +130,7 @@ BOOST_AUTO_TEST_CASE(chainstatemanager) // Create a snapshot-based chainstate. // - const uint256 snapshot_blockhash = GetRandHash(); + const uint256 snapshot_blockhash = active_tip->GetBlockHash(); SeedSnapshotMarker(evodb, snapshot_blockhash); Chainstate* c2_ptr = WITH_LOCK(::cs_main, return manager.ActivateExistingSnapshot( &mempool, snapshot_blockhash)); @@ -150,6 +138,10 @@ BOOST_AUTO_TEST_CASE(chainstatemanager) Chainstate& c2 = *c2_ptr; chainstates.push_back(&c2); + // Only the active chainstate keeps the mempool. + BOOST_CHECK_EQUAL(c2.GetMempool(), &mempool); + BOOST_CHECK(!c1.GetMempool()); + DashChainstateSetup(manager, m_node, /*llmq_dbs_in_memory=*/true, /*llmq_dbs_wipe=*/false); BOOST_CHECK_EQUAL(manager.SnapshotBlockhash().value(), snapshot_blockhash); @@ -157,12 +149,9 @@ BOOST_AUTO_TEST_CASE(chainstatemanager) c2.InitCoinsDB( /*cache_size_bytes=*/1 << 23, /*in_memory=*/true, /*should_wipe=*/false); WITH_LOCK(::cs_main, c2.InitCoinsCache(1 << 23)); - // Give the snapshot chainstate its own genesis candidate and tip. - c2.LoadGenesisBlock(); - WITH_LOCK(::cs_main, c2.setBlockIndexCandidates.insert( - manager.m_blockman.LookupBlockIndex(Params().GenesisBlock().GetHash()))); - BlockValidationState dummy_state; - BOOST_CHECK(c2.ActivateBestChain(dummy_state, nullptr)); + c2.m_chain.SetTip(*active_tip); + BlockValidationState _; + BOOST_CHECK(c2.ActivateBestChain(_, nullptr)); BOOST_CHECK(manager.IsSnapshotActive()); BOOST_CHECK(WITH_LOCK(::cs_main, return !manager.IsSnapshotValidated())); @@ -182,31 +171,7 @@ BOOST_AUTO_TEST_CASE(chainstatemanager) auto exp_tip2 = c2.m_chain.Tip(); BOOST_CHECK_EQUAL(active_tip2, exp_tip2); - // Ensure that these pointers actually correspond to different - // CCoinsViewCache instances. - BOOST_CHECK(exp_tip != exp_tip2); - - // Connect genesis through the now-background chainstate. This exercises - // Dash special-transaction and quorum processing with a non-active caller, - // and background validation must not emit active-tip notifications. - SyncWithValidationInterfaceQueue(); - TipEventCounter event_counter; - RegisterValidationInterface(&event_counter); - int ui_mn_list_changed{0}; - auto ui_connection = uiInterface.NotifyMasternodeListChanged_connect( - [&](const CDeterministicMNList&, const CBlockIndex*) { ++ui_mn_list_changed; }); - BlockValidationState background_state; - BOOST_CHECK(c1.ActivateBestChain(background_state, nullptr)); - WITH_LOCK(::cs_main, c1.ForceFlushStateToDisk()); - SyncWithValidationInterfaceQueue(); - ui_connection.disconnect(); - UnregisterValidationInterface(&event_counter); - BOOST_CHECK_EQUAL(c1.m_chain.Tip(), WITH_LOCK(::cs_main, return manager.ActiveChain().Genesis())); - BOOST_CHECK_EQUAL(event_counter.block_connected, 0); - BOOST_CHECK_EQUAL(event_counter.updated_tip, 0); - BOOST_CHECK_EQUAL(event_counter.mn_list_changed, 0); - BOOST_CHECK_EQUAL(event_counter.chainstate_flushed, 0); - BOOST_CHECK_EQUAL(ui_mn_list_changed, 0); + BOOST_CHECK_EQUAL(exp_tip, exp_tip2); // Let scheduler events finish running to avoid accessing memory that is going to be unloaded SyncWithValidationInterfaceQueue(); @@ -217,7 +182,7 @@ BOOST_AUTO_TEST_CASE(chainstatemanager) } //! Test rebalancing the caches associated with each chainstate. -BOOST_AUTO_TEST_CASE(chainstatemanager_rebalance_caches) +BOOST_FIXTURE_TEST_CASE(chainstatemanager_rebalance_caches, TestChain100Setup) { ChainstateManager& manager = *m_node.chainman; CTxMemPool& mempool = *m_node.mempool; @@ -230,7 +195,7 @@ BOOST_AUTO_TEST_CASE(chainstatemanager_rebalance_caches) // Create a legacy (IBD) chainstate. // - Chainstate& c1 = WITH_LOCK(cs_main, return manager.InitializeChainstate(&mempool, evodb, m_node.chain_helper)); + Chainstate& c1 = manager.ActiveChainstate(); chainstates.push_back(&c1); c1.InitCoinsDB( /*cache_size_bytes=*/1 << 23, /*in_memory=*/true, /*should_wipe=*/false); @@ -238,8 +203,6 @@ BOOST_AUTO_TEST_CASE(chainstatemanager_rebalance_caches) { LOCK(::cs_main); c1.InitCoinsCache(1 << 23); - BOOST_REQUIRE(c1.LoadGenesisBlock()); - c1.CoinsTip().SetBestBlock(InsecureRand256()); manager.MaybeRebalanceCaches(); } @@ -248,9 +211,9 @@ BOOST_AUTO_TEST_CASE(chainstatemanager_rebalance_caches) // Create a snapshot-based chainstate. // - const uint256 snapshot_blockhash = GetRandHash(); - SeedSnapshotMarker(evodb, snapshot_blockhash); - Chainstate* c2_ptr = WITH_LOCK(cs_main, return manager.ActivateExistingSnapshot(&mempool, snapshot_blockhash)); + CBlockIndex* snapshot_base{WITH_LOCK(manager.GetMutex(), return manager.ActiveChain()[manager.ActiveChain().Height() / 2])}; + SeedSnapshotMarker(evodb, snapshot_base->GetBlockHash()); + Chainstate* c2_ptr = WITH_LOCK(cs_main, return manager.ActivateExistingSnapshot(&mempool, snapshot_base->GetBlockHash())); BOOST_REQUIRE(c2_ptr); Chainstate& c2 = *c2_ptr; chainstates.push_back(&c2); @@ -260,8 +223,6 @@ BOOST_AUTO_TEST_CASE(chainstatemanager_rebalance_caches) { LOCK(::cs_main); c2.InitCoinsCache(1 << 23); - BOOST_REQUIRE(c2.LoadGenesisBlock()); - c2.CoinsTip().SetBestBlock(InsecureRand256()); manager.MaybeRebalanceCaches(); } @@ -522,6 +483,55 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_activate_snapshot, SnapshotTestSetup) this->SetupSnapshot(); } +BOOST_FIXTURE_TEST_CASE(chainstatemanager_reconsider_block_candidates, SnapshotTestSetup) +{ + auto [background_chainstate, snapshot_chainstate] = this->SetupSnapshot(); + ChainstateManager& chainman = *Assert(m_node.chainman); + + // Build a valid fork block directly on the background tip. Since the + // background tip is the snapshot base, this block is not on the path the + // background chainstate is allowed to validate. + CScript script_pub_key = CScript() << ToByteVector(coinbaseKey.GetPubKey()) << OP_CHECKSIG; + CBlock fork_block = CreateBlock(/*txns=*/{}, script_pub_key, *background_chainstate); + BOOST_REQUIRE_EQUAL(fork_block.hashPrevBlock, WITH_LOCK(::cs_main, return background_chainstate->m_chain.Tip()->GetBlockHash())); + BOOST_REQUIRE(chainman.ProcessNewBlock(std::make_shared(fork_block), /*force_processing=*/true, /*new_block=*/nullptr)); + + CBlockIndex* fork_index{WITH_LOCK(::cs_main, return chainman.m_blockman.LookupBlockIndex(fork_block.GetHash()))}; + BOOST_REQUIRE(fork_index); + + { + LOCK(::cs_main); + const arith_uint256 original_work{fork_index->nChainWork}; + const CBlockIndex* snapshot_base{snapshot_chainstate->SnapshotBase()}; + BOOST_REQUIRE(snapshot_base); + BOOST_REQUIRE_EQUAL(background_chainstate->m_chain.Tip(), snapshot_base); + BOOST_REQUIRE(!snapshot_base->GetAncestor(fork_index->nHeight)); + + // Make the fork the highest-work block and simulate validation having + // marked it failed. ResetBlockFailureFlags used to add it only to the + // invoking background chainstate's candidate set. + fork_index->nChainWork = snapshot_chainstate->m_chain.Tip()->nChainWork + 1; + fork_index->nStatus |= BLOCK_FAILED_VALID; + chainman.m_failed_blocks.insert(fork_index); + background_chainstate->setBlockIndexCandidates.erase(fork_index); + snapshot_chainstate->setBlockIndexCandidates.erase(fork_index); + + background_chainstate->ResetBlockFailureFlags(fork_index); + + BOOST_CHECK(fork_index->IsValid()); + BOOST_CHECK_EQUAL(background_chainstate->setBlockIndexCandidates.count(fork_index), 0); + BOOST_CHECK_EQUAL(snapshot_chainstate->setBlockIndexCandidates.count(fork_index), 1); + for (const CBlockIndex* candidate : background_chainstate->setBlockIndexCandidates) { + BOOST_CHECK_EQUAL(snapshot_base->GetAncestor(candidate->nHeight), candidate); + } + + // Restore the synthetic work value so the shared block index remains + // internally consistent for fixture teardown. + snapshot_chainstate->setBlockIndexCandidates.erase(fork_index); + fork_index->nChainWork = original_work; + } +} + //! Test LoadBlockIndex behavior when multiple chainstates are in use. //! //! - First, verify that setBlockIndexCandidates is as expected when using a single, @@ -530,7 +540,7 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_activate_snapshot, SnapshotTestSetup) //! - Then mark a region of the chain BLOCK_ASSUMED_VALID and introduce a second chainstate //! that will tolerate assumed-valid blocks. Run LoadBlockIndex() and ensure that the first //! chainstate only contains fully validated blocks and the other chainstate contains all blocks, -//! even those assumed-valid. +//! except those marked assume-valid, because those entries don't HAVE_DATA. //! BOOST_FIXTURE_TEST_CASE(chainstatemanager_loadblockindex, TestChain100Setup) { @@ -545,28 +555,34 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_loadblockindex, TestChain100Setup) const int assumed_valid_start_idx = last_assumed_valid_idx - expected_assumed_valid; CBlockIndex* validated_tip{nullptr}; + CBlockIndex* assumed_base{nullptr}; CBlockIndex* assumed_tip{WITH_LOCK(chainman.GetMutex(), return chainman.ActiveChain().Tip())}; auto reload_all_block_indexes = [&]() { + // For completeness, we also reset the block sequence counters to + // ensure that no state which affects the ranking of tip-candidates is + // retained (even though this isn't strictly necessary). + WITH_LOCK(::cs_main, return chainman.ResetBlockSequenceCounters()); for (Chainstate* cs : chainman.GetAll()) { LOCK(::cs_main); - cs->UnloadBlockIndex(); + cs->ClearBlockIndexCandidates(); BOOST_CHECK(cs->setBlockIndexCandidates.empty()); } WITH_LOCK(::cs_main, chainman.LoadBlockIndex()); }; - // Ensure that without any assumed-valid BlockIndex entries, all entries are considered - // tip candidates. + // Ensure that without any assumed-valid BlockIndex entries, only the current tip is + // considered as a candidate. reload_all_block_indexes(); - BOOST_CHECK_EQUAL(cs1.setBlockIndexCandidates.size(), cs1.m_chain.Height() + 1); + BOOST_CHECK_EQUAL(cs1.setBlockIndexCandidates.size(), 1); - // Mark some region of the chain assumed-valid. + // Mark some region of the chain assumed-valid, and remove the HAVE_DATA flag. for (int i = 0; i <= cs1.m_chain.Height(); ++i) { LOCK(::cs_main); auto index = cs1.m_chain[i]; + // Blocks with heights in range [20, 40) are marked ASSUMED_VALID if (i < last_assumed_valid_idx && i >= assumed_valid_start_idx) { index->nStatus = BlockStatus::BLOCK_VALID_TREE | BlockStatus::BLOCK_ASSUMED_VALID; } @@ -579,37 +595,54 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_loadblockindex, TestChain100Setup) validated_tip = index; BOOST_CHECK(!index->IsAssumedValid()); } + // Note the last assumed valid block as the snapshot base + if (i == last_assumed_valid_idx - 1) { + assumed_base = index; + BOOST_CHECK(index->IsAssumedValid()); + } else if (i == last_assumed_valid_idx) { + BOOST_CHECK(!index->IsAssumedValid()); + } } BOOST_CHECK_EQUAL(expected_assumed_valid, num_assumed_valid); - const uint256 snapshot_blockhash = GetRandHash(); + const uint256 snapshot_blockhash = assumed_base->GetBlockHash(); SeedSnapshotMarker(*m_node.evodb, snapshot_blockhash); Chainstate* cs2_ptr = WITH_LOCK(::cs_main, return chainman.ActivateExistingSnapshot(&mempool, snapshot_blockhash)); BOOST_REQUIRE(cs2_ptr); Chainstate& cs2 = *cs2_ptr; + // Note: cs2's tip is not set when ActivateExistingSnapshot is called. + // Set tip of the fully validated chain to be the validated tip + cs1.m_chain.SetTip(*validated_tip); + + // Set tip of the assume-valid-based chain to the assume-valid block + cs2.m_chain.SetTip(*assumed_base); + reload_all_block_indexes(); - // The fully validated chain only has candidates up to the start of the assumed-valid - // blocks. + // The fully validated chain should have the current validated tip + // and the assumed valid base as candidates. + BOOST_CHECK_EQUAL(cs1.setBlockIndexCandidates.size(), 2); BOOST_CHECK_EQUAL(cs1.setBlockIndexCandidates.count(validated_tip), 1); - BOOST_CHECK_EQUAL(cs1.setBlockIndexCandidates.count(assumed_tip), 0); - BOOST_CHECK_EQUAL(cs1.setBlockIndexCandidates.size(), assumed_valid_start_idx); + BOOST_CHECK_EQUAL(cs1.setBlockIndexCandidates.count(assumed_base), 1); - // The assumed-valid tolerant chain has all blocks as candidates. - BOOST_CHECK_EQUAL(cs2.setBlockIndexCandidates.count(validated_tip), 1); + // The assumed-valid tolerant chain has the assumed valid base as a + // candidate, but otherwise has none of the assumed-valid (which do not + // HAVE_DATA) blocks as candidates. + BOOST_CHECK_EQUAL(cs2.setBlockIndexCandidates.count(validated_tip), 0); BOOST_CHECK_EQUAL(cs2.setBlockIndexCandidates.count(assumed_tip), 1); - BOOST_CHECK_EQUAL(cs2.setBlockIndexCandidates.size(), num_indexes); + BOOST_CHECK_EQUAL(cs2.setBlockIndexCandidates.size(), num_indexes - last_assumed_valid_idx + 1); } //! Ensure that snapshot chainstates initialize properly when found on disk. BOOST_FIXTURE_TEST_CASE(chainstatemanager_snapshot_init, SnapshotTestSetup) { - this->SetupSnapshot(); - ChainstateManager& chainman = *Assert(m_node.chainman); + Chainstate& bg_chainstate = chainman.ActiveChainstate(); + + this->SetupSnapshot(); fs::path snapshot_chainstate_dir = *node::FindSnapshotChainstateDir(); BOOST_CHECK(fs::exists(snapshot_chainstate_dir)); @@ -622,6 +655,24 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_snapshot_init, SnapshotTestSetup) auto all_chainstates = chainman.GetAll(); BOOST_CHECK_EQUAL(all_chainstates.size(), 2); + // Runtime activation moved the mempool to the (active) snapshot chainstate. + BOOST_CHECK_EQUAL(chainman.ActiveChainstate().GetMempool(), m_node.mempool.get()); + BOOST_CHECK(!bg_chainstate.GetMempool()); + + // "Rewind" the background chainstate so that its tip is not at the + // base block of the snapshot - this is so after simulating a node restart, + // it will initialize instead of attempting to complete validation. + // + // Note that this is not a realistic use of DisconnectTip(). + DisconnectedBlockTransactions unused_pool; + BlockValidationState unused_state; + { + LOCK2(::cs_main, bg_chainstate.MempoolMutex()); + BOOST_CHECK(bg_chainstate.DisconnectTip(unused_state, &unused_pool)); + unused_pool.clear(); // to avoid queuedTx assertion errors on teardown + } + BOOST_CHECK_EQUAL(bg_chainstate.m_chain.Height(), 109); + // Test that simulating a shutdown (resetting ChainstateManager) and then performing // chainstate reinitializing successfully cleans up the background-validation // chainstate data, and we end up with a single chainstate that is at tip. @@ -653,7 +704,7 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_snapshot_init, SnapshotTestSetup) // chainstate. for (Chainstate* cs : chainman_restarted.GetAll()) { if (cs != &chainman_restarted.ActiveChainstate()) { - BOOST_CHECK_EQUAL(cs->m_chain.Height(), 110); + BOOST_CHECK_EQUAL(cs->m_chain.Height(), 109); } } } @@ -774,8 +825,23 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_mined_commitment_is_chain_aware, Snaps BOOST_FIXTURE_TEST_CASE(chainstatemanager_evodb_snapshot_only_flush_restart, SnapshotTestSetup) { - this->SetupSnapshot(); + auto chainstates = this->SetupSnapshot(); + Chainstate* background_chainstate = std::get<0>(chainstates); ChainstateManager& chainman = *Assert(m_node.chainman); + + // Keep this M2 marker-independence test below completion height; #25740 now + // completes and cleans up immediately on restart when background is at base. + DisconnectedBlockTransactions unused_pool; + BlockValidationState unused_state; + { + LOCK2(::cs_main, background_chainstate->MempoolMutex()); + BOOST_CHECK(background_chainstate->DisconnectTip(unused_state, &unused_pool)); + unused_pool.clear(); + background_chainstate->TryAddBlockIndexCandidate(background_chainstate->m_chain.Tip()); + background_chainstate->ForceFlushStateToDisk(); + } + BOOST_CHECK_EQUAL(background_chainstate->m_chain.Height(), 109); + uint256 normal_marker; BOOST_REQUIRE(m_node.evodb->ReadBestBlock(EvoDbIdentity::NORMAL, normal_marker)); @@ -830,6 +896,7 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_snapshot_init_missing_evodb_marker, Sn WITH_LOCK(::cs_main, restarted.ResetChainstates()); fs::remove_all(gArgs.GetDataDirNet() / "chainstate_snapshot"); + BOOST_REQUIRE(m_node.evodb->DiscardSnapshotMarkers()); this->LoadVerifyActivateChainstate(); } @@ -855,4 +922,342 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_snapshot_discarded_on_reindex, Snapsho BOOST_CHECK(!m_node.evodb->ReadBestBlock(EvoDbIdentity::SNAPSHOT, stale_marker)); } +BOOST_FIXTURE_TEST_CASE(chainstatemanager_snapshot_completion, SnapshotTestSetup) +{ + this->SetupSnapshot(); + + ChainstateManager& chainman = *Assert(m_node.chainman); + Chainstate& active_cs = chainman.ActiveChainstate(); + auto tip_cache_before_complete = active_cs.m_coinstip_cache_size_bytes; + auto db_cache_before_complete = active_cs.m_coinsdb_cache_size_bytes; + + SnapshotCompletionResult res; + auto mock_shutdown = [](bilingual_str msg) {}; + + fs::path snapshot_chainstate_dir = *node::FindSnapshotChainstateDir(); + BOOST_CHECK(fs::exists(snapshot_chainstate_dir)); + BOOST_CHECK_EQUAL(snapshot_chainstate_dir, gArgs.GetDataDirNet() / "chainstate_snapshot"); + + BOOST_CHECK(chainman.IsSnapshotActive()); + const uint256 snapshot_tip_hash = WITH_LOCK(chainman.GetMutex(), + return chainman.ActiveTip()->GetBlockHash()); + + res = WITH_LOCK(::cs_main, + return chainman.MaybeCompleteSnapshotValidation(mock_shutdown)); + BOOST_CHECK_EQUAL(res, SnapshotCompletionResult::SUCCESS); + + WITH_LOCK(::cs_main, BOOST_CHECK(chainman.IsSnapshotValidated())); + BOOST_CHECK(chainman.IsSnapshotActive()); + + // Cache should have been rebalanced and reallocated to the "only" remaining + // chainstate. + BOOST_CHECK(active_cs.m_coinstip_cache_size_bytes > tip_cache_before_complete); + BOOST_CHECK(active_cs.m_coinsdb_cache_size_bytes > db_cache_before_complete); + + auto all_chainstates = chainman.GetAll(); + BOOST_CHECK_EQUAL(all_chainstates.size(), 1); + BOOST_CHECK_EQUAL(all_chainstates[0], &active_cs); + + // Trying completion again should return false. + res = WITH_LOCK(::cs_main, + return chainman.MaybeCompleteSnapshotValidation(mock_shutdown)); + BOOST_CHECK_EQUAL(res, SnapshotCompletionResult::SKIPPED); + + // The invalid snapshot path should not have been used. + fs::path snapshot_invalid_dir = gArgs.GetDataDirNet() / "chainstate_snapshot_INVALID"; + BOOST_CHECK(!fs::exists(snapshot_invalid_dir)); + // chainstate_snapshot should still exist. + BOOST_CHECK(fs::exists(snapshot_chainstate_dir)); + + // Test that simulating a shutdown (reseting ChainstateManager) and then performing + // chainstate reinitializing successfully cleans up the background-validation + // chainstate data, and we end up with a single chainstate that is at tip. + ChainstateManager& chainman_restarted = this->SimulateNodeRestart(); + + BOOST_TEST_MESSAGE("Performing Load/Verify/Activate of chainstate"); + + // This call reinitializes the chainstates, and should clean up the now unnecessary + // background-validation leveldb contents. + this->LoadVerifyActivateChainstate(); + + BOOST_CHECK(!fs::exists(snapshot_invalid_dir)); + // chainstate_snapshot should now *not* exist. + BOOST_CHECK(!fs::exists(snapshot_chainstate_dir)); + BOOST_CHECK(m_node.evodb->VerifyBestBlock(EvoDbIdentity::NORMAL, snapshot_tip_hash)); + uint256 obsolete_marker; + BOOST_CHECK(!m_node.evodb->ReadBestBlock(EvoDbIdentity::SNAPSHOT, obsolete_marker)); + BOOST_CHECK(!m_node.evodb->ReadSnapshotBaseMNListHash(obsolete_marker)); + BOOST_CHECK(!m_node.evodb->HasDualChainstateMarker()); + + const Chainstate& active_cs2 = chainman_restarted.ActiveChainstate(); + + { + LOCK(chainman_restarted.GetMutex()); + BOOST_CHECK_EQUAL(chainman_restarted.GetAll().size(), 1); + BOOST_CHECK(!chainman_restarted.IsSnapshotActive()); + BOOST_CHECK(!chainman_restarted.IsSnapshotValidated()); + BOOST_CHECK(active_cs2.m_coinstip_cache_size_bytes > tip_cache_before_complete); + BOOST_CHECK(active_cs2.m_coinsdb_cache_size_bytes > db_cache_before_complete); + + BOOST_CHECK_EQUAL(chainman_restarted.ActiveTip()->GetBlockHash(), snapshot_tip_hash); + BOOST_CHECK_EQUAL(chainman_restarted.ActiveHeight(), 210); + } + + BOOST_TEST_MESSAGE( + "Ensure we can mine blocks on top of the \"new\" IBD chainstate"); + mineBlocks(10); + { + LOCK(chainman_restarted.GetMutex()); + BOOST_CHECK_EQUAL(chainman_restarted.ActiveHeight(), 220); + } +} + +BOOST_FIXTURE_TEST_CASE(chainstatemanager_snapshot_completion_incorrect_base_mn_list, SnapshotTestSetup) +{ + auto [validation_chainstate, snapshot_chainstate] = this->SetupSnapshot(); + ChainstateManager& chainman = *Assert(m_node.chainman); + + const CBlockIndex* base_index = WITH_LOCK(::cs_main, return validation_chainstate->m_chain.Tip()); + BOOST_REQUIRE(base_index); + const CDeterministicMNList incorrect_base_list{ + base_index->GetBlockHash(), base_index->nHeight, /*totalRegisteredCount=*/1}; + const uint256 incorrect_hash{SerializeHash(incorrect_base_list)}; + uint256 captured_block; + uint256 captured_hash; + BOOST_REQUIRE(m_node.evodb->ReadBackgroundMNListHash(captured_block, captured_hash)); + BOOST_CHECK_EQUAL(captured_block, base_index->GetBlockHash()); + BOOST_CHECK_NE(captured_hash, incorrect_hash); + m_node.dmnman->SetListForBlockForTesting(incorrect_base_list); + + // The regtest assumeutxo fixture predates DIP3. Temporarily make DIP3 active + // so the legacy completion lookup would consume the poisoned shared cache. + Consensus::Params& mutable_consensus{ + const_cast(Params().GetConsensus())}; + const int old_dip3_height{mutable_consensus.DIP0003Height}; + mutable_consensus.DIP0003Height = 1; + // Boost's execution monitor longjmps past normal unwinding on fatal + // failures, but for ordinary exceptions (BOOST_REQUIRE, deserialization + // errors from the poisoned record below) this guard keeps the process-wide + // consensus params from leaking into later test cases. + struct Dip3HeightRestore { + Consensus::Params& params; + int height; + ~Dip3HeightRestore() { params.DIP0003Height = height; } + } dip3_restore{mutable_consensus, old_dip3_height}; + + { + auto tx = m_node.evodb->BeginTransaction(EvoDbIdentity::SNAPSHOT); + // Seed an incorrect full base list, not merely an incorrect comparison + // hash. The key mirrors DB_LIST_SNAPSHOT in evo/deterministicmns.cpp + // (file-local there); keep them in sync. + m_node.evodb->Write( + std::make_pair(std::string{"dmn_S3"}, base_index->GetBlockHash()), + incorrect_base_list); + m_node.evodb->WriteSnapshotBaseMNListHash(incorrect_hash); + tx->Commit(); + } + BOOST_REQUIRE(m_node.evodb->CommitRootTransaction(EvoDbIdentity::SNAPSHOT, /*sync=*/true)); + + const auto result = WITH_LOCK(::cs_main, + return chainman.MaybeCompleteSnapshotValidation([](bilingual_str) {})); + BOOST_CHECK_EQUAL(result, SnapshotCompletionResult::EVO_STATE_MISMATCH); + BOOST_CHECK_EQUAL(&chainman.ActiveChainstate(), validation_chainstate); + BOOST_CHECK(snapshot_chainstate != &chainman.ActiveChainstate()); + BOOST_CHECK(!m_node.evodb->HasDualChainstateMarker()); + uint256 obsolete_marker; + BOOST_CHECK(!m_node.evodb->ReadBestBlock(EvoDbIdentity::SNAPSHOT, obsolete_marker)); +} + +BOOST_FIXTURE_TEST_CASE(chainstatemanager_snapshot_cleanup_recovers_first_rename, SnapshotTestSetup) +{ + this->SetupSnapshot(); + ChainstateManager& chainman = *Assert(m_node.chainman); + const uint256 snapshot_tip = WITH_LOCK(::cs_main, return chainman.ActiveChainstate().CoinsTip().GetBestBlock()); + BOOST_REQUIRE_EQUAL(WITH_LOCK(::cs_main, + return chainman.MaybeCompleteSnapshotValidation([](bilingual_str) {})), + SnapshotCompletionResult::SUCCESS); + + this->SimulateNodeRestart(); + const fs::path data_dir{gArgs.GetDataDirNet()}; + fs::rename(data_dir / "chainstate", data_dir / "chainstate_todelete"); + + this->LoadVerifyActivateChainstate(); + + BOOST_CHECK(fs::exists(data_dir / "chainstate")); + BOOST_CHECK(!fs::exists(data_dir / "chainstate_snapshot")); + BOOST_CHECK(!fs::exists(data_dir / "chainstate_todelete")); + BOOST_CHECK(m_node.evodb->VerifyBestBlock(EvoDbIdentity::NORMAL, snapshot_tip)); +} + +BOOST_FIXTURE_TEST_CASE(chainstatemanager_snapshot_cleanup_recovers_completed_swap, SnapshotTestSetup) +{ + this->SetupSnapshot(); + ChainstateManager& chainman = *Assert(m_node.chainman); + const uint256 snapshot_tip = WITH_LOCK(::cs_main, return chainman.ActiveChainstate().CoinsTip().GetBestBlock()); + BOOST_REQUIRE_EQUAL(WITH_LOCK(::cs_main, + return chainman.MaybeCompleteSnapshotValidation([](bilingual_str) {})), + SnapshotCompletionResult::SUCCESS); + + this->SimulateNodeRestart(); + const fs::path data_dir{gArgs.GetDataDirNet()}; + fs::rename(data_dir / "chainstate", data_dir / "chainstate_todelete"); + fs::rename(data_dir / "chainstate_snapshot", data_dir / "chainstate"); + + this->LoadVerifyActivateChainstate(); + BOOST_CHECK(m_node.evodb->VerifyBestBlock(EvoDbIdentity::NORMAL, snapshot_tip)); + BOOST_CHECK(!m_node.evodb->HasDualChainstateMarker()); + + // A restart of the already-promoted state is an idempotent no-op. + this->SimulateNodeRestart(); + this->LoadVerifyActivateChainstate(); + BOOST_CHECK(m_node.evodb->VerifyBestBlock(EvoDbIdentity::NORMAL, snapshot_tip)); +} + +BOOST_FIXTURE_TEST_CASE(chainstatemanager_snapshot_cleanup_recovers_invalid_rename, SnapshotTestSetup) +{ + this->SetupSnapshot(); + ChainstateManager& restarted = this->SimulateNodeRestart(); + const fs::path data_dir{gArgs.GetDataDirNet()}; + fs::rename(data_dir / "chainstate_snapshot", data_dir / "chainstate_snapshot_INVALID"); + + this->LoadVerifyActivateChainstate(); + + BOOST_CHECK(fs::exists(data_dir / "chainstate_snapshot_INVALID")); + BOOST_CHECK(!fs::exists(data_dir / "chainstate_snapshot")); + BOOST_CHECK(!m_node.evodb->HasDualChainstateMarker()); + uint256 obsolete; + BOOST_CHECK(!m_node.evodb->ReadBestBlock(EvoDbIdentity::SNAPSHOT, obsolete)); + BOOST_CHECK(m_node.evodb->VerifyBestBlock( + EvoDbIdentity::NORMAL, + WITH_LOCK(::cs_main, return restarted.ActiveChainstate().CoinsTip().GetBestBlock()))); +} + +BOOST_FIXTURE_TEST_CASE(chainstatemanager_snapshot_cleanup_recovers_promoted_swap, SnapshotTestSetup) +{ + this->SetupSnapshot(); + ChainstateManager& chainman = *Assert(m_node.chainman); + const uint256 snapshot_tip = WITH_LOCK(::cs_main, return chainman.ActiveChainstate().CoinsTip().GetBestBlock()); + BOOST_REQUIRE_EQUAL(WITH_LOCK(::cs_main, + return chainman.MaybeCompleteSnapshotValidation([](bilingual_str) {})), + SnapshotCompletionResult::SUCCESS); + + this->SimulateNodeRestart(); + const fs::path data_dir{gArgs.GetDataDirNet()}; + fs::rename(data_dir / "chainstate", data_dir / "chainstate_todelete"); + fs::rename(data_dir / "chainstate_snapshot", data_dir / "chainstate"); + BOOST_REQUIRE(m_node.evodb->PromoteSnapshotMarkers(snapshot_tip)); + + this->LoadVerifyActivateChainstate(); + BOOST_CHECK(!fs::exists(data_dir / "chainstate_todelete")); + BOOST_CHECK(m_node.evodb->VerifyBestBlock(EvoDbIdentity::NORMAL, snapshot_tip)); +} + +BOOST_FIXTURE_TEST_CASE(chainstatemanager_snapshot_base_is_cached, SnapshotTestSetup) +{ + this->SetupSnapshot(); + ChainstateManager& chainman = *Assert(m_node.chainman); + const uint256 base_hash{*chainman.SnapshotBlockhash()}; + + { + LOCK(::cs_main); + const CBlockIndex* cached_base = chainman.ActiveChainstate().SnapshotBase(); + BOOST_REQUIRE(cached_base); + auto node = chainman.BlockIndex().extract(base_hash); + BOOST_REQUIRE(!node.empty()); + BOOST_CHECK_EQUAL(chainman.ActiveChainstate().SnapshotBase(), cached_base); + chainman.BlockIndex().insert(std::move(node)); + } +} + +BOOST_FIXTURE_TEST_CASE(chainstatemanager_snapshot_completion_without_base_list_marker, SnapshotTestSetup) +{ + this->SetupSnapshot(); + ChainstateManager& chainman = *Assert(m_node.chainman); + + // Simulate a cold-start activation, where the base MN list was not + // derivable and so no lifecycle hash markers were captured: completion + // must fall back to the UTXO-set hash alone instead of quarantining a + // valid snapshot. + { + auto tx = m_node.evodb->BeginTransaction(EvoDbIdentity::SNAPSHOT); + m_node.evodb->Erase(EVODB_SNAPSHOT_MNLIST_HASH); + m_node.evodb->Erase(EVODB_BACKGROUND_MNLIST_HASH); + tx->Commit(); + } + BOOST_REQUIRE(m_node.evodb->CommitRootTransaction(EvoDbIdentity::SNAPSHOT, /*sync=*/true)); + + const auto res = WITH_LOCK(::cs_main, + return chainman.MaybeCompleteSnapshotValidation([](bilingual_str) {})); + BOOST_CHECK_EQUAL(res, SnapshotCompletionResult::SUCCESS); + WITH_LOCK(::cs_main, BOOST_CHECK(chainman.IsSnapshotValidated())); +} + +BOOST_FIXTURE_TEST_CASE(chainstatemanager_snapshot_completion_hash_mismatch, SnapshotTestSetup) +{ + auto chainstates = this->SetupSnapshot(); + Chainstate& validation_chainstate = *std::get<0>(chainstates); + ChainstateManager& chainman = *Assert(m_node.chainman); + SnapshotCompletionResult res; + auto mock_shutdown = [](bilingual_str msg) {}; + + // Test tampering with the IBD UTXO set with an extra coin to ensure it causes + // snapshot completion to fail. + CCoinsViewCache& ibd_coins = WITH_LOCK(::cs_main, + return validation_chainstate.CoinsTip()); + Coin badcoin; + badcoin.out.nValue = InsecureRand32(); + badcoin.nHeight = 1; + badcoin.out.scriptPubKey.assign(InsecureRandBits(6), 0); + uint256 txid = InsecureRand256(); + ibd_coins.AddCoin(COutPoint(txid, 0), std::move(badcoin), false); + + fs::path snapshot_chainstate_dir = gArgs.GetDataDirNet() / "chainstate_snapshot"; + BOOST_CHECK(fs::exists(snapshot_chainstate_dir)); + + { + ASSERT_DEBUG_LOG("failed to validate the -assumeutxo snapshot state"); + res = WITH_LOCK(::cs_main, + return chainman.MaybeCompleteSnapshotValidation(mock_shutdown)); + BOOST_CHECK_EQUAL(res, SnapshotCompletionResult::HASH_MISMATCH); + } + + auto all_chainstates = chainman.GetAll(); + BOOST_CHECK_EQUAL(all_chainstates.size(), 1); + BOOST_CHECK_EQUAL(all_chainstates[0], &validation_chainstate); + BOOST_CHECK_EQUAL(&chainman.ActiveChainstate(), &validation_chainstate); + + fs::path snapshot_invalid_dir = gArgs.GetDataDirNet() / "chainstate_snapshot_INVALID"; + BOOST_CHECK(fs::exists(snapshot_invalid_dir)); + + // Test that simulating a shutdown (reseting ChainstateManager) and then performing + // chainstate reinitializing successfully loads only the fully-validated + // chainstate data, and we end up with a single chainstate that is at tip. + ChainstateManager& chainman_restarted = this->SimulateNodeRestart(); + + BOOST_TEST_MESSAGE("Performing Load/Verify/Activate of chainstate"); + + // This call reinitializes the chainstates, and should clean up the now unnecessary + // background-validation leveldb contents. + this->LoadVerifyActivateChainstate(); + + BOOST_CHECK(fs::exists(snapshot_invalid_dir)); + BOOST_CHECK(!fs::exists(snapshot_chainstate_dir)); + + { + LOCK(::cs_main); + BOOST_CHECK_EQUAL(chainman_restarted.GetAll().size(), 1); + BOOST_CHECK(!chainman_restarted.IsSnapshotActive()); + BOOST_CHECK(!chainman_restarted.IsSnapshotValidated()); + BOOST_CHECK_EQUAL(chainman_restarted.ActiveHeight(), 210); + } + + BOOST_TEST_MESSAGE( + "Ensure we can mine blocks on top of the \"new\" IBD chainstate"); + mineBlocks(10); + { + LOCK(::cs_main); + BOOST_CHECK_EQUAL(chainman_restarted.ActiveHeight(), 220); + } +} + BOOST_AUTO_TEST_SUITE_END() diff --git a/src/validation.cpp b/src/validation.cpp index b97d75a32916..436b9edb310a 100644 --- a/src/validation.cpp +++ b/src/validation.cpp @@ -1645,6 +1645,13 @@ std::string Chainstate::EvoDbInconsistencyMessage() return "Found EvoDB inconsistency, you must reindex to continue"; } +const CBlockIndex* Chainstate::SnapshotBase() +{ + if (!m_from_snapshot_blockhash) return nullptr; + if (!m_cached_snapshot_base) m_cached_snapshot_base = Assert(m_chainman.m_blockman.LookupBlockIndex(*m_from_snapshot_blockhash)); + return m_cached_snapshot_base; +} + void Chainstate::InitCoinsDB( size_t cache_size_bytes, bool in_memory, @@ -2751,6 +2758,17 @@ void Chainstate::ForceFlushStateToDisk() } } +void Chainstate::RecordBackgroundMNListHash(const CBlockIndex* pindex, const CDeterministicMNList& mn_list) +{ + if (EvoDbIdentity() != ::EvoDbIdentity::NORMAL) return; + // Only the snapshot base block's list takes part in snapshot completion, + // and it only matters while a snapshot chainstate exists. Hashing the full + // list is too expensive to do on every connect. + const auto base_blockhash = m_chainman.SnapshotBlockhash(); + if (!base_blockhash || *base_blockhash != pindex->GetBlockHash()) return; + m_evoDb.WriteBackgroundMNListHash(pindex->GetBlockHash(), ::SerializeHash(mn_list)); +} + void Chainstate::PruneAndFlush() { BlockValidationState state; @@ -3090,6 +3108,14 @@ bool Chainstate::ConnectTip(BlockValidationState& state, CBlockIndex* pindexNew, ::g_stats_client->gauge("blocks.tip.NumTransactions", blockConnecting.vtx.size(), 1.0f); ::g_stats_client->gauge("blocks.tip.SigOps", nSigOps, 1.0f); + // If we are the background validation chainstate, check to see if we are done + // validating the snapshot (i.e. our tip has reached the snapshot's base block). + if (this != &m_chainman.ActiveChainstate()) { + // This call may set `m_disabled`, which is referenced immediately afterwards in + // ActivateBestChain, so that we stop connecting blocks past the snapshot base. + m_chainman.MaybeCompleteSnapshotValidation(); + } + connectTrace.BlockConnected(pindexNew, std::move(pthisBlock)); return true; } @@ -3321,6 +3347,14 @@ bool Chainstate::ActivateBestChain(BlockValidationState& state, std::shared_ptr< auto start = Now(); + // Belt-and-suspenders check that we aren't attempting to advance the background + // chainstate past the snapshot base block. + if (WITH_LOCK(::cs_main, return m_disabled)) { + LogPrintf("m_disabled is set - this chainstate should not be in operation. " /* Continued */ + "Please report this as a bug. %s\n", PACKAGE_BUGREPORT); + return false; + } + CBlockIndex *pindexMostWork = nullptr; CBlockIndex *pindexNewTip = nullptr; int nStopAtHeight = gArgs.GetIntArg("-stopatheight", DEFAULT_STOPATHEIGHT); @@ -3373,6 +3407,15 @@ bool Chainstate::ActivateBestChain(BlockValidationState& state, std::shared_ptr< GetMainSignals().BlockConnected(trace.pblock, trace.pindex); } } + + // This will have been toggled in + // ActivateBestChainStep -> ConnectTip -> MaybeCompleteSnapshotValidation, + // if at all, so we should catch it here. + // + // Break this do-while to ensure we don't advance past the base snapshot. + if (m_disabled) { + break; + } } while (!m_chain.Tip() || (starting_tip && CBlockIndexWorkComparator()(m_chain.Tip(), starting_tip))); if (!blocks_connected) return true; @@ -3394,13 +3437,19 @@ bool Chainstate::ActivateBestChain(BlockValidationState& state, std::shared_ptr< if (nStopAtHeight && pindexNewTip && pindexNewTip->nHeight >= nStopAtHeight) StartShutdown(); + if (WITH_LOCK(::cs_main, return m_disabled)) { + // Background chainstate has reached the snapshot base block, so exit. + break; + } + // We check shutdown only after giving ActivateBestChainStep a chance to run once so that we // never shutdown before connecting the genesis block during LoadChainTip(). Previously this // caused an assert() failure during shutdown in such cases as the UTXO DB flushing checks // that the best block hash is non-null. if (ShutdownRequested()) break; } while (pindexNewTip != pindexMostWork); - CheckBlockIndex(); + + m_chainman.CheckBlockIndex(); auto finish = Now(); auto diff = finish - start; @@ -3424,17 +3473,17 @@ bool Chainstate::PreciousBlock(BlockValidationState& state, CBlockIndex* pindex) // Nothing to do, this block is not at the tip. return true; } - if (m_chain.Tip()->nChainWork > nLastPreciousChainwork) { + if (m_chain.Tip()->nChainWork > m_chainman.nLastPreciousChainwork) { // The chain has been extended since the last call, reset the counter. - nBlockReverseSequenceId = -1; + m_chainman.nBlockReverseSequenceId = -1; } - nLastPreciousChainwork = m_chain.Tip()->nChainWork; + m_chainman.nLastPreciousChainwork = m_chain.Tip()->nChainWork; setBlockIndexCandidates.erase(pindex); - pindex->nSequenceId = nBlockReverseSequenceId; - if (nBlockReverseSequenceId > std::numeric_limits::min()) { + pindex->nSequenceId = m_chainman.nBlockReverseSequenceId; + if (m_chainman.nBlockReverseSequenceId > std::numeric_limits::min()) { // We can't keep reducing the counter if somebody really wants to // call preciousblock 2**31-1 times on the same set of tips... - nBlockReverseSequenceId--; + m_chainman.nBlockReverseSequenceId--; } if (pindex->IsValid(BLOCK_VALID_TRANSACTIONS) && !(pindex->nStatus & BLOCK_CONFLICT_CHAINLOCK) && pindex->HaveTxsDownloaded()) { setBlockIndexCandidates.insert(pindex); @@ -3484,6 +3533,7 @@ bool Chainstate::InvalidateBlock(BlockValidationState& state, CBlockIndex* pinde if (!m_chain.Contains(candidate) && !CBlockIndexWorkComparator()(candidate, pindex->pprev) && candidate->IsValid(BLOCK_VALID_TRANSACTIONS) && + !(candidate->nStatus & BLOCK_CONFLICT_CHAINLOCK) && candidate->HaveTxsDownloaded()) { candidate_blocks_by_work.insert(std::make_pair(candidate->nChainWork, candidate)); } @@ -3561,7 +3611,7 @@ bool Chainstate::InvalidateBlock(BlockValidationState& state, CBlockIndex* pinde to_mark_failed = invalid_walk_tip; } - CheckBlockIndex(); + m_chainman.CheckBlockIndex(); { LOCK(cs_main); @@ -3721,6 +3771,7 @@ void Chainstate::ResetBlockFailureFlags(CBlockIndex *pindex, bool ignore_chainlo } int nHeight = pindex->nHeight; + std::vector reconsidered_blocks; // Remove the invalidity flag from this block and all its descendants. for (auto& [_, block_index] : m_blockman.m_block_index) { @@ -3730,11 +3781,7 @@ void Chainstate::ResetBlockFailureFlags(CBlockIndex *pindex, bool ignore_chainlo block_index.nStatus &= ~BLOCK_CONFLICT_CHAINLOCK; } m_blockman.m_dirty_blockindex.insert(&block_index); - if (block_index.IsValid(BLOCK_VALID_TRANSACTIONS) && block_index.HaveTxsDownloaded() && setBlockIndexCandidates.value_comp()(m_chain.Tip(), &block_index)) { - if (ignore_chainlocks || !(block_index.nStatus & BLOCK_CONFLICT_CHAINLOCK)) { - setBlockIndexCandidates.insert(&block_index); - } - } + reconsidered_blocks.push_back(&block_index); if (&block_index == m_chainman.m_best_invalid) { // Reset invalid block marker if it was pointing to one of those. m_chainman.m_best_invalid = nullptr; @@ -3751,11 +3798,7 @@ void Chainstate::ResetBlockFailureFlags(CBlockIndex *pindex, bool ignore_chainlo pindex->nStatus &= ~BLOCK_CONFLICT_CHAINLOCK; } m_blockman.m_dirty_blockindex.insert(pindex); - if (pindex->IsValid(BLOCK_VALID_TRANSACTIONS) && pindex->HaveTxsDownloaded() && setBlockIndexCandidates.value_comp()(m_chain.Tip(), pindex)) { - if (ignore_chainlocks || !(pindex->nStatus & BLOCK_CONFLICT_CHAINLOCK)) { - setBlockIndexCandidates.insert(pindex); - } - } + reconsidered_blocks.push_back(pindex); if (pindex == m_chainman.m_best_invalid) { // Reset invalid block marker if it was pointing to one of those. m_chainman.m_best_invalid = nullptr; @@ -3774,10 +3817,48 @@ void Chainstate::ResetBlockFailureFlags(CBlockIndex *pindex, bool ignore_chainlo } pindex = pindex->pprev; } + + // Failure flags and m_best_invalid are shared by all chainstates, so + // candidate admission must be recomputed for all of them as well. + for (CBlockIndex* reconsidered : reconsidered_blocks) { + if (!reconsidered->IsValid(BLOCK_VALID_TRANSACTIONS) || !reconsidered->HaveTxsDownloaded()) continue; + for (Chainstate* chainstate : m_chainman.GetAll()) { + chainstate->TryAddBlockIndexCandidate(reconsidered); + } + } +} + +void Chainstate::TryAddBlockIndexCandidate(CBlockIndex* pindex) +{ + AssertLockHeld(cs_main); + // ChainLock-conflicting blocks are never eligible for activation, even + // though CBlockIndex::IsValid() only considers BLOCK_FAILED_MASK. + if (pindex->nStatus & BLOCK_CONFLICT_CHAINLOCK) { + return; + } + // The block only is a candidate for the most-work-chain if it has more work than our current tip. + if (m_chain.Tip() != nullptr && setBlockIndexCandidates.value_comp()(pindex, m_chain.Tip())) { + return; + } + + bool is_active_chainstate = this == &m_chainman.ActiveChainstate(); + if (is_active_chainstate) { + // The active chainstate should always add entries that have more + // work than the tip. + setBlockIndexCandidates.insert(pindex); + } else if (!m_disabled) { + // For the background chainstate, we only consider connecting blocks + // towards the snapshot base (which can't be nullptr or else we'll + // never make progress). + const CBlockIndex* snapshot_base{Assert(m_chainman.GetSnapshotBaseBlock())}; + if (snapshot_base->GetAncestor(pindex->nHeight) == pindex) { + setBlockIndexCandidates.insert(pindex); + } + } } /** Mark a block as having its data received and checked (up to BLOCK_VALID_TRANSACTIONS). */ -void Chainstate::ReceivedBlockTransactions(const CBlock& block, CBlockIndex* pindexNew, const FlatFilePos& pos) +void ChainstateManager::ReceivedBlockTransactions(const CBlock& block, CBlockIndex* pindexNew, const FlatFilePos& pos) { AssertLockHeld(cs_main); pindexNew->nTx = block.vtx.size(); @@ -3800,10 +3881,8 @@ void Chainstate::ReceivedBlockTransactions(const CBlock& block, CBlockIndex* pin queue.pop_front(); pindex->nChainTx = (pindex->pprev ? pindex->pprev->nChainTx : 0) + pindex->nTx; pindex->nSequenceId = nBlockSequenceId++; - if (m_chain.Tip() == nullptr || !setBlockIndexCandidates.value_comp()(pindex, m_chain.Tip())) { - if (!(pindex->nStatus & BLOCK_CONFLICT_CHAINLOCK)) { - setBlockIndexCandidates.insert(pindex); - } + for (Chainstate *c : GetAll()) { + c->TryAddBlockIndexCandidate(pindex); } std::pair::iterator, std::multimap::iterator> range = m_blockman.m_blocks_unlinked.equal_range(pindex); while (range.first != range.second) { @@ -4235,7 +4314,7 @@ bool ChainstateManager::ProcessNewBlockHeaders(const std::vector& for (const CBlockHeader& header : headers) { CBlockIndex *pindex = nullptr; // Use a temp pindex instead of ppindex to avoid a const_cast bool accepted{AcceptBlockHeader(header, state, &pindex, header.GetHash())}; - ActiveChainstate().CheckBlockIndex(); + CheckBlockIndex(); if (!accepted) { return false; @@ -4257,7 +4336,7 @@ bool ChainstateManager::ProcessNewBlockHeaders(const std::vector& } /** Store block on disk. If dbp is non-nullptr, the file is known to already reside on disk */ -bool Chainstate::AcceptBlock(const std::shared_ptr& pblock, BlockValidationState& state, CBlockIndex** ppindex, bool fRequested, const FlatFilePos* dbp, bool* fNewBlock, const uint256* known_hash) +bool ChainstateManager::AcceptBlock(const std::shared_ptr& pblock, BlockValidationState& state, CBlockIndex** ppindex, bool fRequested, const FlatFilePos* dbp, bool* fNewBlock, const uint256* known_hash) { auto start = Now(); @@ -4275,23 +4354,24 @@ bool Chainstate::AcceptBlock(const std::shared_ptr& pblock, BlockV ASSERT_IF_DEBUG(!known_hash || *known_hash == block.GetHash()); const uint256 hash{known_hash ? *known_hash : block.GetHash()}; - bool accepted_header{m_chainman.AcceptBlockHeader(block, state, &pindex, hash)}; + bool accepted_header{AcceptBlockHeader(block, state, &pindex, hash)}; CheckBlockIndex(); if (!accepted_header) return false; - // Try to process all requested blocks that we don't have, but only - // process an unrequested block if it's new and has enough work to - // advance our tip, and isn't too many blocks ahead. + // Check all requested blocks that we do not already have for validity and + // save them to disk. Skip processing of unrequested blocks as an anti-DoS + // measure, unless the blocks have more work than the active chain tip, and + // aren't too far ahead of it, so are likely to be attached soon. bool fAlreadyHave = pindex->nStatus & BLOCK_HAVE_DATA; - bool fHasMoreOrSameWork = (m_chain.Tip() ? pindex->nChainWork >= m_chain.Tip()->nChainWork : true); + bool fHasMoreOrSameWork = (ActiveTip() ? pindex->nChainWork >= ActiveTip()->nChainWork : true); // Blocks that are too out-of-order needlessly limit the effectiveness of // pruning, because pruning will not delete block files that contain any // blocks which are too close in height to the tip. Apply this test // regardless of whether pruning is enabled; it should generally be safe to // not process unrequested blocks. - bool fTooFarAhead{pindex->nHeight > m_chain.Height() + int(MIN_BLOCKS_TO_KEEP)}; + bool fTooFarAhead{pindex->nHeight > ActiveHeight() + int(MIN_BLOCKS_TO_KEEP)}; // TODO: Decouple this function from the block download logic by removing fRequested // This requires some new chain data structure to efficiently look up if a @@ -4314,8 +4394,8 @@ bool Chainstate::AcceptBlock(const std::shared_ptr& pblock, BlockV if (pindex->nChainWork < nMinimumChainWork) return true; } - if (!CheckBlock(block, state, m_params.GetConsensus(), true, true, &hash) || - !ContextualCheckBlock(block, state, m_chainman, pindex->pprev)) { + if (!CheckBlock(block, state, GetConsensus(), true, true, &hash) || + !ContextualCheckBlock(block, state, *this, pindex->pprev)) { if (state.IsInvalid() && state.GetResult() != BlockValidationResult::BLOCK_MUTATED) { pindex->nStatus |= BLOCK_FAILED_VALID; m_blockman.m_dirty_blockindex.insert(pindex); @@ -4325,13 +4405,13 @@ bool Chainstate::AcceptBlock(const std::shared_ptr& pblock, BlockV // Header is valid/has work, merkle tree is good...RELAY NOW // (but if it does not build on our best tip, let the SendMessages loop relay it) - if (!IsInitialBlockDownload() && m_chain.Tip() == pindex->pprev) + if (!ActiveChainstate().IsInitialBlockDownload() && ActiveTip() == pindex->pprev) GetMainSignals().NewPoWValidBlock(pindex, pblock); // Write block to history file if (fNewBlock) *fNewBlock = true; try { - FlatFilePos blockPos{m_blockman.SaveBlockToDisk(block, pindex->nHeight, m_chain, dbp)}; + FlatFilePos blockPos{m_blockman.SaveBlockToDisk(block, pindex->nHeight, dbp)}; if (blockPos.IsNull()) { state.Error(strprintf("%s: Failed to find position to write new block to disk", __func__)); return false; @@ -4341,8 +4421,15 @@ bool Chainstate::AcceptBlock(const std::shared_ptr& pblock, BlockV return AbortNode(state, std::string("System error: ") + e.what()); } - if (CanFlushToDisk()) { - FlushStateToDisk(state, FlushStateMode::NONE); + // TODO: FlushStateToDisk() handles flushing of both block and chainstate + // data, so we should move this to ChainstateManager so that we can be more + // intelligent about how we flush. + // For now, since FlushStateMode::NONE is used, all that can happen is that + // the block files may be pruned, so we can just call this on one + // chainstate (particularly if we haven't implemented pruning with + // background validation yet). + if (ActiveChainstate().CanFlushToDisk()) { + ActiveChainstate().FlushStateToDisk(state, FlushStateMode::NONE); } CheckBlockIndex(); @@ -4375,7 +4462,7 @@ bool ChainstateManager::ProcessNewBlock(const std::shared_ptr& blo bool ret = CheckBlock(*block, state, GetConsensus()); if (ret) { // Store to disk - ret = ActiveChainstate().AcceptBlock(block, state, &pindex, force_processing, nullptr, new_block); + ret = AcceptBlock(block, state, &pindex, force_processing, nullptr, new_block); } if (!ret) { GetMainSignals().BlockChecked(*block, state); @@ -4748,10 +4835,9 @@ bool Chainstate::ReplayBlocks() return true; } -void Chainstate::UnloadBlockIndex() +void Chainstate::ClearBlockIndexCandidates() { AssertLockHeld(::cs_main); - nBlockSequenceId = 1; setBlockIndexCandidates.clear(); } @@ -4770,60 +4856,19 @@ bool ChainstateManager::LoadBlockIndex() std::sort(vSortedByHeight.begin(), vSortedByHeight.end(), CBlockIndexHeightOnlyComparator()); - // Find start of assumed-valid region. - int first_assumed_valid_height = std::numeric_limits::max(); - - for (const CBlockIndex* block : vSortedByHeight) { - if (block->IsAssumedValid()) { - auto chainstates = GetAll(); - - // If we encounter an assumed-valid block index entry, ensure that we have - // one chainstate that tolerates assumed-valid entries and another that does - // not (i.e. the background validation chainstate), since assumed-valid - // entries should always be pending validation by a fully-validated chainstate. - auto any_chain = [&](auto fnc) { return std::any_of(chainstates.cbegin(), chainstates.cend(), fnc); }; - assert(any_chain([](auto chainstate) { return chainstate->reliesOnAssumedValid(); })); - assert(any_chain([](auto chainstate) { return !chainstate->reliesOnAssumedValid(); })); - - first_assumed_valid_height = block->nHeight; - break; - } - } - for (CBlockIndex* pindex : vSortedByHeight) { if (ShutdownRequested()) return false; - if (pindex->IsAssumedValid() || + // If we have an assumeutxo-based chainstate, then the snapshot + // block will be a candidate for the tip, but it may not be + // VALID_TRANSACTIONS (eg if we haven't yet downloaded the block), + // so we special-case the snapshot block as a potential candidate + // here. + if (pindex == GetSnapshotBaseBlock() || (pindex->IsValid(BLOCK_VALID_TRANSACTIONS) && (pindex->HaveTxsDownloaded() || pindex->pprev == nullptr))) { - // Fill each chainstate's block candidate set. Only add assumed-valid - // blocks to the tip candidate set if the chainstate is allowed to rely on - // assumed-valid blocks. - // - // If all setBlockIndexCandidates contained the assumed-valid blocks, the - // background chainstate's ActivateBestChain() call would add assumed-valid - // blocks to the chain (based on how FindMostWorkChain() works). Obviously - // we don't want this since the purpose of the background validation chain - // is to validate assued-valid blocks. - // - // Note: This is considering all blocks whose height is greater or equal to - // the first assumed-valid block to be assumed-valid blocks, and excluding - // them from the background chainstate's setBlockIndexCandidates set. This - // does mean that some blocks which are not technically assumed-valid - // (later blocks on a fork beginning before the first assumed-valid block) - // might not get added to the background chainstate, but this is ok, - // because they will still be attached to the active chainstate if they - // actually contain more work. - // - // Instead of this height-based approach, an earlier attempt was made at - // detecting "holistically" whether the block index under consideration - // relied on an assumed-valid ancestor, but this proved to be too slow to - // be practical. for (Chainstate* chainstate : GetAll()) { - if (chainstate->reliesOnAssumedValid() || - pindex->nHeight < first_assumed_valid_height) { - chainstate->setBlockIndexCandidates.insert(pindex); - } + chainstate->TryAddBlockIndexCandidate(pindex); } } if (pindex->nStatus & BLOCK_FAILED_MASK && (!m_best_invalid || pindex->nChainWork > m_best_invalid->nChainWork)) { @@ -4850,12 +4895,12 @@ bool ChainstateManager::LoadBlockIndex() bool Chainstate::AddGenesisBlock(const CBlock& block, BlockValidationState& state) { - FlatFilePos blockPos{m_blockman.SaveBlockToDisk(block, 0, m_chain, nullptr)}; + FlatFilePos blockPos{m_blockman.SaveBlockToDisk(block, 0, nullptr)}; if (blockPos.IsNull()) { return error("%s: writing genesis block to disk failed (%s)", __func__, state.ToString()); } CBlockIndex* pindex = m_blockman.AddToBlockIndex(block, block.GetHash(), m_chainman.m_best_header); - ReceivedBlockTransactions(block, pindex, blockPos); + m_chainman.ReceivedBlockTransactions(block, pindex, blockPos); return true; } @@ -4882,7 +4927,7 @@ bool Chainstate::LoadGenesisBlock() m_params.DevNetGenesisBlock()); bool fCheckBlock = CheckBlock(*shared_pblock, state, m_params.GetConsensus()); assert(fCheckBlock); - if (!AcceptBlock(shared_pblock, state, nullptr, true, nullptr, nullptr)) + if (!m_chainman.AcceptBlock(shared_pblock, state, nullptr, true, nullptr, nullptr)) return false; } } catch (const std::runtime_error &e) { @@ -4892,17 +4937,16 @@ bool Chainstate::LoadGenesisBlock() return true; } -void Chainstate::LoadExternalBlockFile( +void ChainstateManager::LoadExternalBlockFile( FILE* fileIn, FlatFilePos* dbp, std::multimap* blocks_with_unknown_parent) { - AssertLockNotHeld(m_chainstate_mutex); - // Either both should be specified (-reindex), or neither (-loadblock). assert(!dbp == !blocks_with_unknown_parent); const auto start{SteadyClock::now()}; + const CChainParams& params{GetParams()}; int nLoaded = 0; try { @@ -4922,10 +4966,10 @@ void Chainstate::LoadExternalBlockFile( try { // locate a header unsigned char buf[CMessageHeader::MESSAGE_START_SIZE]; - blkdat.FindByte(m_params.MessageStart()[0]); + blkdat.FindByte(params.MessageStart()[0]); nRewind = blkdat.GetPos() + 1; blkdat >> buf; - if (memcmp(buf, m_params.MessageStart(), CMessageHeader::MESSAGE_START_SIZE)) { + if (memcmp(buf, params.MessageStart(), CMessageHeader::MESSAGE_START_SIZE)) { continue; } // read size @@ -4956,7 +5000,7 @@ void Chainstate::LoadExternalBlockFile( { LOCK(cs_main); // detect out of order blocks, and store them for later - if (hash != m_params.GetConsensus().hashGenesisBlock && !m_blockman.LookupBlockIndex(header.hashPrevBlock)) { + if (hash != params.GetConsensus().hashGenesisBlock && !m_blockman.LookupBlockIndex(header.hashPrevBlock)) { LogPrint(BCLog::REINDEX, "%s: Out of order block %s, parent %s not known\n", __func__, hash.ToString(), header.hashPrevBlock.ToString()); if (dbp && blocks_with_unknown_parent) { @@ -4981,15 +5025,22 @@ void Chainstate::LoadExternalBlockFile( if (state.IsError()) { break; } - } else if (hash != m_params.GetConsensus().hashGenesisBlock && pindex->nHeight % 1000 == 0) { + } else if (hash != params.GetConsensus().hashGenesisBlock && pindex->nHeight % 1000 == 0) { LogPrint(BCLog::REINDEX, "Block Import: already had block %s at height %d\n", hash.ToString(), pindex->nHeight); } } // Activate the genesis block so normal node progress can continue - if (hash == m_params.GetConsensus().hashGenesisBlock) { - BlockValidationState state; - if (!ActivateBestChain(state, nullptr)) { + if (hash == params.GetConsensus().hashGenesisBlock) { + bool genesis_activation_failure = false; + for (auto c : GetAll()) { + BlockValidationState state; + if (!c->ActivateBestChain(state, nullptr)) { + genesis_activation_failure = true; + break; + } + } + if (genesis_activation_failure) { break; } } @@ -5002,14 +5053,21 @@ void Chainstate::LoadExternalBlockFile( // until after all of the block files are loaded. ActivateBestChain can be // called by concurrent network message processing. but, that is not // reliable for the purpose of pruning while importing. - BlockValidationState state; - if (!ActivateBestChain(state, pblock)) { - LogPrint(BCLog::REINDEX, "failed to activate chain (%s)\n", state.ToString()); + bool activation_failure = false; + for (auto c : GetAll()) { + BlockValidationState state; + if (!c->ActivateBestChain(state, pblock)) { + LogPrint(BCLog::REINDEX, "failed to activate chain (%s)\n", state.ToString()); + activation_failure = true; + break; + } + } + if (activation_failure) { break; } } - NotifyHeaderTip(*this); + NotifyHeaderTip(ActiveChainstate()); if (!blocks_with_unknown_parent) continue; @@ -5023,7 +5081,7 @@ void Chainstate::LoadExternalBlockFile( while (range.first != range.second) { std::multimap::iterator it = range.first; std::shared_ptr pblockrecursive = std::make_shared(); - if (auto opt_hash{ReadBlockFromDisk(*pblockrecursive, it->second, m_params.GetConsensus())}) { + if (auto opt_hash{ReadBlockFromDisk(*pblockrecursive, it->second, params.GetConsensus())}) { const uint256& blockhash = *opt_hash; LogPrint(BCLog::REINDEX, "%s: Processing out of order child %s of %s\n", __func__, blockhash.ToString(), head.ToString()); @@ -5036,7 +5094,7 @@ void Chainstate::LoadExternalBlockFile( } range.first++; blocks_with_unknown_parent->erase(it); - NotifyHeaderTip(*this); + NotifyHeaderTip(ActiveChainstate()); } } } catch (const std::exception& e) { @@ -5060,7 +5118,7 @@ void Chainstate::LoadExternalBlockFile( LogPrintf("Loaded %i blocks from external file in %dms\n", nLoaded, Ticks(SteadyClock::now() - start)); } -void Chainstate::CheckBlockIndex() +void ChainstateManager::CheckBlockIndex() { if (!fCheckBlockIndex) { return; @@ -5071,7 +5129,7 @@ void Chainstate::CheckBlockIndex() // During a reindex, we read the genesis block and call CheckBlockIndex before ActivateBestChain, // so we have the genesis block in m_blockman.m_block_index but no active chain. (A few of the // tests when iterating the block tree require that m_chain has been initialized.) - if (m_chain.Height() < 0) { + if (ActiveChain().Height() < 0) { assert(m_blockman.m_block_index.size() <= 1); return; } @@ -5102,13 +5160,13 @@ void Chainstate::CheckBlockIndex() CBlockIndex* pindexFirstNotTransactionsValid = nullptr; // Oldest ancestor of pindex which does not have BLOCK_VALID_TRANSACTIONS (regardless of being valid or not). CBlockIndex* pindexFirstNotChainValid = nullptr; // Oldest ancestor of pindex which does not have BLOCK_VALID_CHAIN (regardless of being valid or not). CBlockIndex* pindexFirstNotScriptsValid = nullptr; // Oldest ancestor of pindex which does not have BLOCK_VALID_SCRIPTS (regardless of being valid or not). + CBlockIndex* pindexFirstAssumeValid = nullptr; // Oldest ancestor of pindex which has BLOCK_ASSUMED_VALID while (pindex != nullptr) { nNodes++; + if (pindexFirstAssumeValid == nullptr && pindex->nStatus & BLOCK_ASSUMED_VALID) pindexFirstAssumeValid = pindex; if (pindexFirstInvalid == nullptr && pindex->nStatus & BLOCK_FAILED_VALID) pindexFirstInvalid = pindex; if (pindexFirstConflicing == nullptr && pindex->nStatus & BLOCK_CONFLICT_CHAINLOCK) pindexFirstConflicing = pindex; - // Assumed-valid index entries will not have data since we haven't downloaded the - // full block yet. - if (pindexFirstMissing == nullptr && !(pindex->nStatus & BLOCK_HAVE_DATA) && !pindex->IsAssumedValid()) { + if (pindexFirstMissing == nullptr && !(pindex->nStatus & BLOCK_HAVE_DATA)) { pindexFirstMissing = pindex; } if (pindexFirstNeverProcessed == nullptr && pindex->nTx == 0) pindexFirstNeverProcessed = pindex; @@ -5137,8 +5195,12 @@ void Chainstate::CheckBlockIndex() // Begin: actual consistency checks. if (pindex->pprev == nullptr) { // Genesis block checks. - assert(pindex->GetBlockHash() == m_params.GetConsensus().hashGenesisBlock); // Genesis block's hash must match. - assert(pindex == m_chain.Genesis()); // The current active chain's genesis block must be this block. + assert(pindex->GetBlockHash() == GetConsensus().hashGenesisBlock); // Genesis block's hash must match. + for (auto c : GetAll()) { + if (c->m_chain.Genesis() != nullptr) { + assert(pindex == c->m_chain.Genesis()); // The chain's genesis block must be this block. + } + } } if (!pindex->HaveTxsDownloaded()) assert(pindex->nSequenceId <= 0); // nSequenceId can't be set positive for blocks that aren't linked (negative is used for preciousblock) // VALID_TRANSACTIONS is equivalent to nTx > 0 for all nodes (whether or not pruning has occurred). @@ -5148,7 +5210,13 @@ void Chainstate::CheckBlockIndex() if (!m_blockman.m_have_pruned && !pindex->IsAssumedValid()) { // If we've never pruned, then HAVE_DATA should be equivalent to nTx > 0 assert(!(pindex->nStatus & BLOCK_HAVE_DATA) == (pindex->nTx == 0)); - assert(pindexFirstMissing == pindexFirstNeverProcessed); + if (pindexFirstAssumeValid == nullptr) { + // If we've got some assume valid blocks, then we might have + // missing blocks (not HAVE_DATA) but still treat them as + // having been processed (with a fake nTx value). Otherwise, we + // can assert that these are the same. + assert(pindexFirstMissing == pindexFirstNeverProcessed); + } } else { // If we have pruned, then we can only say that HAVE_DATA implies nTx > 0 if (pindex->nStatus & BLOCK_HAVE_DATA) assert(pindex->nTx > 0); @@ -5179,30 +5247,35 @@ void Chainstate::CheckBlockIndex() assert((pindex->nStatus & BLOCK_FAILED_MASK) == 0); // The failed mask cannot be set for blocks without invalid parents. } if (pindexFirstConflicing == nullptr) { - // Checks for not-conflciting blocks. + // Checks for non-conflicting blocks. assert((pindex->nStatus & BLOCK_CONFLICT_CHAINLOCK) == 0); // The conflicting mask cannot be set for blocks without conflicting parents. } - if (!CBlockIndexWorkComparator()(pindex, m_chain.Tip()) && pindexFirstNeverProcessed == nullptr) { - if (pindexFirstInvalid == nullptr && pindexFirstConflicing == nullptr) { - const bool is_active = this == &m_chainman.ActiveChainstate(); - - // If this block sorts at least as good as the current tip and - // is valid and we have all data for its parents, it must be in - // setBlockIndexCandidates. m_chain.Tip() must also be there - // even if some data has been pruned. - // - // Don't perform this check for the background chainstate since - // its setBlockIndexCandidates shouldn't have some entries (i.e. those past the - // snapshot block) which do exist in the block index for the active chainstate. - if (is_active && (pindexFirstMissing == nullptr || pindex == m_chain.Tip())) { - assert(setBlockIndexCandidates.count(pindex)); + // Chainstate-specific checks on setBlockIndexCandidates + for (auto c : GetAll()) { + if (c->m_chain.Tip() == nullptr) continue; + if (!CBlockIndexWorkComparator()(pindex, c->m_chain.Tip()) && pindexFirstNeverProcessed == nullptr) { + if (pindexFirstInvalid == nullptr && pindexFirstConflicing == nullptr) { + const bool is_active = c == &ActiveChainstate(); + // If this block sorts at least as good as the current tip and + // is valid and we have all data for its parents, it must be in + // setBlockIndexCandidates. m_chain.Tip() must also be there + // even if some data has been pruned. + // + if ((pindexFirstMissing == nullptr || pindex == c->m_chain.Tip())) { + // The active chainstate should always have this block + // as a candidate, but a background chainstate should + // only have it if it is an ancestor of the snapshot base. + if (is_active || Assert(GetSnapshotBaseBlock())->GetAncestor(pindex->nHeight) == pindex) { + assert(c->setBlockIndexCandidates.count(pindex)); + } + } + // If some parent is missing, then it could be that this block was in + // setBlockIndexCandidates but had to be removed because of the missing data. + // In this case it must be in m_blocks_unlinked -- see test below. } - // If some parent is missing, then it could be that this block was in - // setBlockIndexCandidates but had to be removed because of the missing data. - // In this case it must be in m_blocks_unlinked -- see test below. + } else { // If this block sorts worse than the current tip or some ancestor's block has never been seen, it cannot be in setBlockIndexCandidates. + assert(c->setBlockIndexCandidates.count(pindex) == 0); } - } else { // If this block sorts worse than the current tip or some ancestor's block has never been seen, it cannot be in setBlockIndexCandidates. - assert(setBlockIndexCandidates.count(pindex) == 0); } // Check whether this block is in m_blocks_unlinked. std::pair::iterator,std::multimap::iterator> rangeUnlinked = m_blockman.m_blocks_unlinked.equal_range(pindex->pprev); @@ -5223,18 +5296,23 @@ void Chainstate::CheckBlockIndex() if (pindexFirstMissing == nullptr) assert(!foundInUnlinked); // We aren't missing data for any parent -- cannot be in m_blocks_unlinked. if (pindex->pprev && (pindex->nStatus & BLOCK_HAVE_DATA) && pindexFirstNeverProcessed == nullptr && pindexFirstMissing != nullptr) { // We HAVE_DATA for this block, have received data for all parents at some point, but we're currently missing data for some parent. - assert(m_blockman.m_have_pruned); // We must have pruned. + assert(m_blockman.m_have_pruned || pindexFirstAssumeValid != nullptr); // We must have pruned, or else we're using a snapshot (causing us to have faked the received data for some parent(s)). // This block may have entered m_blocks_unlinked if: // - it has a descendant that at some point had more work than the // tip, and // - we tried switching to that descendant but were missing // data for some intermediate block between m_chain and the // tip. - // So if this block is itself better than m_chain.Tip() and it wasn't in + // So if this block is itself better than any m_chain.Tip() and it wasn't in // setBlockIndexCandidates, then it must be in m_blocks_unlinked. - if (!CBlockIndexWorkComparator()(pindex, m_chain.Tip()) && setBlockIndexCandidates.count(pindex) == 0) { - if (pindexFirstInvalid == nullptr) { - assert(foundInUnlinked); + for (auto c : GetAll()) { + const bool is_active = c == &ActiveChainstate(); + if (!CBlockIndexWorkComparator()(pindex, c->m_chain.Tip()) && c->setBlockIndexCandidates.count(pindex) == 0) { + if (pindexFirstInvalid == nullptr && pindexFirstConflicing == nullptr) { + if (is_active || Assert(GetSnapshotBaseBlock())->GetAncestor(pindex->nHeight) == pindex) { + assert(foundInUnlinked); + } + } } } } @@ -5262,6 +5340,7 @@ void Chainstate::CheckBlockIndex() if (pindex == pindexFirstNotTransactionsValid) pindexFirstNotTransactionsValid = nullptr; if (pindex == pindexFirstNotChainValid) pindexFirstNotChainValid = nullptr; if (pindex == pindexFirstNotScriptsValid) pindexFirstNotScriptsValid = nullptr; + if (pindex == pindexFirstAssumeValid) pindexFirstAssumeValid = nullptr; // Find our parent. CBlockIndex* pindexPar = pindex->pprev; // Find which child we just visited. @@ -5370,12 +5449,8 @@ std::vector ChainstateManager::GetAll() LOCK(::cs_main); std::vector out; - if (!IsSnapshotValidated() && m_ibd_chainstate) { - out.push_back(m_ibd_chainstate.get()); - } - - if (m_snapshot_chainstate) { - out.push_back(m_snapshot_chainstate.get()); + for (Chainstate* cs : {m_ibd_chainstate.get(), m_snapshot_chainstate.get()}) { + if (this->IsUsable(cs)) out.push_back(cs); } return out; @@ -5577,6 +5652,17 @@ bool ChainstateManager::ActivateSnapshot( m_active_chainstate = m_snapshot_chainstate.get(); m_snapshot_chainstate->m_evoDb.SetDefaultIdentity(EvoDbIdentity::SNAPSHOT); + // Move the mempool to the snapshot chainstate: only the active + // chainstate keeps one, so background block connects cannot touch + // mempool state built on the snapshot tip. The mempool is empty at + // this point because snapshot activation happens during IBD. + Assume(!m_snapshot_chainstate->m_mempool); + if (m_ibd_chainstate->m_mempool) { + Assume(m_ibd_chainstate->m_mempool->size() == 0); + m_snapshot_chainstate->m_mempool = m_ibd_chainstate->m_mempool; + m_ibd_chainstate->m_mempool = nullptr; + } + LogPrintf("[snapshot] successfully activated snapshot %s\n", base_blockhash.ToString()); LogPrintf("[snapshot] (%.2f MB)\n", m_snapshot_chainstate->CoinsTip().DynamicMemoryUsage() / (1000 * 1000)); @@ -5597,6 +5683,19 @@ static void FlushSnapshotToDisk(CCoinsViewCache& coins_cache, bool snapshot_load coins_cache.Flush(); } +struct StopHashingException : public std::exception +{ + const char* what() const throw() override + { + return "ComputeUTXOStats interrupted by shutdown."; + } +}; + +static void SnapshotUTXOHashBreakpoint() +{ + if (ShutdownRequested()) throw StopHashingException(); +} + bool ChainstateManager::PopulateAndValidateSnapshot( Chainstate& snapshot_chainstate, AutoFile& coins_file, @@ -5720,13 +5819,18 @@ bool ChainstateManager::PopulateAndValidateSnapshot( assert(coins_cache.GetBestBlock() == base_blockhash); - auto breakpoint_fnc = [] { /* TODO insert breakpoint here? */ }; - // As above, okay to immediately release cs_main here since no other context knows // about the snapshot_chainstate. CCoinsViewDB* snapshot_coinsdb = WITH_LOCK(::cs_main, return &snapshot_chainstate.CoinsDB()); - const std::optional maybe_stats = ComputeUTXOStats(CoinStatsHashType::HASH_SERIALIZED, snapshot_coinsdb, m_blockman, breakpoint_fnc); + std::optional maybe_stats; + + try { + maybe_stats = ComputeUTXOStats( + CoinStatsHashType::HASH_SERIALIZED, snapshot_coinsdb, m_blockman, SnapshotUTXOHashBreakpoint); + } catch (StopHashingException const&) { + return false; + } if (!maybe_stats.has_value()) { LogPrintf("[snapshot] failed to generate coins stats\n"); return false; @@ -5783,13 +5887,49 @@ bool ChainstateManager::PopulateAndValidateSnapshot( index->nChainTx = au_data.nChainTx; snapshot_chainstate.setBlockIndexCandidates.insert(snapshot_start_block); + // Until the loadtxoutset milestone the snapshot carries no Dash payload, + // so the base MN list is only derivable when this node's own background + // chainstate has already validated the base block. On a cold start + // (background tip below the base) it is not derivable at all: attempting + // the lookup would take GetListForBlockInternal's legacy bootstrap branch + // (the dual-chainstate marker is not durable yet at this point), fabricate + // an empty "initial snapshot" list for the base block, and poison the + // shared list cache that the background chainstate later derives base+1 + // from. Capture the lifecycle hashes only when the base state genuinely + // exists; completion skips the comparison when the markers are absent. + // The background chainstate never re-connects a base block it has already + // validated, so RecordBackgroundMNListHash cannot fire for it either -- + // this capture stands in for it. + // TODO(assumeutxo, loadtxoutset): once the snapshot payload carries the + // base MN list, derive the SNAPSHOT-side marker from the payload so it is + // always present and independent of local state. + std::optional base_mn_list_hash; + if (const CBlockIndex* ibd_tip = m_ibd_chainstate->m_chain.Tip(); + ibd_tip != nullptr && ibd_tip->GetBlockHash() == base_blockhash) { + base_mn_list_hash = + snapshot_chainstate.ChainHelper().GetDeterministicMNListHash(snapshot_start_block); + auto db_tx = snapshot_chainstate.m_evoDb.BeginTransaction(::EvoDbIdentity::NORMAL); + snapshot_chainstate.m_evoDb.WriteBackgroundMNListHash(base_blockhash, *base_mn_list_hash); + db_tx->Commit(); + } + + // Snapshot lifecycle recovery depends on the background chainstate's + // independently captured MN-list hash. Make all preceding NORMAL writes + // durable before publishing the snapshot markers. + if (!snapshot_chainstate.m_evoDb.CommitRootTransaction(EvoDbIdentity::NORMAL, /*sync=*/true)) { + LogPrintf("[snapshot] failed to sync background EvoDB state\n"); + return false; + } { auto db_tx = snapshot_chainstate.m_evoDb.BeginTransaction(EvoDbIdentity::SNAPSHOT); snapshot_chainstate.m_evoDb.WriteBestBlock(EvoDbIdentity::SNAPSHOT, base_blockhash); + if (base_mn_list_hash.has_value()) { + snapshot_chainstate.m_evoDb.WriteSnapshotBaseMNListHash(*base_mn_list_hash); + } snapshot_chainstate.m_evoDb.WriteDualChainstateMarker(); db_tx->Commit(); } - if (!snapshot_chainstate.m_evoDb.CommitRootTransaction(EvoDbIdentity::SNAPSHOT)) { + if (!snapshot_chainstate.m_evoDb.CommitRootTransaction(EvoDbIdentity::SNAPSHOT, /*sync=*/true)) { LogPrintf("[snapshot] failed to commit snapshot EvoDB marker\n"); return false; } @@ -5799,6 +5939,218 @@ bool ChainstateManager::PopulateAndValidateSnapshot( return true; } +// Currently, this function holds cs_main for its duration, which could be for +// multiple minutes due to the ComputeUTXOStats call. This hold is necessary +// because we need to avoid advancing the background validation chainstate +// farther than the snapshot base block - and this function is also invoked +// from within ConnectTip, i.e. from within ActivateBestChain, so cs_main is +// held anyway. +// +// Eventually (TODO), we could somehow separate this function's runtime from +// maintenance of the active chain, but that will either require +// +// (i) setting `m_disabled` immediately and ensuring all chainstate accesses go +// through IsUsable() checks, or +// +// (ii) giving each chainstate its own lock instead of using cs_main for everything. +SnapshotCompletionResult ChainstateManager::MaybeCompleteSnapshotValidation( + std::function shutdown_fnc) +{ + AssertLockHeld(cs_main); + if (m_ibd_chainstate.get() == &this->ActiveChainstate() || + !this->IsUsable(m_snapshot_chainstate.get()) || + !this->IsUsable(m_ibd_chainstate.get()) || + !m_ibd_chainstate->m_chain.Tip()) { + // Nothing to do - this function only applies to the background + // validation chainstate. + return SnapshotCompletionResult::SKIPPED; + } + const auto snapshot_base_height_opt = this->GetSnapshotBaseHeight(); + if (!snapshot_base_height_opt) { + if (!m_snapshot_chainstate->CoinsDB().StoragePath()) { + // Some Dash unit fixtures construct a synthetic in-memory snapshot + // chainstate before inserting its base block into the block index. + return SnapshotCompletionResult::SKIPPED; + } + LogPrintf("[snapshot] on-disk snapshot base block is missing from the block index\n"); + return SnapshotCompletionResult::BASE_BLOCKHASH_MISMATCH; + } + const int snapshot_tip_height = this->ActiveHeight(); + const int snapshot_base_height = *snapshot_base_height_opt; + const CBlockIndex& index_new = *Assert(m_ibd_chainstate->m_chain.Tip()); + + if (index_new.nHeight < snapshot_base_height) { + // Background IBD not complete yet. + return SnapshotCompletionResult::SKIPPED; + } + + assert(SnapshotBlockhash()); + uint256 snapshot_blockhash = *Assert(SnapshotBlockhash()); + + // Completion is serialized by cs_main. Flush each identity in sequence so + // CEvoDB's single-open-transaction invariant is preserved and marker + // promotion can atomically operate on fully committed transaction trees. + m_ibd_chainstate->ForceFlushStateToDisk(); + m_snapshot_chainstate->ForceFlushStateToDisk(); + if (!m_ibd_chainstate->m_evoDb.CommitRootTransaction(EvoDbIdentity::NORMAL, /*sync=*/true) || + !m_snapshot_chainstate->m_evoDb.CommitRootTransaction(EvoDbIdentity::SNAPSHOT, /*sync=*/true)) { + // A failed sync here is an unrecoverable database write error, and the + // caller (ConnectTip) discards the result: with the background tip + // already at the base, nothing would retry completion until restart. + // Abort like the other unrecoverable EvoDB paths. + AbortNode("Failed to sync EvoDB state for snapshot completion"); + return SnapshotCompletionResult::STATS_FAILED; + } + + auto handle_invalid_snapshot = [&]() EXCLUSIVE_LOCKS_REQUIRED(::cs_main) { + bilingual_str user_error = strprintf(_( + "%s failed to validate the -assumeutxo snapshot state. " + "This indicates a hardware problem, or a bug in the software, or a " + "bad software modification that allowed an invalid snapshot to be " + "loaded. As a result of this, the node will shut down and stop using any " + "state that was built on the snapshot, resetting the chain height " + "from %d to %d. On the next " + "restart, the node will resume syncing from %d " + "without using any snapshot data. " + "Please report this incident to %s, including how you obtained the snapshot. " + "The invalid snapshot chainstate will be left on disk in case it is " + "helpful in diagnosing the issue that caused this error."), + PACKAGE_NAME, snapshot_tip_height, snapshot_base_height, snapshot_base_height, PACKAGE_BUGREPORT + ); + + LogPrintf("[snapshot] !!! %s\n", user_error.original); + LogPrintf("[snapshot] deleting snapshot, reverting to validated chain, and stopping node\n"); + + m_active_chainstate = m_ibd_chainstate.get(); + // Hand the mempool back so the again-active background chainstate owns + // it for the remainder of this (shutting-down) run. + m_ibd_chainstate->m_mempool = m_snapshot_chainstate->m_mempool; + m_snapshot_chainstate->m_mempool = nullptr; + m_snapshot_chainstate->m_disabled = true; + assert(!this->IsUsable(m_snapshot_chainstate.get())); + assert(this->IsUsable(m_ibd_chainstate.get())); + + auto rename_result = m_snapshot_chainstate->InvalidateCoinsDBOnDisk(); + if (!rename_result) { + user_error += Untranslated("\n") + util::ErrorString(rename_result); + } else if (!m_ibd_chainstate->m_evoDb.DiscardSnapshotMarkers()) { + LogPrintf("[snapshot] failed to remove invalid snapshot EvoDB markers\n"); + } + + shutdown_fnc(user_error); + }; + + if (index_new.GetBlockHash() != snapshot_blockhash) { + LogPrintf("[snapshot] supposed base block %s does not match the " /* Continued */ + "snapshot base block %s (height %d). Snapshot is not valid.", + index_new.ToString(), snapshot_blockhash.ToString(), snapshot_base_height); + handle_invalid_snapshot(); + return SnapshotCompletionResult::BASE_BLOCKHASH_MISMATCH; + } + + assert(index_new.nHeight == snapshot_base_height); + + int curr_height = m_ibd_chainstate->m_chain.Height(); + + assert(snapshot_base_height == curr_height); + assert(snapshot_base_height == index_new.nHeight); + assert(this->IsUsable(m_snapshot_chainstate.get())); + assert(this->GetAll().size() == 2); + + CCoinsViewDB& ibd_coins_db = m_ibd_chainstate->CoinsDB(); + + auto maybe_au_data = ExpectedAssumeutxo(curr_height, ::Params()); + if (!maybe_au_data) { + LogPrintf("[snapshot] assumeutxo data not found for height " /* Continued */ + "(%d) - refusing to validate snapshot\n", curr_height); + handle_invalid_snapshot(); + return SnapshotCompletionResult::MISSING_CHAINPARAMS; + } + + const AssumeutxoData& au_data = *maybe_au_data; + std::optional maybe_ibd_stats; + LogPrintf("[snapshot] computing UTXO stats for background chainstate to validate " /* Continued */ + "snapshot - this could take a few minutes\n"); + try { + maybe_ibd_stats = ComputeUTXOStats( + CoinStatsHashType::HASH_SERIALIZED, + &ibd_coins_db, + m_blockman, + SnapshotUTXOHashBreakpoint); + } catch (StopHashingException const&) { + return SnapshotCompletionResult::STATS_FAILED; + } + + // XXX note that this function is slow and will hold cs_main for potentially minutes. + if (!maybe_ibd_stats) { + LogPrintf("[snapshot] failed to generate stats for validation coins db\n"); + // While this isn't a problem with the snapshot per se, this condition + // prevents us from validating the snapshot, so we should shut down and let the + // user handle the issue manually. + handle_invalid_snapshot(); + return SnapshotCompletionResult::STATS_FAILED; + } + const auto& ibd_stats = *maybe_ibd_stats; + + // Compare the background validation chainstate's UTXO set hash against the hard-coded + // assumeutxo hash we expect. + // + // TODO: For belt-and-suspenders, we could cache the UTXO set + // hash for the snapshot when it's loaded in its chainstate's leveldb. We could then + // reference that here for an additional check. + if (AssumeutxoHash{ibd_stats.hashSerialized} != au_data.hash_serialized) { + LogPrintf("[snapshot] hash mismatch: actual=%s, expected=%s\n", + ibd_stats.hashSerialized.ToString(), + au_data.hash_serialized.ToString()); + handle_invalid_snapshot(); + return SnapshotCompletionResult::HASH_MISMATCH; + } + + // The snapshot marker records the derived deterministic-MN state that was + // available when the snapshot chainstate began using the base block. Compare + // it with the state independently derived by background validation. + // + // TODO(assumeutxo, M4-B4): extend the snapshot format and this comparison to + // the CbTx merkleRootMNList, merkleRootQuorums, and creditPool commitments. + uint256 snapshot_mn_list_hash; + if (!m_ibd_chainstate->m_evoDb.ReadSnapshotBaseMNListHash(snapshot_mn_list_hash)) { + // Cold-start activation could not capture the base MN list (the + // snapshot format carries no Dash payload yet), so there is nothing to + // compare against. The UTXO-set hash above remains the completion + // criterion, exactly as upstream. + LogPrintf("[snapshot] no base MN-list marker was captured at activation; skipping deterministic MN-list comparison\n"); + } else { + uint256 background_mn_list_block; + uint256 background_mn_list_hash; + if (!m_ibd_chainstate->m_evoDb.ReadBackgroundMNListHash( + background_mn_list_block, background_mn_list_hash) || + background_mn_list_block != snapshot_blockhash || + snapshot_mn_list_hash != background_mn_list_hash) { + LogPrintf("[snapshot] deterministic MN list mismatch at base block: captured_block=%s, actual=%s, expected=%s\n", + background_mn_list_block.ToString(), background_mn_list_hash.ToString(), + snapshot_mn_list_hash.ToString()); + handle_invalid_snapshot(); + return SnapshotCompletionResult::EVO_STATE_MISMATCH; + } + } + + const uint256 snapshot_tip = m_snapshot_chainstate->CoinsTip().GetBestBlock(); + if (!m_ibd_chainstate->m_evoDb.VerifyBestBlock(EvoDbIdentity::NORMAL, snapshot_blockhash) || + !m_snapshot_chainstate->m_evoDb.VerifyBestBlock(EvoDbIdentity::SNAPSHOT, snapshot_tip)) { + LogPrintf("[snapshot] EvoDB best-block markers do not match their chainstate tips\n"); + handle_invalid_snapshot(); + return SnapshotCompletionResult::EVO_STATE_MISMATCH; + } + + LogPrintf("[snapshot] snapshot beginning at %s has been fully validated\n", + snapshot_blockhash.ToString()); + + m_ibd_chainstate->m_disabled = true; + this->MaybeRebalanceCaches(); + + return SnapshotCompletionResult::SUCCESS; +} + Chainstate& ChainstateManager::ActiveChainstate() const { LOCK(::cs_main); @@ -5815,7 +6167,7 @@ bool ChainstateManager::IsSnapshotActive() const bool ChainstateManager::IsSnapshotActiveAndUnvalidated() const { LOCK(::cs_main); - return m_snapshot_chainstate && m_active_chainstate == m_snapshot_chainstate.get() && !m_snapshot_validated; + return m_snapshot_chainstate && m_active_chainstate == m_snapshot_chainstate.get() && !IsSnapshotValidated(); } bool ChainstateManager::IsQuorumTypeEnabled(const Consensus::LLMQType llmqType, @@ -5878,17 +6230,22 @@ bool ChainstateManager::IsQuorumTypeEnabled(const Consensus::LLMQType llmqType, void ChainstateManager::MaybeRebalanceCaches() { AssertLockHeld(::cs_main); - if (m_ibd_chainstate && !m_snapshot_chainstate) { - // Allocate everything to the IBD chainstate. This will always happen - // when we are not using a snapshot + bool ibd_usable = this->IsUsable(m_ibd_chainstate.get()); + bool snapshot_usable = this->IsUsable(m_snapshot_chainstate.get()); + assert(ibd_usable || snapshot_usable); + + if (ibd_usable && !snapshot_usable) { + LogPrintf("[snapshot] allocating all cache to the IBD chainstate\n"); + // Allocate everything to the IBD chainstate. m_ibd_chainstate->ResizeCoinsCaches(m_total_coinstip_cache, m_total_coinsdb_cache); } - else if (m_snapshot_chainstate && !m_ibd_chainstate) { + else if (snapshot_usable && !ibd_usable) { + // If background validation has completed and snapshot is our active chain... LogPrintf("[snapshot] allocating all cache to the snapshot chainstate\n"); // Allocate everything to the snapshot chainstate. m_snapshot_chainstate->ResizeCoinsCaches(m_total_coinstip_cache, m_total_coinsdb_cache); } - else if (m_ibd_chainstate && m_snapshot_chainstate) { + else if (ibd_usable && snapshot_usable) { // If both chainstates exist, determine who needs more cache based on IBD status. // // Note: shrink caches first so that we don't inadvertently overwhelm available memory. @@ -5982,6 +6339,169 @@ Chainstate* ChainstateManager::ActivateExistingSnapshot(CTxMemPool* mempool, uin base_blockhash); LogPrintf("[snapshot] switching active chainstate to %s\n", m_snapshot_chainstate->ToString()); m_active_chainstate = m_snapshot_chainstate.get(); + // Only the active chainstate keeps the mempool: the background chainstate + // connects historical blocks and must not touch mempool state built on the + // snapshot tip (e.g. removeExpiredAssetUnlock with a lower height). + if (m_ibd_chainstate) { + m_ibd_chainstate->m_mempool = nullptr; + } evo_db.SetDefaultIdentity(EvoDbIdentity::SNAPSHOT); return m_snapshot_chainstate.get(); } +util::Result Chainstate::InvalidateCoinsDBOnDisk() +{ + AssertLockHeld(::cs_main); + // Should never be called on a non-snapshot chainstate. + assert(m_from_snapshot_blockhash); + auto storage_path_maybe = this->CoinsDB().StoragePath(); + // Should never be called with a non-existent storage path. + assert(storage_path_maybe); + const fs::path& snapshot_datadir = *storage_path_maybe; + + // Coins views no longer usable. + m_coins_views.reset(); + + auto invalid_path = snapshot_datadir + "_INVALID"; + std::string dbpath = fs::PathToString(snapshot_datadir); + std::string target = fs::PathToString(invalid_path); + LogPrintf("[snapshot] renaming snapshot datadir %s to %s\n", dbpath, target); + + // The invalid snapshot datadir is simply moved and not deleted because we may + // want to do forensics later during issue investigation. The user is instructed + // accordingly in MaybeCompleteSnapshotValidation(). + try { + fs::rename(snapshot_datadir, invalid_path); + DirectoryCommit(snapshot_datadir.parent_path()); + } catch (const fs::filesystem_error& e) { + auto src_str = fs::PathToString(snapshot_datadir); + auto dest_str = fs::PathToString(invalid_path); + + LogPrintf("%s: error renaming file '%s' -> '%s': %s\n", + __func__, src_str, dest_str, e.what()); + return util::Error{strprintf(_( + "Rename of '%s' -> '%s' failed. " + "You should resolve this by manually moving or deleting the invalid " + "snapshot directory %s, otherwise you will encounter the same error again " + "on the next startup."), + src_str, dest_str, src_str)}; + } + return {}; +} + +const CBlockIndex* ChainstateManager::GetSnapshotBaseBlock() const +{ + // Deliberately bypass Chainstate::SnapshotBase(), which Asserts a missing + // base block out of existence: startup snapshot completion must be able to + // observe "base not in the block index" and fail with + // BASE_BLOCKHASH_MISMATCH instead of aborting the node. Callers that + // require existence Assert at the call site. + if (!m_active_chainstate || !m_active_chainstate->m_from_snapshot_blockhash) { + return nullptr; + } + return m_blockman.LookupBlockIndex(*m_active_chainstate->m_from_snapshot_blockhash); +} + +std::optional ChainstateManager::GetSnapshotBaseHeight() const +{ + const CBlockIndex* base = this->GetSnapshotBaseBlock(); + return base ? std::make_optional(base->nHeight) : std::nullopt; +} + +bool ChainstateManager::ValidatedSnapshotCleanup() +{ + AssertLockHeld(::cs_main); + auto get_storage_path = [](auto& chainstate) EXCLUSIVE_LOCKS_REQUIRED(::cs_main) -> std::optional { + if (!(chainstate && chainstate->HasCoinsViews())) { + return {}; + } + return chainstate->CoinsDB().StoragePath(); + }; + std::optional ibd_chainstate_path_maybe = get_storage_path(m_ibd_chainstate); + std::optional snapshot_chainstate_path_maybe = get_storage_path(m_snapshot_chainstate); + + if (!this->IsSnapshotValidated()) { + // No need to clean up. + return false; + } + // If either path doesn't exist, that means at least one of the chainstates + // is in-memory, in which case we can't do on-disk cleanup. You'd better be + // in a unittest! + if (!ibd_chainstate_path_maybe || !snapshot_chainstate_path_maybe) { + LogPrintf("[snapshot] snapshot chainstate cleanup cannot happen with " /* Continued */ + "in-memory chainstates. You are testing, right?\n"); + return false; + } + + const auto& snapshot_chainstate_path = *snapshot_chainstate_path_maybe; + const auto& ibd_chainstate_path = *ibd_chainstate_path_maybe; + + const uint256 snapshot_tip = m_snapshot_chainstate->CoinsTip().GetBestBlock(); + CEvoDB& evo_db = m_snapshot_chainstate->m_evoDb; + + // Since we're going to be moving around the underlying leveldb filesystem content + // for each chainstate, make sure that the chainstates (and their constituent + // CoinsViews members) have been destructed first. + // + // The caller of this method will be responsible for reinitializing chainstates + // if they want to continue operation. + this->ResetChainstates(); + + // No chainstates should be considered usable. + assert(this->GetAll().size() == 0); + + LogPrintf("[snapshot] deleting background chainstate directory (now unnecessary) (%s)\n", + fs::PathToString(ibd_chainstate_path)); + + fs::path tmp_old{ibd_chainstate_path + "_todelete"}; + + auto rename_failed_abort = []( + fs::path p_old, + fs::path p_new, + const fs::filesystem_error& err) { + LogPrintf("%s: error renaming file (%s): %s\n", + __func__, fs::PathToString(p_old), err.what()); + AbortNode(strprintf( + "Rename of '%s' -> '%s' failed. " + "Cannot clean up the background chainstate leveldb directory.", + fs::PathToString(p_old), fs::PathToString(p_new))); + }; + + try { + fs::rename(ibd_chainstate_path, tmp_old); + DirectoryCommit(ibd_chainstate_path.parent_path()); + } catch (const fs::filesystem_error& e) { + rename_failed_abort(ibd_chainstate_path, tmp_old, e); + throw; + } + + LogPrintf("[snapshot] moving snapshot chainstate (%s) to " /* Continued */ + "default chainstate directory (%s)\n", + fs::PathToString(snapshot_chainstate_path), fs::PathToString(ibd_chainstate_path)); + + try { + fs::rename(snapshot_chainstate_path, ibd_chainstate_path); + DirectoryCommit(snapshot_chainstate_path.parent_path()); + } catch (const fs::filesystem_error& e) { + rename_failed_abort(snapshot_chainstate_path, ibd_chainstate_path, e); + throw; + } + + // Only after both directory renames are durable can the snapshot marker be + // promoted to NORMAL and the lifecycle metadata be removed. + if (!evo_db.PromoteSnapshotMarkers(snapshot_tip)) { + LogPrintf("[snapshot] failed to promote snapshot EvoDB markers during cleanup\n"); + return false; + } + + if (!DeleteCoinsDBFromDisk(tmp_old, /*is_snapshot=*/false)) { + // No need to AbortNode because once the unneeded bg chainstate data is + // moved, it will not interfere with subsequent initialization. + LogPrintf("Deletion of %s failed. Please remove it manually, as the " /* Continued */ + "directory is now unnecessary.\n", + fs::PathToString(tmp_old)); + } else { + LogPrintf("[snapshot] deleted background chainstate directory (%s)\n", + fs::PathToString(ibd_chainstate_path)); + } + return true; +} diff --git a/src/validation.h b/src/validation.h index a2c0ae83dac1..f2a8be4cae64 100644 --- a/src/validation.h +++ b/src/validation.h @@ -24,6 +24,7 @@ #include #include