Skip to content

fix(consensus/XDPoS): rebuild missing V2 snapshot and deduplication - #2475

Open
gzliudan wants to merge 2 commits into
XinFinOrg:dev-upgradefrom
gzliudan:rebuild-snapshot
Open

fix(consensus/XDPoS): rebuild missing V2 snapshot and deduplication#2475
gzliudan wants to merge 2 commits into
XinFinOrg:dev-upgradefrom
gzliudan:rebuild-snapshot

Conversation

@gzliudan

@gzliudan gzliudan commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Proposed changes

This PR adds a recovery path for missing V2 gap snapshots and prevents redundant concurrent snapshot rebuilds.

If a node is missing a persisted snapshot because the database was written before the ordering fix, getSnapshot now reconstructs the snapshot from the committed state trie for V2-era gap blocks, persists the rebuilt snapshot to LevelDB, and caches it in memory. A singleflight guard deduplicates concurrent rebuild requests so multiple vote handlers do not rebuild the same snapshot at the same time.

This reduces the risk of nodes silently falling out of consensus during affected epochs and keeps snapshot reconstruction aligned with the existing trie-based snapshot generation logic.

Types of changes

What types of changes does your code introduce to XDC network?
Put an in the boxes that apply

  • build: Changes that affect the build system or external dependencies
  • ci: Changes to CI configuration files and scripts
  • chore: Changes that don't change source code or tests
  • docs: Documentation only changes
  • feat: A new feature
  • fix: A bug fix
  • perf: A code change that improves performance
  • refactor: A code change that neither fixes a bug nor adds a feature
  • revert: Revert something
  • style: Changes that do not affect the meaning of the code
  • test: Adding missing tests or correcting existing tests

Impacted Components

Which parts of the codebase does this PR touch?
Put an in the boxes that apply

  • Consensus
  • Account
  • Network
  • Geth
  • Smart Contract
  • External components
  • Not sure (Please specify below)

Checklist

Put an in the boxes once you have confirmed below actions (or provide reasons on not doing so) that

  • This PR has sufficient test coverage (unit/integration test) OR I have provided reason in the PR description for not having test coverage
  • Tested on a private network from the genesis block and monitored the chain operating correctly for multiple epochs.
  • Provide an end-to-end test plan in the PR description on how to manually test it on the devnet/testnet.
  • Tested the backwards compatibility.
  • Tested with XDC nodes running this version co-exist with those running the previous version.
  • Relevant documentation has been updated as part of this PR
  • N/A

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a consensus liveness failure in XDPoS v2 where a missing on-disk “gap snapshot” (e.g., due to an unclean shutdown between head write and snapshot persistence) causes getSnapshot to error repeatedly and drop a node out of consensus for up to an epoch window.

Changes:

  • Adds an optional chainStateReader extension interface to open committed state by root without changing consensus.ChainReader.
  • Implements rebuildSnapshotFromState to reconstruct a missing gap snapshot directly from the committed state trie and persist it back to LevelDB + in-memory LRU.
  • Extends getSnapshot to attempt this self-healing only for V2-era gap blocks (gapBlockNum > SwitchBlock), preserving the pre-Initial() behavior around the V1→V2 boundary.

Comment thread consensus/XDPoS/engines/engine_v2/snapshot.go Outdated
Comment thread consensus/XDPoS/engines/engine_v2/snapshot.go
Comment thread consensus/XDPoS/engines/engine_v2/snapshot.go Outdated
@coderabbitai

coderabbitai Bot commented Jul 20, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c642f5b8-9a0d-4ec8-aa67-77e9708107f7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gzliudan
gzliudan force-pushed the rebuild-snapshot branch 4 times, most recently from d3d77c6 to d29f5ec Compare July 20, 2026 15:22
@gzliudan
gzliudan requested a review from Copilot July 20, 2026 15:56

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

consensus/XDPoS/engines/engine_v2/snapshot.go:94

  • The sort comparator must be a strict ordering. Using >= 0 means equal stakes will return true for both (i,j) and (j,i), which violates the sort.Interface contract and can lead to undefined ordering.

Use a strict > 0 comparison, and add a deterministic tie-breaker (e.g., address) so rebuilt snapshots are stable across nodes when stakes are equal.

	xdc_sort.Slice(pairs, func(i, j int) bool {
		return pairs[i].Stake.Cmp(pairs[j].Stake) >= 0
	})

consensus/XDPoS/engines/engine_v2/snapshot.go:76

  • The added tests do not exercise rebuildSnapshotFromState or the new self-healing path in getSnapshot (they only validate mock helper methods / singleflight behavior). This risks giving a false sense of coverage for consensus-critical recovery logic.

Consider adding a unit test that builds a real state.StateDB (in-memory), populates the validator contract storage slots (candidates + caps), and calls rebuildSnapshotFromState, asserting the persisted snapshot order and DB write.

// rebuildSnapshotFromState reconstructs a missing gap-block snapshot by reading
// candidate addresses and stakes directly from the committed state trie at
// gapRoot. It uses the same StateDB slot-read path as core.BlockChain. UpdateM1
// (when it can read candidates from state) and eth/downloader.generateSnapshot,
// so no EVM client is required. On success the snapshot is persisted to the
// database and added to the in-memory LRU cache.
func (x *XDPoS_v2) rebuildSnapshotFromState(sr chainStateReader, number uint64, hash common.Hash, gapRoot common.Hash) (*SnapshotV2, error) {

Comment thread consensus/XDPoS/engines/engine_v2/snapshot.go
Comment thread consensus/XDPoS/engines/engine_v2/snapshot_test.go Outdated
Comment thread consensus/XDPoS/engines/engine_v2/snapshot_test.go Outdated
@gzliudan
gzliudan force-pushed the rebuild-snapshot branch 2 times, most recently from 4c64d74 to d492c72 Compare July 20, 2026 16:53
@gzliudan gzliudan changed the title fix(consensus/XDPoS): self-heal missing V2 gap snapshot from state trie fix(consensus/XDPoS): self-heal and deduplicate V2 snapshot reconstruction Jul 20, 2026
@gzliudan
gzliudan requested a review from Copilot July 20, 2026 16:56

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

consensus/XDPoS/engines/engine_v2/snapshot.go:97

  • The sort comparator used for rebuilding snapshots is not a strict ordering: it returns true even when stakes are equal (Cmp == 0). Go's sort requires a strict weak ordering; violating it can produce non-deterministic ordering across Go versions/architectures, which is risky for consensus because candidate ordering can change when stakes tie. Add a deterministic tie-breaker (e.g., by address) and use > 0 for the stake comparison.
	xdc_sort.Slice(pairs, func(i, j int) bool {
		return pairs[i].Stake.Cmp(pairs[j].Stake) >= 0
	})

Comment thread consensus/XDPoS/engines/engine_v2/snapshot_test.go Outdated
Comment thread consensus/XDPoS/engines/engine_v2/snapshot_test.go Outdated
@gzliudan
gzliudan force-pushed the rebuild-snapshot branch 2 times, most recently from bbb3139 to 6997de9 Compare July 21, 2026 00:28
@gzliudan
gzliudan requested a review from Copilot July 21, 2026 00:28

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

consensus/XDPoS/engines/engine_v2/snapshot.go:97

  • The sort comparator used to order candidates is not a strict ordering (>= 0). For equal stakes it returns true in both directions, which violates sort.Less requirements and can yield non-deterministic ordering across nodes (consensus-critical). Use > 0 for stake ordering and add a deterministic tie-breaker (e.g., address comparison).
	xdc_sort.Slice(pairs, func(i, j int) bool {
		return pairs[i].Stake.Cmp(pairs[j].Stake) >= 0
	})

consensus/XDPoS/engines/engine_v2/snapshot_test.go:56

  • The new self-healing logic (rebuildSnapshot + singleflight path in getSnapshot) is not covered by tests. The added tests here only re-test snapshot (de)serialization and candidate membership, and the mock types below are currently unused, so regressions in the rebuild path (e.g., candidate ordering, persistence, singleflight reuse) would go unnoticed.
// ============================================================================
// Mock implementations for self-healing tests
// ============================================================================

// mockStateDB is a minimal mock implementation of state.StateDB
// for testing snapshot reconstruction from state trie
type mockStateDB struct {
	candidates  []common.Address
	stakes      map[common.Address]*big.Int

consensus/XDPoS/engines/engine_v2/snapshot_test.go:86

  • The mock mockChainStateReader currently cannot return a successful StateAt result (it always returns an error), and mockStateDB can't actually be used where *state.StateDB is required. As written, this mock section is dead code/misleading; either remove it or replace it with a test setup that can construct a real *state.StateDB for the rebuild success path.
// mockChainStateReader is a mock implementation of chainStateReader
// that returns a mock StateDB
type mockChainStateReader struct {
	stateDB      *mockStateDB
	shouldError  bool
	errorMessage string
}

func (m *mockChainStateReader) StateAt(root common.Hash) (*state.StateDB, error) {
	if m.shouldError {
		return nil, errors.New(m.errorMessage)
	}
	return nil, errors.New("mockChainStateReader: no StateDB configured")
}

@gzliudan gzliudan changed the title fix(consensus/XDPoS): self-heal and deduplicate V2 snapshot reconstruction fix(consensus/XDPoS): rebuild missing V2 snapshot Jul 21, 2026
@gzliudan gzliudan changed the title fix(consensus/XDPoS): rebuild missing V2 snapshot fix(consensus/XDPoS): rebuild missing V2 snapshot and deduplication Jul 21, 2026
@gzliudan
gzliudan force-pushed the rebuild-snapshot branch 2 times, most recently from 2be0535 to e6dea92 Compare July 23, 2026 06:35
gzliudan added 2 commits July 23, 2026 18:00
Nodes whose database pre-dates the writeBlockWithState ordering fix may
have a gap snapshot missing from leveldb (process was killed between
writeHeadBlock and StoreSnapshot). Every call to getSnapshot for any block
in the affected epoch then returns 'Cannot find snapshot from last gap block',
silently dropping the node out of consensus for up to 900 blocks.

Add a self-healing path in getSnapshot: when loadSnapshot returns an error
and the gap block is a V2-era block (gapBlockNum > SwitchBlock), attempt to
rebuild the snapshot by opening the committed state trie at gapHeader.Root
and reading candidates/stakes directly (same trie-read path used by
updateM1ForBlock and downloader.generateSnapshot). On success the rebuilt
snapshot is persisted to leveldb and added to the in-memory LRU cache, so
subsequent calls hit the fast path.

Self-healing is intentionally skipped for gap blocks at or before SwitchBlock:
those initial V2 snapshots are created by Initial() from the epoch-switch
block's validator list, not from the state trie, and tests verify that
GetSnapshot returns nil before Initial() is called.

The chain.(chainStateReader) type assertion ensures lightweight mock
ChainReaders used in tests gracefully skip self-healing without any interface
changes.

Snapshot reconstruction follows the same state-trie read path as updateM1 and
downloader.generateSnapshot, ensuring consistency. The rebuilding process is
carefully guarded: only V2-era gap blocks (number > SwitchBlock) attempt
self-healing, allowing Initial() to remain the authoritative source for
initial V2 snapshots. On success, the rebuilt snapshot is immediately
persisted to leveldb and cached in memory for subsequent fast-path hits.

Tests added to verify:
- Snapshot persistence and retrieval from database
- Candidate sorting and candidate list queries
- Empty snapshot and edge case handling
- Mock infrastructure for testing self-healing logic (chainStateReader, mockStateDB)
…t reconstruction

Add TestSnapshot_SingleflightDeduplication to verify singleflight is correctly
integrated into the snapshot reconstruction path. The test verifies that:

1. Multiple concurrent goroutines can safely call singleflight.Do
2. All goroutines receive consistent snapshot data
3. The API is used correctly per golang.org/x/sync/singleflight patterns

In production, when multiple P2P message handlers concurrently process votes
for the same gap block, singleflight prevents redundant snapshot rebuilds by
deduplicating concurrent requests. This test verifies the API integration is
correct (result consistency), though full concurrency deduplication is best
validated in integration tests with realistic P2P message timing.

The test validates:
- No errors when multiple goroutines call Do with same key
- Consistent results returned to all callers
- Correct API usage per golang.org/x/sync/singleflight
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants