Skip to content

[Subscription] Fix WAL replay after leader failover - #18474

Open
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:fix/subscription-wal-replay-failover
Open

[Subscription] Fix WAL replay after leader failover#18474
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:fix/subscription-wal-replay-failover

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Problem

After an IoTConsensus leader failover, the new leader can replay WAL entries replicated from the former leader. These entries carry valid writer progress but keep searchIndex = -1 because they do not have a local search index.

Replay-start lookup treated an uncovered follower entry before the first locally indexed entry as non-replayable and returned LOCATE_MISS. This surfaced as subscription error 1904 when one DataNode went down in a replicated cluster.

Fix

  • Use only non-negative local search indexes to choose the local replay cursor.
  • Keep replicated follower entries visible to ProgressWALIterator; writer progress continues to deduplicate already committed entries.
  • Preserve the recovery writer progress when the WAL contains only uncovered follower entries.
  • Add regression coverage for follower entries before a local lower bound and for the follower-only boundary.

No client change is required for this case. With another provider available, the existing consumer poll path marks the failed provider unavailable and continues failover; an application may temporarily receive an empty poll result, but it does not need a special retry for error 1904.

Validation

  • mvn spotless:apply -pl iotdb-core/datanode
  • mvn test -pl iotdb-core/datanode "-Dtest=ConsensusPrefetchingQueueTest,ProgressWALIteratorTest,ConsensusPrefetchingQueueWalBackpressureTest"
    • 40 tests run, 0 failures, 0 errors, 1 existing conditional skip
  • mvn checkstyle:check -pl iotdb-core/datanode
  • git diff --check

This PR has:

  • been self-reviewed.
  • added comments explaining the intent of the non-obvious replay behavior.
  • added or updated unit tests to cover the changed code paths.

Key changed/added classes
  • ConsensusPrefetchingQueue
  • ConsensusPrefetchingQueueTest
  • ProgressWALIteratorTest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant