Blocks lost during fast payload ingest: bodies unavailable in the replayed range (geth unaffected on the same CL)

[artemrootman]

Summary

On Morph mainnet, morph-reth loses a subset of blocks while derivation runs ahead of the network's block rate — i.e. whenever a node catches up after being bootstrapped from a snapshot. The affected blocks are gone from storage: they are not retrievable by number or by hash, and state queries at those heights fail. Blocks that came from the snapshot itself, and blocks ingested at normal tip rate, are stored correctly.

go-ethereum (morph 2.2.5) driven by the same morph-node on the same host does not lose blocks in the same ranges.

The node contradicts its own storage

This needs no external comparison — two queries against the affected node:

> eth_getBlockByNumber("0x17d784d", false)          # 25 000 013
{"hash":"0x0d351cf5fe7fc62a3ebb5019fa411995b716a10cee264edda0fa9ab91bd11205",
 "parentHash":"0x7307c4ef6cbb20a90bc61e165ebff9cf9228f294808597d4d31b9e12137e9f3c", ...}

> eth_getBlockByHash("0x7307c4ef6cbb20a90bc61e165ebff9cf9228f294808597d4d31b9e12137e9f3c", false)
{"result": null}

The node serves a block and cannot produce that block's parent. On https://rpc.morphl2.io that hash is block 25 000 012 with one transaction.

The missing blocks form contiguous runs — here 25 000 005 … 25 000 012, with 25 000 004 and 25 000 013 both present. For blocks in a run:

• eth_getBlockByNumber → null
• eth_getBlockByHash → null (so this is not a number→hash lookup issue)
• eth_getBlockTransactionCountByNumber → null
• eth_getBalance → -32001 block not found: 0x17d7845

Execution itself is unaffected: the head hash always matches the public RPC, and state and receipts around the gaps are correct.

Environment

• morph-reth v1.1.0. The published image cannot start (Docker images v1.1.0 and latest fail to start: GLIBC_2.38 / GLIBC_2.39 not found #151), so this is the official morph-reth-1.1.0-x86_64-linux.tar.gz release binary on ubuntu:24.04.
• ghcr.io/morph-l2/node:0.6.0
• Morph mainnet, archive, no pruning flags, storage v2
• Bootstrapped from snapshot-archive-reth-20260803-1 using a paired reth + node restore at the same height (24 980 107), as the README requires: reth-data/ → EL datadir, data/ → node-data. Nothing was started on the extracted copy beforehand.

With that bootstrap morph-node drives the EL through the expected path (ApplyBlockV2 success per block, EL and CL advancing together) and derivation reached the tip in about 11 minutes, roughly 309 blocks/s.

Measurements

A block counts as absent only when the node answers {"result": null}; JSON-RPC errors are counted separately so transient failures are not mistaken for missing data. rpc_errors was 0 in every scan below.

Same node, blocks grouped by how they arrived:

how the blocks got in	window	absent
from the snapshot (CL stopped, no derivation)	24 979 507–24 980 107	0 / 601
from the snapshot	24 900 000–24 900 600	0 / 601
from the snapshot	24 664 700–24 665 300	0 / 601
ingested while catching up	24 980 157–24 980 230	7 / 74
ingested while catching up	25 000 000–25 000 600	32 / 601
ingested while catching up	25 100 000–25 100 600	16 / 601
ingested at tip rate (last 141 blocks)	head−150 … head−10	0 / 141

The first missing block sits only 59 blocks above the snapshot base, so loss begins almost as soon as derivation outpaces the network.

Control — go-ethereum 2.2.5, same host, same morph-node, its own datadir, started with no CL attached:

window	absent
24 664 700–24 665 300	0 / 601
24 900 000–24 900 600	0 / 601
25 000 000–25 000 600	0 / 601

Where the loss happens

Both import entry points in crates/engine-api/src/builder.rs are strict about continuity:

• new_l2_block requires data.number == current_head.number + 1 and data.parent_hash == current_head.hash, otherwise it returns DiscontinuousBlockNumber / WrongParentHash, or logs ignoring past block number when the number is behind.
• new_l2_block_v2, which is what morph-node uses (ApplyBlockV2 in its logs), resolves the parent with sealed_header_by_hash and fails with parent block not found if it is missing, then requires data.number == parent.number() + 1.

A block therefore cannot be skipped silently at this boundary — a skip would surface as an error on the very next block. Across 5 037 136 log lines captured from the affected runs there is not one occurrence of discontinuous, wrong parent hash, parent block not found or ignoring past block. As a control, matching known strings (Canonical chain committed, reference index) over the same data returns 1 181 752 hits, so the search itself is sound.

So every block is accepted, executed and made canonical, and the loss happens after acceptance — on the persistence path, presumably in import_l2_block_via_engine and the engine tree's handling of a stream of one newPayload + forkchoiceUpdated per block arriving faster than the persistence task drains.

This matches the symptoms: the canonical chain stays linked by headers (otherwise sealed_header_by_hash for the child of a missing block would have failed and derivation would have stopped), state and receipts around the gaps are correct, and only retrieval of the block itself fails — by number and by hash alike.

Things that do not help

--engine.persistence-threshold 0. The obvious theory was blocks piling up in memory ahead of the last persisted block. Re-ran the full bootstrap with --engine.persistence-threshold 0 and --engine.memory-block-buffer-target 0 (verified in /proc/1/cmdline) — gaps still appear, and their positions move between runs rather than recurring at fixed heights. Consistent with the section above: if the blocks never reach the persistence task, forcing it to flush more eagerly changes nothing.

Falling back to P2P sync. Started morph-reth on an empty datadir with --trusted-peers set to the three mainnet peers and no consensus client. It connects (connected_peers=3) but never advances — latest_block=0 after several minutes. So every node, whether restored from a snapshot or synced from genesis, must pass through the fast-ingest path.

Knock-on effects

The reference index can never finish a backfill. It walks the canonical range and stops at the first gap, then retries the same range forever:

INFO  morph::reference_index: reference index phase changed phase=Backfill
WARN  morph::reference_index: reference index reconciliation failed; execution continues
      error="canonical block range 24664765..=24665276 returned 47 of 512 blocks" retry_backoff=30s

24 664 765 + 47 = 24 664 812, exactly the first absent block in that run. On a datadir without gaps the same code reaches phase=Live in about 20 s, so the index logic works — it is starved by the missing blocks.

Re-fetching the bodies fails at the block immediately before the first gap:

morph-reth stage run --from 24664700 --to 24670000 --commit --trusted-peers <...> bodies
Error: stage encountered an error in block #24664811: validation error: mismatched block state root:
  got 0x113ca1c87b3acfdc7ac6631bf3840d5dbd1c1002bef88d44ed4545269a875d8c,
  expected 0x03a772765105775e1b4e4e776c05209e7952e007cd0761e976c8f649309bf313

During catch-up the trie proof workers also log HeaderNotFound heavily (~16 000 occurrences in 3 minutes across ~47 distinct block numbers) alongside State root task timed out, racing serial fallback.

Reproduction

1. Restore an official snapshot-archive-reth-* snapshot, both halves at the same height, into fresh datadirs. Do not start anything on the extracted copy first.
2. Start morph-reth without morph-node; scan windows below the snapshot head — all blocks present.
3. Start morph-node and let derivation carry the node to the tip.
4. Scan windows inside the range derivation just replayed — blocks are missing.

Scanner, keeping null and errors apart:

for ((n=from; n<=to; n++)); do
  h=$(printf "0x%x" "$n")
  r=$(curl -s -X POST -H "content-type: application/json" \
      --data "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"eth_getBlockByNumber\",\"params\":[\"$h\",false]}" \
      "$rpc")
  if echo "$r" | jq -e ".error" >/dev/null 2>&1; then errs=$((errs+1))
  elif [ "$(echo "$r" | jq -r ".result")" = "null" ]; then absent=$((absent+1))
  else ok=$((ok+1)); fi
done

Impact and questions

Every published snapshot necessarily lags the tip, and there is no P2P path to fall back on, so the snapshot → catch-up route always passes through the mode in which blocks are lost. That makes it hard to bring up a complete archive node at all.

• Is the fast-ingest path expected to persist every block, or is some backfill meant to run afterwards?
• Is there a supported way to repair an existing datadir, given stage run bodies fails on the state root check?
• Would a snapshot taken at or very near the current tip avoid this?

Happy to run further diagnostics. In particular I can run an instrumented build that logs accepted versus persisted block numbers around import_l2_block_via_engine and report which blocks are acknowledged but never written, if that would help narrow it down.