From 77c3f6d43eb6659d535e202a86d4029307a63d6b Mon Sep 17 00:00:00 2001 From: "liquan.eth" Date: Sun, 26 Jul 2026 10:05:10 +0800 Subject: [PATCH 01/15] add full-node --- docs/SUMMARY.md | 1 + docs/node/full-node.md | 414 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 415 insertions(+) create mode 100644 docs/node/full-node.md diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 8c421a3..077c2f5 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -47,4 +47,5 @@ - [Stateless Validation](node/stateless-validation.md) - [Validator Architecture](node/validator-architecture.md) - [Get Block Witness](node/witness.md) +- [Run a Full Node](node/full-node.md) - [Debug Trace Server](node/debug-trace-server.md) diff --git a/docs/node/full-node.md b/docs/node/full-node.md new file mode 100644 index 0000000..be611c7 --- /dev/null +++ b/docs/node/full-node.md @@ -0,0 +1,414 @@ +--- +description: Run a MegaETH full node — sync blocks and state from the sequencer stream and re-execute every block with the embedded stateless validator. +--- + +# Run a full node + +A **full node** is a role of `mega-reth`, the MegaETH node client derived from [reth](https://github.com/paradigmxyz/reth). +Like a replica node, it syncs blocks and state from the sequencer stream and serves the full JSON-RPC surface. +In addition, it runs an embedded, asynchronous stateless validator that re-executes every committed block against a [SALT witness](witness.md) and halts on any state-root, receipts-root, logs-bloom, or gas mismatch produced by the sequencer. +Full nodes do not need to trust the sequencer's execution results. + +For how full nodes fit into the network, see [Architecture](../architecture.md). +For the standalone validator binary that verifies blocks without holding any chain state, see [Stateless Validation](stateless-validation.md). + +{% hint style="info" %} +The `mega-reth` repository and its node distributions are currently permissioned — the source and binaries are not generally available. +To run a full node, [contact the MegaETH team](https://megaeth.com) to request access, the network genesis file, and the current peer configuration. +{% endhint %} + +## Node types + +`mega-reth node` runs one of several roles, selected with `--node-type`: + +| Value | Role | +| ----------- | ----------------------------------------------------------------------------------------------------------------------- | +| `full-node` | Syncs blocks and state, serves JSON-RPC, and re-validates every block with the embedded stateless validator. This page. | +| `rpc-node` | Replica node: syncs blocks and state and serves JSON-RPC without re-execution. | +| `replayer` | Re-executes blocks to generate witness data and serves `mega_getBlockWitness`. | +| `sequencer` | The block producer. Default value — a full node must pass `--node-type full-node` explicitly. | + +`full-node` and `rpc-node` share the same sync path and data directory layout; the only difference is the embedded validator pipeline that produces attestations. +Both types serve `mega_getValidatedChain`, but on an `rpc-node` it only reflects validated tips pushed by an upstream full node via `--validator.report-to`. + +## Prerequisites + +A full node needs four inputs beyond the binary itself: + +1. **Genesis file** — the network's chain-spec JSON, passed via `--chain`. + `mega-reth` bundles no Mainnet or Testnet chain spec; the file is distributed with node access and must match the network byte-for-byte. +2. **Sequencer public key** — the secp256k1 key the node uses to verify sequencer signatures on every header, mini-block, and stream fragment it ingests, passed via `--sequencer-public-key`. +3. **Trusted peers** — enode URLs of upstream nodes to sync from, passed via `--trusted-peers`. + MegaETH publishes no bootnodes and block delivery is trusted-peer-only by default, so a node without trusted peers has no upstream and never starts syncing. +4. **Witness endpoint** — one or more RPC URLs serving [`mega_getBlockWitness`](witness.md), passed via `--validator.rpc-urls`. + The endpoint needs no other JSON-RPC method: the validator reads blocks and bytecode from the node's own database and fetches only witnesses remotely. + +Plan for a fast NVMe SSD and several hundred GB of storage — the MegaETH Mainnet data directory measures roughly 400 GB as of July 2026 and grows with chain history. +Validation throughput scales with CPU cores; the validator defaults to one worker per two physical cores. + +## Quick start + +### First run + +Replace the placeholders and launch: + +```bash +mega-reth node \ + --node-type full-node \ + --chain \ + --datadir \ + --sequencer-public-key \ + --trusted-peers , \ + --disable-discovery \ + --max-load 100 \ + --validator.rpc-urls \ + --metrics 127.0.0.1:9001 \ + --log.file.directory +``` + +- `--disable-discovery` is recommended: MegaETH has no public discovery network, and all sync traffic flows through the trusted peers anyway. +- `--max-load` caps how many downstream children this node serves in one streaming tree; it is required and has no default. +- `--metrics` binds the Prometheus endpoint; omit it to disable metrics. + +To also serve JSON-RPC, add the standard reth server flags: + +```bash + --http --http.addr 0.0.0.0 --http.port 8545 \ + --http.api eth,net,web3,debug,trace,txpool,admin \ + --ws --ws.addr 0.0.0.0 --ws.port 8546 +``` + +On start, the node: + +1. Opens the data directory and runs crash-consistency recovery. +2. Initializes P2P networking and logs its own `enode` URL. +3. Handshakes with the trusted peers and starts state sync (`Committed block to engine` lines mark per-block progress). +4. Resolves the validator anchor and starts the validation pipeline. + +Healthy startup output includes these lines: + +```text +INFO Starting mega-reth version=... +INFO MegaETH P2P networking initialized enode=enode://... +INFO Committed block to engine block_number=... source=Fetcher +INFO validator: anchor resolved number=... action=SeedFresh +INFO megaeth validator pipeline started workers=8 in_flight_multiplier=2 rpc_endpoints=1 mode=delta +INFO validator: blocks validated and advanced from=... to=... count=... +``` + +### Subsequent runs + +All flags shown above are operational, not persisted — re-supply them on every run. +The validator resumes from its persisted cursor automatically; only pass `--validator.start-block` when you deliberately want to re-anchor (see [Validator pipeline](#validator-pipeline)). + +## Initial sync + +`--bootstrap-policy` selects how an empty data directory reaches the chain tip: + +- **`never` (default)** — fetch and apply every historical block from the trusted peers, starting at genesis. + The node serves full chain history but initial sync replays the entire chain. +- **`required`** — fetch a snapshot of the current SALT state from peers instead of replaying history, then sync blocks forward from there. + Bootstrap requires an empty data directory, and a full node additionally needs at least 256 recent blocks before the bootstrap can finish. + A bootstrapped node cannot serve history from before its bootstrap block and cannot unwind below it. + +{% hint style="warning" %} +An interrupted bootstrap can only resume if the node is fewer than 1,800 blocks behind the tip. +Beyond that, the node exits with `The last bootstrap is unfinished and it is too far behind to recover` — clear the data directory and restart the bootstrap. +{% endhint %} + +The embedded validator anchors at the first synced head and validates forward only; blocks before the anchor are not re-attested. + +## Command-line reference + +Every flag in the tables below has an equivalent `MEGARETH_*` environment variable, convenient for service managers. +Command-line flags take precedence over environment variables. + +### Core flags + +| Flag | Env variable | Default | Description | +| ------------------------ | ------------------------------- | ---------------- | ------------------------------------------------------------------------------------------- | +| `--node-type` | `MEGARETH_NODE_TYPE` | `sequencer` | Node role. Pass `full-node`. | +| `--chain` | `MEGARETH_CHAIN` | `dev` | Path to the network genesis JSON (`dev` is the only built-in chain). | +| `--datadir` | `MEGARETH_DATA_DIR` | OS data dir | Data directory, used as-is when set. The OS default appends the chain ID as a subdirectory. | +| `--sequencer-public-key` | `MEGARETH_SEQUENCER_PUBLIC_KEY` | — (required) | secp256k1 public key used to verify sequencer signatures on ingested headers and fragments. | +| `--max-load` | `MEGARETH_MAX_LOAD` | — (required) | Maximum number of downstream children served in one streaming tree. | +| `--metrics` | `MEGARETH_METRICS` | unset (disabled) | Prometheus listen address (`HOST:PORT`). | + +### Networking and state sync + +| Flag | Env variable | Default | Description | +| -------------------------- | --------------------------------- | --------- | --------------------------------------------------------------------------------------------------------- | +| `--trusted-peers` | `MEGARETH_TRUSTED_PEERS` | empty | Comma-separated enode URLs of upstream peers. Effectively required — see [Prerequisites](#prerequisites). | +| `--subscribe-trusted-only` | `MEGARETH_SUBSCRIBE_TRUSTED_ONLY` | `true` | Subscribe to block streams from trusted peers only. | +| `--disable-discovery` | `MEGARETH_DISABLE_DISCOVERY` | `false` | Disable peer discovery. Recommended — MegaETH publishes no bootnodes. | +| `--port` | `MEGARETH_PORT` | `30303` | P2P listen port. | +| `--addr` | `MEGARETH_ADDR` | `0.0.0.0` | P2P listen address. | +| `--min-handshake-peers` | `MEGARETH_MIN_HANDSHAKE_PEERS` | `1` | Peers that must complete the state-sync handshake before syncing starts. | +| `--bootstrap-policy` | `MEGARETH_BOOTSTRAP_POLICY` | `never` | `never` (replay from genesis) or `required` (state bootstrap). See [Initial sync](#initial-sync). | + +### Validator flags + +Only consulted when `--node-type full-node`; other node types ignore them. + +| Flag | Env variable | Default | Description | +| ----------------------------------- | ------------------------------------------ | ------------------ | -------------------------------------------------------------------------------------------------------------- | +| `--validator.rpc-urls` | `MEGARETH_VALIDATOR_RPC_URLS` | — (required) | Witness RPC URLs (comma-separated, round-robin failover). Each must serve `mega_getBlockWitness`. | +| `--validator.mode` | `MEGARETH_VALIDATOR_MODE` | `delta` | Validation strategy: `delta` or `full`. See [Validation modes](#validation-modes). | +| `--validator.workers` | `MEGARETH_VALIDATOR_WORKERS` | physical cores / 2 | Parallel validation workers (minimum 1). Worker count mainly matters during catch-up. | +| `--validator.start-block` | `MEGARETH_VALIDATOR_START_BLOCK` | unset | Trusted block hash to anchor at; validation begins at the next block. Overrides the persisted anchor. | +| `--validator.start-block-wait-secs` | `MEGARETH_VALIDATOR_START_BLOCK_WAIT_SECS` | `0` (indefinite) | How long to wait for the start block to appear locally before the pipeline halts. | +| `--validator.witness-wait-secs` | `MEGARETH_VALIDATOR_WITNESS_WAIT_SECS` | `60` | Per-block witness-fetch deadline; on expiry the block is re-enqueued. | +| `--validator.tip-buffer` | `MEGARETH_VALIDATOR_TIP_BUFFER` | `1` | Blocks of headroom below the local tip before fetching a witness, giving the witness generator time to finish. | +| `--validator.channel-capacity` | `MEGARETH_VALIDATOR_CHANNEL_CAPACITY` | workers × 2 | Target total in-flight witness fetches; values below the worker count are clamped up. | +| `--validator.poll-interval-ms` | `MEGARETH_VALIDATOR_POLL_INTERVAL_MS` | `100` | Tip-refresh interval when the validator is caught up. | +| `--validator.report-to` | `MEGARETH_VALIDATOR_REPORT_TO` | empty (disabled) | RPC-node URLs that receive best-effort `mega_setValidatedBlocks` pushes of the validated tip. | + +{% hint style="info" %} +When pointing `--validator.rpc-urls` at a rate-limited endpoint, lower `--validator.channel-capacity` to cap concurrent witness fetches — the default of two in-flight fetches per worker is tuned for a dedicated witness provider. +{% endhint %} + +### Logging flags + +| Flag | Env variable | Default | Description | +| ---------------------- | ----------------------------- | ------------------------------ | ------------------------------------------------------------- | +| `--log.stdout.filter` | `MEGARETH_LOG_STDOUT_FILTER` | unset (`info` via verbosity) | Console log filter (`tracing` directive syntax, e.g. `info`). | +| `--log.stdout.format` | `MEGARETH_LOG_STDOUT_FORMAT` | `terminal` | Console format: `terminal`, `log-fmt`, or `json`. | +| `--log.file.directory` | `MEGARETH_LOG_FILE_DIRECTORY` | OS cache dir (`.../reth/logs`) | Directory for rotated log files; the chain ID is appended. | +| `--log.file.filter` | `MEGARETH_LOG_FILE_FILTER` | `debug` | Log filter for file output. | +| `--log.file.max-size` | `MEGARETH_LOG_FILE_MAX_SIZE` | `200` | Max log file size (MB) before rotation. | +| `--log.file.max-files` | `MEGARETH_LOG_FILE_MAX_FILES` | `5` | Number of rotated log files to keep. | +| `--color` | `MEGARETH_LOG_COLOR` | `always` | ANSI color: `always`, `auto`, or `never`. | + +## Validator pipeline + +The embedded validator runs off the commit critical path: state sync commits blocks at full speed, and validation follows asynchronously. +A validator failure never tears down the node — RPC and sync keep running. + +### Anchor + +The anchor is the trusted block validation starts from; blocks are validated from `anchor + 1` onward. +At startup the validator resolves it in this order: + +1. `--validator.start-block ` set — anchor at that block, overriding any persisted anchor and clearing the validated cursor. + The node waits (bounded by `--validator.start-block-wait-secs`) for the hash to appear locally via state sync. +2. Flag unset, persisted anchor exists — resume from the persisted cursor. +3. Flag unset, fresh database — anchor at the last finalized block, or at the first synced canonical head if nothing is finalized yet. + +The resolved anchor is persisted, so the choice survives restarts. +The `validator: anchor resolved` log line reports the outcome: `action=Skip` (anchor unchanged), `SeedFresh` (first anchor), or `OverrideWithRollback` (re-anchored via flag). + +{% hint style="warning" %} +`--validator.start-block` takes a block **hash**, not a number, and re-anchoring clears the validated cursor — the range validated under the old anchor is no longer attested. +Once the anchor is persisted, restarts with the same flag are skipped without rewriting it, but remove the flag anyway so a copied unit file or later restart does not silently pin validation to an old block. +{% endhint %} + +### Validation modes + +Both modes IPA-verify the witness and re-execute the block with the `stateless-validator` library; they differ in how the replay result is bound to the block header: + +- **`delta` (default)** — compare the replay-derived SALT changeset against the hash-verified changeset that state sync persisted for the same block, skipping the per-block SALT trie recompute. + Blocks whose stored changeset is unavailable automatically fall back to `full` behavior (counted by `reth_megaeth_validator_changeset_fallback_full_total`). +- **`full`** — recompute the SALT trie root from the replay output and compare it with the header's state root. + Slower, but independent of stored changesets. + +### Mismatch handling + +A deterministic validation failure — a state-root, receipts-root, logs-bloom, or gas mismatch, or a changeset mismatch — halts the validation pipeline. +The node logs `megaeth validator pipeline halted` at error level, sets the `reth_megaeth_validator_halted` gauge to 1, and keeps serving RPC and syncing. +The stalled validated height is the operator alert: it means the sequencer produced a block this node could not reproduce. + +On a reorg, the validator restarts from its persisted cursor — it does not search for the divergence point — and re-validates forward on the new canonical chain; `reth_megaeth_validator_reorg_resets_total` counts these events. + +## Monitoring + +### Checking progress + +With `--metrics` set, compare the sync tip against the validated cursor: + +```bash +curl -s http://localhost:9001/metrics | grep -v '^#' | grep -E 'state_sync_backend_provider_local_block_height|megaeth_validator_cursor' +``` + +```text +reth_state_sync_backend_provider_local_block_height 6907400 +reth_megaeth_validator_cursor 6907396 +``` + +The difference is the validation lag in blocks. +A small, stable lag is normal; a growing lag means validation cannot keep pace (see [Troubleshooting](#troubleshooting)). + +Full nodes also serve `mega_getValidatedChain`, which returns the anchor and the validated tip. +`mega_*` methods are merged into every enabled transport, so the namespace does not need to be listed in `--http.api`: + +```bash +curl -sX POST http://localhost:8545 \ + -H 'Content-Type: application/json' \ + -d '{"jsonrpc":"2.0","method":"mega_getValidatedChain","params":[],"id":1}' +``` + +### Useful metrics + +| Metric | Type | What it tells you | +| ---------------------------------------------------------------- | --------- | ----------------------------------------------------------------------------------------- | +| `reth_state_sync_backend_provider_local_block_height` | Gauge | Latest committed block — the sync tip. | +| `reth_megaeth_validator_cursor` | Gauge | Highest contiguously validated block. | +| `reth_megaeth_validator_halted` | Gauge | `1` when the pipeline halted — on a mismatch or any fatal validator error. Alert on this. | +| `reth_megaeth_validator_validated_blocks_total` | Counter | Blocks re-executed and validated. | +| `reth_megaeth_validator_failures_validate_total` | Counter | Deterministic validation failures. | +| `reth_megaeth_validator_changeset_fallback_full_total` | Counter | Delta-mode blocks that fell back to the full path. Steady growth is worth investigating. | +| `reth_megaeth_validator_block_validation_duration_seconds` | Histogram | End-to-end validation time per block. | +| `reth_megaeth_validator_validation_witness_verification_seconds` | Histogram | Witness IPA-proof verification time per block. | +| `reth_megaeth_validator_validation_block_replay_seconds` | Histogram | EVM replay time per block. | +| `reth_megaeth_validator_reorg_resets_total` | Counter | Validator cursor rollbacks caused by reorgs. | +| `reth_db_table_size{table=...}` | Gauge | On-disk size per database table — watch for disk planning. | + +The duration metrics are exposed as Prometheus summaries (quantile series plus `_sum`/`_count`), not bucketed histograms. + +### Key log lines + +| Line | Level | Meaning | +| ------------------------------------------ | ----- | ----------------------------------------------------------------------------------- | +| `Committed block to engine` | info | A synced block was committed; the per-block liveness line. | +| `validator: anchor resolved` | info | Anchor decided at startup (`action` field says how). | +| `megaeth validator pipeline started` | info | Validator running; fields include `workers` and `mode`. | +| `validator: blocks validated and advanced` | info | Validated cursor advanced (`from`, `to`, `count`). | +| `megaeth validator pipeline halted` | error | Validation stopped on a fatal error — most commonly a mismatch; node keeps serving. | + +## Deployment + +Run the node under a service manager in production. +A minimal systemd setup: + +```ini +# /etc/megaeth/full-node.env +MEGARETH_NODE_TYPE=full-node +MEGARETH_CHAIN=/etc/megaeth/genesis.json +MEGARETH_DATA_DIR=/var/lib/megaeth/full-node +MEGARETH_SEQUENCER_PUBLIC_KEY= +MEGARETH_TRUSTED_PEERS=, +MEGARETH_DISABLE_DISCOVERY=true +MEGARETH_MAX_LOAD=100 +MEGARETH_VALIDATOR_RPC_URLS= +MEGARETH_METRICS=127.0.0.1:9001 +MEGARETH_LOG_FILE_DIRECTORY=/var/log/megaeth +``` + +```ini +# /etc/systemd/system/megaeth-full-node.service +[Unit] +Description=MegaETH full node +After=network-online.target +Wants=network-online.target + +[Service] +User=megaeth +EnvironmentFile=/etc/megaeth/full-node.env +ExecStart=/usr/local/bin/mega-reth node +Restart=on-failure +RestartSec=5 +LimitNOFILE=1048576 +TimeoutStopSec=120 + +[Install] +WantedBy=multi-user.target +``` + +`TimeoutStopSec=120` matters: on shutdown the node flushes its databases, and a clean stop can take tens of seconds (5 s for ordinary tasks and 30 s total for critical tasks by default; `MEGARETH_CRITICAL_WAIT_SECS=N` replaces the critical budget with 5 s + N s). + +## Data directory and maintenance + +An explicitly passed `--datadir` is used as-is (the OS default adds a `/` level) and contains: + +| Path | Contents | +| ------------------ | ---------------------------------------------------------------------------------------------- | +| `db/main_db/` | RocksDB instance holding headers, transactions, receipts, SALT state and trie, and changesets. | +| `db/index_db/` | RocksDB instance holding history indexes and transaction lookups (full and rpc nodes only). | +| `known-peers.json` | Persisted peer set, written on shutdown. | +| `jwt.hex` | Auto-generated Engine API JWT secret. | +| `discovery-secret` | P2P identity key. | + +The data directory layout is tied to the node type — reuse a data directory only with a node type that shares its layout (`full-node` and `rpc-node` do; a `sequencer` data directory does not). + +### Graceful shutdown and backups + +Stop the node with SIGTERM or SIGINT and wait for the process to exit before touching the data directory. + +{% hint style="warning" %} +Never SIGKILL the node or copy the data directory while it runs. +`mega-reth` writes RocksDB with the write-ahead log disabled: memtable contents are lost on a hard kill, and `main_db` and `index_db` flush independently, so a live copy captures the two databases at different heights. +The node repairs an inconsistent directory in place on the next start, but a backup taken from a running or killed node bakes the inconsistency in. +{% endhint %} + +To back up: stop the node, copy the data directory, and verify the copy: + +```bash +mega-reth tables-height \ + --datadir \ + --chain \ + --node-type full-node +# Expect: "tables height is consistent true at block height: " +``` + +### Unwinding to an earlier block + +`mega-reth recovery` rewinds the database to a target block after verifying consistency: + +```bash +mega-reth recovery \ + --datadir \ + --chain \ + --node-type full-node \ + --target_block_number +``` + +{% hint style="warning" %} +`recovery` rewrites the database destructively — back up the data directory first. +Maintenance subcommands (`tables-height`, `recovery`, `db`) do not raise the process file-descriptor limit the way `node` does; raise it yourself (`ulimit -n 1048576`) before running them against a large database. +{% endhint %} + +## Trust model + +A full node removes the sequencer's execution from its trusted computing base in two layers: + +- **Stream integrity** — every header, mini-block, and stream fragment ingested via state sync must carry a valid signature from `--sequencer-public-key`, so a compromised peer cannot inject fabricated data. +- **Execution correctness** — the embedded validator re-executes every block and halts loudly on any mismatch, so the node never silently serves state the sequencer computed incorrectly. + +The witness endpoint does not need to be trusted for correctness: witness contents are cryptographically verified, so a faulty endpoint can only stall validation, not produce a false attestation. + +Like the standalone stateless validator, a full node validates the block sequence its peers feed it — it does not derive the canonical chain from L1, and it does not check consistency with the rollup batches posted to L1. +On a full node, the `safe` and `finalized` block tags advance only when an external consensus client drives `engine_forkchoiceUpdated`; sequencer-provided finality markers are ignored. + +## Troubleshooting + +**The node starts but never syncs.** +State sync waits for at least `--min-handshake-peers` trusted peers to complete the handshake. +Check that `--trusted-peers` is set, the enode URLs are current, and the peers are reachable on their P2P ports. + +**Startup fails with `--node-type full requires at least one --validator.rpc-urls`.** +Full nodes cannot start without a witness endpoint. +Pass `--validator.rpc-urls` (see [Prerequisites](#prerequisites)). + +**`megaeth validator pipeline halted` and `reth_megaeth_validator_halted` is 1.** +The validator hit a deterministic mismatch: this node could not reproduce a block the sequencer committed. +The node keeps syncing and serving RPC, but blocks past the stalled cursor are unattested. +Preserve the logs around the halt and report the block number to the MegaETH team. + +**The lag between the sync tip and the validator cursor keeps growing.** +Either witness fetches are stalling (witness endpoint slow, rate-limited, or persistently behind the local tip — consider raising `--validator.tip-buffer`) or validation is CPU-bound (raise `--validator.workers` up to the physical core count). +The `block_validation_duration_seconds`, `validation_witness_verification_seconds`, and `validation_block_replay_seconds` histograms show where the time goes. + +**`reth_megaeth_validator_changeset_fallback_full_total` climbs steadily.** +Delta mode cannot find stored changesets for current blocks, so every block takes the slower full path. +Validation results remain correct; expect reduced throughput. +On a database whose recent blocks were synced by a current build, steady growth is unexpected and worth reporting. + +**Startup fails with `Bootstrap is required, but the database is not empty`.** +`--bootstrap-policy required` only works on an empty data directory. +Either clear the data directory to bootstrap, or drop the flag to keep the existing database. + +## Related pages + +- [Stateless Validation](stateless-validation.md) — the standalone validator binary and its trust model +- [Validator Architecture](validator-architecture.md) — how witness-based block validation works +- [Get Block Witness](witness.md) — `mega_getBlockWitness` reference and witness data layout +- [Architecture](../architecture.md) — sequencer, replica nodes, and full nodes in the network From 12b92439df824d7492cfccdc87656282e39295e8 Mon Sep 17 00:00:00 2001 From: "liquan.eth" Date: Sun, 26 Jul 2026 13:41:50 +0800 Subject: [PATCH 02/15] doc: address full node review feedback - Bind the quickstart JSON-RPC example to loopback with eth,net,web3 and warn about exposing admin/debug/trace/txpool on a public interface - Label the validator duration metrics as Prometheus summaries and explain that histogram_quantile() does not apply to them - Add replay-full and verifier to the node types table - Note that index_db is created for full-node, rpc-node, and replay-full - Use an angle-bracket placeholder for the reth_db_table_size label --- docs/node/full-node.md | 58 +++++++++++++++++++++++++----------------- 1 file changed, 34 insertions(+), 24 deletions(-) diff --git a/docs/node/full-node.md b/docs/node/full-node.md index be611c7..d862210 100644 --- a/docs/node/full-node.md +++ b/docs/node/full-node.md @@ -21,12 +21,14 @@ To run a full node, [contact the MegaETH team](https://megaeth.com) to request a `mega-reth node` runs one of several roles, selected with `--node-type`: -| Value | Role | -| ----------- | ----------------------------------------------------------------------------------------------------------------------- | -| `full-node` | Syncs blocks and state, serves JSON-RPC, and re-validates every block with the embedded stateless validator. This page. | -| `rpc-node` | Replica node: syncs blocks and state and serves JSON-RPC without re-execution. | -| `replayer` | Re-executes blocks to generate witness data and serves `mega_getBlockWitness`. | -| `sequencer` | The block producer. Default value — a full node must pass `--node-type full-node` explicitly. | +| Value | Role | +| ------------- | ----------------------------------------------------------------------------------------------------------------------- | +| `full-node` | Syncs blocks and state, serves JSON-RPC, and re-validates every block with the embedded stateless validator. This page. | +| `rpc-node` | Replica node: syncs blocks and state and serves JSON-RPC without re-execution. | +| `replayer` | Re-executes blocks to generate witness data and serves `mega_getBlockWitness`. | +| `replay-full` | Isolated successor to the legacy `replayer` role; keeps a block index and also serves `mega_getBlockWitness`. | +| `verifier` | Syncs block data and re-executes transactions to check state consistency. | +| `sequencer` | The block producer. Default value — a full node must pass `--node-type full-node` explicitly. | `full-node` and `rpc-node` share the same sync path and data directory layout; the only difference is the embedded validator pipeline that produces attestations. Both types serve `mega_getValidatedChain`, but on an `rpc-node` it only reflects validated tips pushed by an upstream full node via `--validator.report-to`. @@ -73,11 +75,19 @@ mega-reth node \ To also serve JSON-RPC, add the standard reth server flags: ```bash - --http --http.addr 0.0.0.0 --http.port 8545 \ - --http.api eth,net,web3,debug,trace,txpool,admin \ - --ws --ws.addr 0.0.0.0 --ws.port 8546 + --http --http.addr 127.0.0.1 --http.port 8545 \ + --http.api eth,net,web3 \ + --ws --ws.addr 127.0.0.1 --ws.port 8546 ``` +The `mega_*` methods are not part of this selection — they are merged into every enabled transport regardless of `--http.api`, so `mega_getValidatedChain` works with the namespace set above. + +{% hint style="warning" %} +Bind to `127.0.0.1` unless the node is deliberately a public RPC endpoint. +Serving `0.0.0.0` with `admin`, `debug`, `trace`, or `txpool` enabled exposes `admin_addPeer` (unauthenticated peer manipulation), `debug_trace*` (unmetered remote CPU and memory), and `txpool_content` (pending-transaction leakage) to anyone who can reach the port. +To serve traffic publicly, keep the node on loopback and put a reverse proxy in front of it that terminates TLS, rate-limits, and forwards only the namespaces you intend to expose. +{% endhint %} + On start, the node: 1. Opens the data directory and runs crash-consistency recovery. @@ -237,8 +247,7 @@ reth_megaeth_validator_cursor 6907396 The difference is the validation lag in blocks. A small, stable lag is normal; a growing lag means validation cannot keep pace (see [Troubleshooting](#troubleshooting)). -Full nodes also serve `mega_getValidatedChain`, which returns the anchor and the validated tip. -`mega_*` methods are merged into every enabled transport, so the namespace does not need to be listed in `--http.api`: +Full nodes also serve `mega_getValidatedChain`, which returns the anchor and the validated tip — available on any enabled transport, whatever `--http.api` selects: ```bash curl -sX POST http://localhost:8545 \ @@ -256,13 +265,14 @@ curl -sX POST http://localhost:8545 \ | `reth_megaeth_validator_validated_blocks_total` | Counter | Blocks re-executed and validated. | | `reth_megaeth_validator_failures_validate_total` | Counter | Deterministic validation failures. | | `reth_megaeth_validator_changeset_fallback_full_total` | Counter | Delta-mode blocks that fell back to the full path. Steady growth is worth investigating. | -| `reth_megaeth_validator_block_validation_duration_seconds` | Histogram | End-to-end validation time per block. | -| `reth_megaeth_validator_validation_witness_verification_seconds` | Histogram | Witness IPA-proof verification time per block. | -| `reth_megaeth_validator_validation_block_replay_seconds` | Histogram | EVM replay time per block. | +| `reth_megaeth_validator_block_validation_duration_seconds` | Summary | End-to-end validation time per block. | +| `reth_megaeth_validator_validation_witness_verification_seconds` | Summary | Witness IPA-proof verification time per block. | +| `reth_megaeth_validator_validation_block_replay_seconds` | Summary | EVM replay time per block. | | `reth_megaeth_validator_reorg_resets_total` | Counter | Validator cursor rollbacks caused by reorgs. | -| `reth_db_table_size{table=...}` | Gauge | On-disk size per database table — watch for disk planning. | +| `reth_db_table_size{table=}` | Gauge | On-disk size per database table — watch for disk planning. | -The duration metrics are exposed as Prometheus summaries (quantile series plus `_sum`/`_count`), not bucketed histograms. +The three duration metrics are recorded as histograms internally but exported as Prometheus **summaries** — pre-computed quantile series plus `_sum` and `_count`, with no `_bucket` series. +Read the quantiles directly (`reth_megaeth_validator_block_validation_duration_seconds{quantile="0.99"}`); `histogram_quantile()` returns nothing for them. ### Key log lines @@ -319,13 +329,13 @@ WantedBy=multi-user.target An explicitly passed `--datadir` is used as-is (the OS default adds a `/` level) and contains: -| Path | Contents | -| ------------------ | ---------------------------------------------------------------------------------------------- | -| `db/main_db/` | RocksDB instance holding headers, transactions, receipts, SALT state and trie, and changesets. | -| `db/index_db/` | RocksDB instance holding history indexes and transaction lookups (full and rpc nodes only). | -| `known-peers.json` | Persisted peer set, written on shutdown. | -| `jwt.hex` | Auto-generated Engine API JWT secret. | -| `discovery-secret` | P2P identity key. | +| Path | Contents | +| ------------------ | ------------------------------------------------------------------------------------------------------------------- | +| `db/main_db/` | RocksDB instance holding headers, transactions, receipts, SALT state and trie, and changesets. | +| `db/index_db/` | RocksDB instance holding history indexes and transaction lookups (`full-node`, `rpc-node`, and `replay-full` only). | +| `known-peers.json` | Persisted peer set, written on shutdown. | +| `jwt.hex` | Auto-generated Engine API JWT secret. | +| `discovery-secret` | P2P identity key. | The data directory layout is tied to the node type — reuse a data directory only with a node type that shares its layout (`full-node` and `rpc-node` do; a `sequencer` data directory does not). @@ -395,7 +405,7 @@ Preserve the logs around the halt and report the block number to the MegaETH tea **The lag between the sync tip and the validator cursor keeps growing.** Either witness fetches are stalling (witness endpoint slow, rate-limited, or persistently behind the local tip — consider raising `--validator.tip-buffer`) or validation is CPU-bound (raise `--validator.workers` up to the physical core count). -The `block_validation_duration_seconds`, `validation_witness_verification_seconds`, and `validation_block_replay_seconds` histograms show where the time goes. +The `block_validation_duration_seconds`, `validation_witness_verification_seconds`, and `validation_block_replay_seconds` quantiles show where the time goes. **`reth_megaeth_validator_changeset_fallback_full_total` climbs steadily.** Delta mode cannot find stored changesets for current blocks, so every block takes the slower full path. From 5ed7536df49aee896775948231031e6a30204230 Mon Sep 17 00:00:00 2001 From: "liquan.eth" Date: Sun, 26 Jul 2026 14:51:44 +0800 Subject: [PATCH 03/15] doc: limit node types table to externally available roles & fix lint The replayer, replay-full and verifier roles are internal and not open to external operators, so drop them from the table and say plainly that it lists the roles operators can actually run. Also stop claiming index_db is exclusive to full-node and rpc-node, and apply prettier formatting. --- docs/node/full-node.md | 62 ++++++++++++++++++++---------------------- lychee.toml | 2 +- 2 files changed, 31 insertions(+), 33 deletions(-) diff --git a/docs/node/full-node.md b/docs/node/full-node.md index d862210..ac303dd 100644 --- a/docs/node/full-node.md +++ b/docs/node/full-node.md @@ -4,12 +4,12 @@ description: Run a MegaETH full node — sync blocks and state from the sequence # Run a full node -A **full node** is a role of `mega-reth`, the MegaETH node client derived from [reth](https://github.com/paradigmxyz/reth). +A **full node** is a role of `mega-reth`, the MegaETH node client. Like a replica node, it syncs blocks and state from the sequencer stream and serves the full JSON-RPC surface. In addition, it runs an embedded, asynchronous stateless validator that re-executes every committed block against a [SALT witness](witness.md) and halts on any state-root, receipts-root, logs-bloom, or gas mismatch produced by the sequencer. Full nodes do not need to trust the sequencer's execution results. -For how full nodes fit into the network, see [Architecture](../architecture.md). +For how full nodes and replica nodes fit into the network, see [RPC Nodes](../architecture.md#rpc-nodes) in the architecture overview. For the standalone validator binary that verifies blocks without holding any chain state, see [Stateless Validation](stateless-validation.md). {% hint style="info" %} @@ -19,16 +19,14 @@ To run a full node, [contact the MegaETH team](https://megaeth.com) to request a ## Node types -`mega-reth node` runs one of several roles, selected with `--node-type`: +`mega-reth node` selects its role with `--node-type`. +The roles available to external operators are: -| Value | Role | -| ------------- | ----------------------------------------------------------------------------------------------------------------------- | -| `full-node` | Syncs blocks and state, serves JSON-RPC, and re-validates every block with the embedded stateless validator. This page. | -| `rpc-node` | Replica node: syncs blocks and state and serves JSON-RPC without re-execution. | -| `replayer` | Re-executes blocks to generate witness data and serves `mega_getBlockWitness`. | -| `replay-full` | Isolated successor to the legacy `replayer` role; keeps a block index and also serves `mega_getBlockWitness`. | -| `verifier` | Syncs block data and re-executes transactions to check state consistency. | -| `sequencer` | The block producer. Default value — a full node must pass `--node-type full-node` explicitly. | +| Value | Role | +| ----------- | ----------------------------------------------------------------------------------------------------------------------- | +| `full-node` | Syncs blocks and state, serves JSON-RPC, and re-validates every block with the embedded stateless validator. This page. | +| `rpc-node` | Replica node: syncs blocks and state and serves JSON-RPC without re-execution. | +| `sequencer` | The block producer. Default value — a full node must pass `--node-type full-node` explicitly. | `full-node` and `rpc-node` share the same sync path and data directory layout; the only difference is the embedded validator pipeline that produces attestations. Both types serve `mega_getValidatedChain`, but on an `rpc-node` it only reflects validated tips pushed by an upstream full node via `--validator.report-to`. @@ -72,7 +70,7 @@ mega-reth node \ - `--max-load` caps how many downstream children this node serves in one streaming tree; it is required and has no default. - `--metrics` binds the Prometheus endpoint; omit it to disable metrics. -To also serve JSON-RPC, add the standard reth server flags: +To also serve JSON-RPC, add the standard server flags: ```bash --http --http.addr 127.0.0.1 --http.port 8545 \ @@ -257,19 +255,19 @@ curl -sX POST http://localhost:8545 \ ### Useful metrics -| Metric | Type | What it tells you | -| ---------------------------------------------------------------- | --------- | ----------------------------------------------------------------------------------------- | -| `reth_state_sync_backend_provider_local_block_height` | Gauge | Latest committed block — the sync tip. | -| `reth_megaeth_validator_cursor` | Gauge | Highest contiguously validated block. | -| `reth_megaeth_validator_halted` | Gauge | `1` when the pipeline halted — on a mismatch or any fatal validator error. Alert on this. | -| `reth_megaeth_validator_validated_blocks_total` | Counter | Blocks re-executed and validated. | -| `reth_megaeth_validator_failures_validate_total` | Counter | Deterministic validation failures. | -| `reth_megaeth_validator_changeset_fallback_full_total` | Counter | Delta-mode blocks that fell back to the full path. Steady growth is worth investigating. | -| `reth_megaeth_validator_block_validation_duration_seconds` | Summary | End-to-end validation time per block. | -| `reth_megaeth_validator_validation_witness_verification_seconds` | Summary | Witness IPA-proof verification time per block. | -| `reth_megaeth_validator_validation_block_replay_seconds` | Summary | EVM replay time per block. | -| `reth_megaeth_validator_reorg_resets_total` | Counter | Validator cursor rollbacks caused by reorgs. | -| `reth_db_table_size{table=
}` | Gauge | On-disk size per database table — watch for disk planning. | +| Metric | Type | What it tells you | +| ---------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------- | +| `reth_state_sync_backend_provider_local_block_height` | Gauge | Latest committed block — the sync tip. | +| `reth_megaeth_validator_cursor` | Gauge | Highest contiguously validated block. | +| `reth_megaeth_validator_halted` | Gauge | `1` when the pipeline halted — on a mismatch or any fatal validator error. Alert on this. | +| `reth_megaeth_validator_validated_blocks_total` | Counter | Blocks re-executed and validated. | +| `reth_megaeth_validator_failures_validate_total` | Counter | Deterministic validation failures. | +| `reth_megaeth_validator_changeset_fallback_full_total` | Counter | Delta-mode blocks that fell back to the full path. Steady growth is worth investigating. | +| `reth_megaeth_validator_block_validation_duration_seconds` | Summary | End-to-end validation time per block. | +| `reth_megaeth_validator_validation_witness_verification_seconds` | Summary | Witness IPA-proof verification time per block. | +| `reth_megaeth_validator_validation_block_replay_seconds` | Summary | EVM replay time per block. | +| `reth_megaeth_validator_reorg_resets_total` | Counter | Validator cursor rollbacks caused by reorgs. | +| `reth_db_table_size{table=
}` | Gauge | On-disk size per database table — watch for disk planning. | The three duration metrics are recorded as histograms internally but exported as Prometheus **summaries** — pre-computed quantile series plus `_sum` and `_count`, with no `_bucket` series. Read the quantiles directly (`reth_megaeth_validator_block_validation_duration_seconds{quantile="0.99"}`); `histogram_quantile()` returns nothing for them. @@ -329,13 +327,13 @@ WantedBy=multi-user.target An explicitly passed `--datadir` is used as-is (the OS default adds a `/` level) and contains: -| Path | Contents | -| ------------------ | ------------------------------------------------------------------------------------------------------------------- | -| `db/main_db/` | RocksDB instance holding headers, transactions, receipts, SALT state and trie, and changesets. | -| `db/index_db/` | RocksDB instance holding history indexes and transaction lookups (`full-node`, `rpc-node`, and `replay-full` only). | -| `known-peers.json` | Persisted peer set, written on shutdown. | -| `jwt.hex` | Auto-generated Engine API JWT secret. | -| `discovery-secret` | P2P identity key. | +| Path | Contents | +| ------------------ | ---------------------------------------------------------------------------------------------------------- | +| `db/main_db/` | RocksDB instance holding headers, transactions, receipts, SALT state and trie, and changesets. | +| `db/index_db/` | RocksDB instance holding history indexes and transaction lookups (created for `full-node` and `rpc-node`). | +| `known-peers.json` | Persisted peer set, written on shutdown. | +| `jwt.hex` | Auto-generated Engine API JWT secret. | +| `discovery-secret` | P2P identity key. | The data directory layout is tied to the node type — reuse a data directory only with a node type that shares its layout (`full-node` and `rpc-node` do; a `sequencer` data directory does not). diff --git a/lychee.toml b/lychee.toml index 3cfc41e..f1fc459 100644 --- a/lychee.toml +++ b/lychee.toml @@ -9,7 +9,7 @@ offline = true root_dir = "docs" # Check anchor fragments (#section-name) -include_fragments = "full" +include_fragments = true # Skip these URL patterns (regex) exclude = [ From c420c8b244dc8c1f530269bc870cbcf26e18658b Mon Sep 17 00:00:00 2001 From: "liquan.eth" Date: Sun, 26 Jul 2026 15:11:04 +0800 Subject: [PATCH 04/15] doc: sync full node guide with the current validator branch Re-verified the page against mega-reth ba30a29a6 (liquan/perf/fullnode-state-sync-validator). Flags, defaults, log lines and anchor resolution are unchanged; the metrics surface and delta-mode fallbacks grew. - Document the metrics added on the branch: validation_salt_update_seconds, anchor_fallback_full_total, failures_panic_total, canonical_lag_seconds, and the report-to pair publisher_active / publish_failures_total{peer} - Delta mode has two fallback causes now (missing changeset row, missing parent anchor), each with its own counter; note it never runs unanchored - Split the fallback troubleshooting entry to say which counter means what - Note that mega_getValidatedChain reports anchor=null on an rpc-node and tip=null until the first attestation lands - Fix the initial-sync anchor sentence, which claimed the validator anchors at the first synced head rather than the last finalized block --- docs/node/full-node.md | 56 +++++++++++++++++++++++++++++++++--------- 1 file changed, 45 insertions(+), 11 deletions(-) diff --git a/docs/node/full-node.md b/docs/node/full-node.md index ac303dd..204da74 100644 --- a/docs/node/full-node.md +++ b/docs/node/full-node.md @@ -29,7 +29,8 @@ The roles available to external operators are: | `sequencer` | The block producer. Default value — a full node must pass `--node-type full-node` explicitly. | `full-node` and `rpc-node` share the same sync path and data directory layout; the only difference is the embedded validator pipeline that produces attestations. -Both types serve `mega_getValidatedChain`, but on an `rpc-node` it only reflects validated tips pushed by an upstream full node via `--validator.report-to`. +Both types serve `mega_getValidatedChain`, but on an `rpc-node` it only reflects validated tips pushed by an upstream full node via `--validator.report-to` — that node validated nothing itself, and its `anchor` always reads `null`. +To see where attestation actually starts, query the full node that produced the tips. ## Prerequisites @@ -124,7 +125,7 @@ An interrupted bootstrap can only resume if the node is fewer than 1,800 blocks Beyond that, the node exits with `The last bootstrap is unfinished and it is too far behind to recover` — clear the data directory and restart the bootstrap. {% endhint %} -The embedded validator anchors at the first synced head and validates forward only; blocks before the anchor are not re-attested. +Either way the embedded validator validates forward only, from the anchor it resolves at startup (see [Anchor](#anchor)); blocks before that point are not re-attested. ## Command-line reference @@ -214,10 +215,19 @@ Once the anchor is persisted, restarts with the same flag are skipped without re Both modes IPA-verify the witness and re-execute the block with the `stateless-validator` library; they differ in how the replay result is bound to the block header: -- **`delta` (default)** — compare the replay-derived SALT changeset against the hash-verified changeset that state sync persisted for the same block, skipping the per-block SALT trie recompute. - Blocks whose stored changeset is unavailable automatically fall back to `full` behavior (counted by `reth_megaeth_validator_changeset_fallback_full_total`). +- **`delta` (default)** — verify the witness against a pre-state anchored to the parent header, then compare the replay-derived SALT changeset against the hash-verified changeset that state sync persisted for the same block, skipping the per-block SALT trie recompute. - **`full`** — recompute the SALT trie root from the replay output and compare it with the header's state root. - Slower, but independent of stored changesets. + Slower, but independent of stored changesets and parent anchors. + +Delta mode never runs unanchored: a block falls back to `full` behavior for that block alone when either input is missing, and each cause has its own counter. + +| Missing input | Counter | +| ---------------------------------------------------------------- | ------------------------------------------------------ | +| Stored changeset row | `reth_megaeth_validator_changeset_fallback_full_total` | +| Parent-anchor pair (reorg mid-fetch, or a pruned/bootstrap edge) | `reth_megaeth_validator_anchor_fallback_full_total` | + +Both are counted once per validated block, not per fetch attempt. +Validation results are identical either way — only throughput differs. ### Mismatch handling @@ -253,6 +263,9 @@ curl -sX POST http://localhost:8545 \ -d '{"jsonrpc":"2.0","method":"mega_getValidatedChain","params":[],"id":1}' ``` +`(anchor, tip]` is the range this node has re-executed and validated itself. +`tip` reads `null` until the first attestation lands after a fresh start or a re-anchor, so treat a null tip on a freshly started node as "not yet", not as a failure. + ### Useful metrics | Metric | Type | What it tells you | @@ -262,14 +275,28 @@ curl -sX POST http://localhost:8545 \ | `reth_megaeth_validator_halted` | Gauge | `1` when the pipeline halted — on a mismatch or any fatal validator error. Alert on this. | | `reth_megaeth_validator_validated_blocks_total` | Counter | Blocks re-executed and validated. | | `reth_megaeth_validator_failures_validate_total` | Counter | Deterministic validation failures. | -| `reth_megaeth_validator_changeset_fallback_full_total` | Counter | Delta-mode blocks that fell back to the full path. Steady growth is worth investigating. | +| `reth_megaeth_validator_failures_panic_total` | Counter | Panics inside the validation closure. Any non-zero value is a bug — report it. | +| `reth_megaeth_validator_changeset_fallback_full_total` | Counter | Delta-mode blocks that fell back for a missing changeset row. | +| `reth_megaeth_validator_anchor_fallback_full_total` | Counter | Delta-mode blocks that fell back for a missing parent anchor. | | `reth_megaeth_validator_block_validation_duration_seconds` | Summary | End-to-end validation time per block. | | `reth_megaeth_validator_validation_witness_verification_seconds` | Summary | Witness IPA-proof verification time per block. | | `reth_megaeth_validator_validation_block_replay_seconds` | Summary | EVM replay time per block. | +| `reth_megaeth_validator_validation_salt_update_seconds` | Summary | SALT stage time — trie recompute in `full`, changeset compare in `delta`. | +| `reth_megaeth_validator_canonical_lag_seconds` | Summary | Wall time from dispatch to persist, a per-block lag proxy. | | `reth_megaeth_validator_reorg_resets_total` | Counter | Validator cursor rollbacks caused by reorgs. | | `reth_db_table_size{table=
}` | Gauge | On-disk size per database table — watch for disk planning. | -The three duration metrics are recorded as histograms internally but exported as Prometheus **summaries** — pre-computed quantile series plus `_sum` and `_count`, with no `_bucket` series. +Two more appear only when `--validator.report-to` is set: + +| Metric | Type | What it tells you | +| ----------------------------------------------------------- | ------- | --------------------------------------------------------------------------- | +| `reth_megaeth_validator_publisher_active` | Gauge | `1` while the validated-block publisher loop is running, `0` once it exits. | +| `reth_megaeth_validator_publish_failures_total{peer=}` | Counter | Failed `mega_setValidatedBlocks` pushes per peer. | + +A peer that is reachable but refuses a report is not counted as a failure — only transport errors, timeouts, RPC-layer errors, and publish-client panics are. +The `peer` label is the configured URL after `Url` normalization, so `http://rpc-a:8545` appears as `http://rpc-a:8545/` and Prometheus matchers must use the trailing-slash form. + +The `_seconds` metrics are recorded as histograms internally but exported as Prometheus **summaries** — pre-computed quantile series plus `_sum` and `_count`, with no `_bucket` series. Read the quantiles directly (`reth_megaeth_validator_block_validation_duration_seconds{quantile="0.99"}`); `histogram_quantile()` returns nothing for them. ### Key log lines @@ -403,12 +430,19 @@ Preserve the logs around the halt and report the block number to the MegaETH tea **The lag between the sync tip and the validator cursor keeps growing.** Either witness fetches are stalling (witness endpoint slow, rate-limited, or persistently behind the local tip — consider raising `--validator.tip-buffer`) or validation is CPU-bound (raise `--validator.workers` up to the physical core count). -The `block_validation_duration_seconds`, `validation_witness_verification_seconds`, and `validation_block_replay_seconds` quantiles show where the time goes. +The `validation_witness_verification_seconds`, `validation_block_replay_seconds`, and `validation_salt_update_seconds` quantiles break the per-block cost into fetch-verify, replay, and SALT stages; `block_validation_duration_seconds` is the total. -**`reth_megaeth_validator_changeset_fallback_full_total` climbs steadily.** -Delta mode cannot find stored changesets for current blocks, so every block takes the slower full path. +**A fallback counter climbs steadily.** +Delta mode is degrading to the slower full path on most blocks. Validation results remain correct; expect reduced throughput. -On a database whose recent blocks were synced by a current build, steady growth is unexpected and worth reporting. +Which counter is moving says why: + +- `reth_megaeth_validator_changeset_fallback_full_total` — the stored changeset row is missing. + On a database whose recent blocks were synced by a current build this is unexpected and worth reporting. +- `reth_megaeth_validator_anchor_fallback_full_total` — the parent-anchor pair is missing. + A burst around a reorg is normal; sustained growth points at a pruned or bootstrap-edge datadir. + +Switching to `--validator.mode full` removes the fallback churn but not the underlying cost. **Startup fails with `Bootstrap is required, but the database is not empty`.** `--bootstrap-policy required` only works on an empty data directory. From 27994225497b7c778bae0dfb885a03ed658070ac Mon Sep 17 00:00:00 2001 From: "liquan.eth" Date: Sun, 26 Jul 2026 15:33:59 +0800 Subject: [PATCH 05/15] ci: pin lychee so local and CI resolve the same version "latest" resolved to 0.24.2 on CI and 0.23.0 locally, and include_fragments changed type in 0.24 (bool -> string), so lychee.toml could only parse on one of them. Pin 0.23.0: it is the newest release whose macOS assets still match the aqua registry, so it installs on both CI and Apple Silicon. --- .mise.toml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/.mise.toml b/.mise.toml index 52027c9..213255b 100644 --- a/.mise.toml +++ b/.mise.toml @@ -1,5 +1,10 @@ [tools] -lychee = "latest" +# Pinned: `include_fragments` changed type in 0.24 (bool -> string), so an unpinned "latest" +# resolved to a different version on CI than locally and the config parsed on only one of them. +# 0.24.x also renamed its macOS assets (lychee-arm64-macos.tar.gz -> lychee-aarch64-apple-darwin +# .tar.gz) without a matching aqua registry update, so it cannot install on Apple Silicon at all. +# Keep this version in sync with `include_fragments` in lychee.toml. +lychee = "0.23.0" "npm:markdownlint-cli2" = "latest" # Pinned: prettier >= 3.9.0 mangles GitBook template tags ({% tab %}/{% endtab %}) # into markdown table rows, breaking page rendering. Re-evaluate before bumping. From fd58b440023c06f4ccdb98a4fe557cb8fbb64041 Mon Sep 17 00:00:00 2001 From: "liquan.eth" Date: Sun, 26 Jul 2026 17:25:49 +0800 Subject: [PATCH 06/15] Update full-node.md --- docs/node/full-node.md | 67 ++++++++++++++++++++++++++++++++---------- 1 file changed, 51 insertions(+), 16 deletions(-) diff --git a/docs/node/full-node.md b/docs/node/full-node.md index 204da74..235d0ca 100644 --- a/docs/node/full-node.md +++ b/docs/node/full-node.md @@ -44,7 +44,8 @@ A full node needs four inputs beyond the binary itself: 4. **Witness endpoint** — one or more RPC URLs serving [`mega_getBlockWitness`](witness.md), passed via `--validator.rpc-urls`. The endpoint needs no other JSON-RPC method: the validator reads blocks and bytecode from the node's own database and fetches only witnesses remotely. -Plan for a fast NVMe SSD and several hundred GB of storage — the MegaETH Mainnet data directory measures roughly 400 GB as of July 2026 and grows with chain history. +Plan for a fast NVMe SSD and several hundred GB of storage — a full-history MegaETH Mainnet data directory measures roughly 400 GB as of July 2026 and grows with chain history. +A node bootstrapped at the tip (see [Initial sync](#initial-sync)) starts far smaller and grows from its bootstrap block. Validation throughput scales with CPU cores; the validator defaults to one worker per two physical cores. ## Quick start @@ -61,16 +62,25 @@ mega-reth node \ --sequencer-public-key \ --trusted-peers , \ --disable-discovery \ + --bootstrap-policy required \ --max-load 100 \ --validator.rpc-urls \ --metrics 127.0.0.1:9001 \ --log.file.directory ``` +- `--bootstrap-policy required` starts the node at the current chain tip: it fetches a snapshot of the SALT state from the trusted peers and syncs forward from there. + The flag defaults to `never`, and a node started without it begins at block 1 and replays the entire chain — see [Initial sync](#initial-sync) for the trade-off. - `--disable-discovery` is recommended: MegaETH has no public discovery network, and all sync traffic flows through the trusted peers anyway. - `--max-load` caps how many downstream children this node serves in one streaming tree; it is required and has no default. +- `--validator.rpc-urls` must be quoted when the URL carries query parameters — an unquoted `&` splits the command in the shell and the node starts with a truncated URL. - `--metrics` binds the Prometheus endpoint; omit it to disable metrics. +{% hint style="warning" %} +`--bootstrap-policy required` needs an empty `--datadir`. +Pointing it at a directory that already holds a chain synced from genesis exits with `Bootstrap is required, but the database is not empty`. +{% endhint %} + To also serve JSON-RPC, add the standard server flags: ```bash @@ -91,34 +101,52 @@ On start, the node: 1. Opens the data directory and runs crash-consistency recovery. 2. Initializes P2P networking and logs its own `enode` URL. -3. Handshakes with the trusted peers and starts state sync (`Committed block to engine` lines mark per-block progress). -4. Resolves the validator anchor and starts the validation pipeline. +3. Handshakes with the trusted peers and reports what it found (`Loaded local state`), then — on the first run only — fetches the SALT state at the tip and rebuilds the state trie. +4. Starts state sync (`Committed block to engine` lines mark per-block progress). +5. Resolves the validator anchor and starts the validation pipeline. Healthy startup output includes these lines: ```text INFO Starting mega-reth version=... INFO MegaETH P2P networking initialized enode=enode://... +INFO Loaded local state local_block=0 known_block=... bootstrap_status=NotOccurred +INFO Rebuilding trie block_number=... +INFO Rebuilt salt and withdrawal tries salt_elapsed=... withdrawal_elapsed=... INFO Committed block to engine block_number=... source=Fetcher INFO validator: anchor resolved number=... action=SeedFresh INFO megaeth validator pipeline started workers=8 in_flight_multiplier=2 rpc_endpoints=1 mode=delta INFO validator: blocks validated and advanced from=... to=... count=... ``` +`known_block` in the `Loaded local state` line is the tip the trusted peers reported — with `--bootstrap-policy required` that is where the node lands, so `Committed block to engine` should start near it, not at block 1. + ### Subsequent runs All flags shown above are operational, not persisted — re-supply them on every run. + +Keep `--bootstrap-policy required` in place. +Once a bootstrap has finished the flag is a no-op on restart, and leaving it set means a data directory that is ever cleared bootstraps again instead of quietly replaying from genesis. +An interrupted bootstrap resumes on the next start on its own, whatever the flag says. + The validator resumes from its persisted cursor automatically; only pass `--validator.start-block` when you deliberately want to re-anchor (see [Validator pipeline](#validator-pipeline)). ## Initial sync `--bootstrap-policy` selects how an empty data directory reaches the chain tip: -- **`never` (default)** — fetch and apply every historical block from the trusted peers, starting at genesis. - The node serves full chain history but initial sync replays the entire chain. -- **`required`** — fetch a snapshot of the current SALT state from peers instead of replaying history, then sync blocks forward from there. - Bootstrap requires an empty data directory, and a full node additionally needs at least 256 recent blocks before the bootstrap can finish. +- **`required`** — fetch a snapshot of the current SALT state from the trusted peers instead of replaying history, then sync blocks forward from there. + The node bootstraps at the tip its peers report at startup, so it is caught up and serving current state without replaying the chain. + Bootstrap requires an empty data directory, and on a full node it does not complete until the node has also applied the next 256 blocks past that base block. A bootstrapped node cannot serve history from before its bootstrap block and cannot unwind below it. +- **`never` (default)** — fetch and apply every historical block from the trusted peers, starting at genesis. + The node serves full chain history, but initial sync replays the entire chain. + +Pass `required` unless the node has to answer queries about blocks and state that predate its own start. +The flag defaults to `never`, so omitting it is what makes a fresh node start at block 1. + +Bootstrapped state is verified, not taken on trust: when the fetched buckets are complete the node rebuilds the SALT and withdrawal tries from scratch and compares the recomputed state root against the header's, failing the bootstrap on a mismatch. +The rebuild is CPU-bound and can take a while on a large state. {% hint style="warning" %} An interrupted bootstrap can only resume if the node is fewer than 1,800 blocks behind the tip. @@ -126,6 +154,7 @@ Beyond that, the node exits with `The last bootstrap is unfinished and it is too {% endhint %} Either way the embedded validator validates forward only, from the anchor it resolves at startup (see [Anchor](#anchor)); blocks before that point are not re-attested. +On a bootstrapped node both candidates sit at or near the bootstrap block, so attestation begins there rather than at genesis. ## Command-line reference @@ -145,15 +174,15 @@ Command-line flags take precedence over environment variables. ### Networking and state sync -| Flag | Env variable | Default | Description | -| -------------------------- | --------------------------------- | --------- | --------------------------------------------------------------------------------------------------------- | -| `--trusted-peers` | `MEGARETH_TRUSTED_PEERS` | empty | Comma-separated enode URLs of upstream peers. Effectively required — see [Prerequisites](#prerequisites). | -| `--subscribe-trusted-only` | `MEGARETH_SUBSCRIBE_TRUSTED_ONLY` | `true` | Subscribe to block streams from trusted peers only. | -| `--disable-discovery` | `MEGARETH_DISABLE_DISCOVERY` | `false` | Disable peer discovery. Recommended — MegaETH publishes no bootnodes. | -| `--port` | `MEGARETH_PORT` | `30303` | P2P listen port. | -| `--addr` | `MEGARETH_ADDR` | `0.0.0.0` | P2P listen address. | -| `--min-handshake-peers` | `MEGARETH_MIN_HANDSHAKE_PEERS` | `1` | Peers that must complete the state-sync handshake before syncing starts. | -| `--bootstrap-policy` | `MEGARETH_BOOTSTRAP_POLICY` | `never` | `never` (replay from genesis) or `required` (state bootstrap). See [Initial sync](#initial-sync). | +| Flag | Env variable | Default | Description | +| -------------------------- | --------------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------- | +| `--trusted-peers` | `MEGARETH_TRUSTED_PEERS` | empty | Comma-separated enode URLs of upstream peers. Effectively required — see [Prerequisites](#prerequisites). | +| `--subscribe-trusted-only` | `MEGARETH_SUBSCRIBE_TRUSTED_ONLY` | `true` | Subscribe to block streams from trusted peers only. | +| `--disable-discovery` | `MEGARETH_DISABLE_DISCOVERY` | `false` | Disable peer discovery. Recommended — MegaETH publishes no bootnodes. | +| `--port` | `MEGARETH_PORT` | `30303` | P2P listen port. | +| `--addr` | `MEGARETH_ADDR` | `0.0.0.0` | P2P listen address. | +| `--min-handshake-peers` | `MEGARETH_MIN_HANDSHAKE_PEERS` | `1` | Peers that must complete the state-sync handshake before syncing starts. | +| `--bootstrap-policy` | `MEGARETH_BOOTSTRAP_POLICY` | `never` | `never` (replay from genesis) or `required` (state bootstrap at the current tip). See [Initial sync](#initial-sync). | ### Validator flags @@ -322,6 +351,7 @@ MEGARETH_DATA_DIR=/var/lib/megaeth/full-node MEGARETH_SEQUENCER_PUBLIC_KEY= MEGARETH_TRUSTED_PEERS=, MEGARETH_DISABLE_DISCOVERY=true +MEGARETH_BOOTSTRAP_POLICY=required MEGARETH_MAX_LOAD=100 MEGARETH_VALIDATOR_RPC_URLS= MEGARETH_METRICS=127.0.0.1:9001 @@ -350,6 +380,10 @@ WantedBy=multi-user.target `TimeoutStopSec=120` matters: on shutdown the node flushes its databases, and a clean stop can take tens of seconds (5 s for ordinary tasks and 30 s total for critical tasks by default; `MEGARETH_CRITICAL_WAIT_SECS=N` replaces the critical budget with 5 s + N s). +`MEGARETH_BOOTSTRAP_POLICY=required` belongs in the env file permanently. +It applies only to an empty data directory and is ignored once the bootstrap has finished, so it costs nothing on restart and keeps a future re-sync from replaying the chain from genesis. +Adding it to a data directory that was already synced from genesis is the one case that fails — see [Troubleshooting](#troubleshooting). + ## Data directory and maintenance An explicitly passed `--datadir` is used as-is (the OS default adds a `/` level) and contains: @@ -422,6 +456,7 @@ Check that `--trusted-peers` is set, the enode URLs are current, and the peers a **Startup fails with `--node-type full requires at least one --validator.rpc-urls`.** Full nodes cannot start without a witness endpoint. Pass `--validator.rpc-urls` (see [Prerequisites](#prerequisites)). +The `--node-type full` in that message is a stale spelling in the error text itself — the value the flag accepts is `full-node`. **`megaeth validator pipeline halted` and `reth_megaeth_validator_halted` is 1.** The validator hit a deterministic mismatch: this node could not reproduce a block the sequencer committed. From 099f026e8e31903b20ca641fbea89036b0d415fb Mon Sep 17 00:00:00 2001 From: "liquan.eth" Date: Mon, 27 Jul 2026 09:58:49 +0800 Subject: [PATCH 07/15] Update full-node.md --- docs/node/full-node.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/node/full-node.md b/docs/node/full-node.md index 235d0ca..a76f4ba 100644 --- a/docs/node/full-node.md +++ b/docs/node/full-node.md @@ -445,7 +445,8 @@ A full node removes the sequencer's execution from its trusted computing base in The witness endpoint does not need to be trusted for correctness: witness contents are cryptographically verified, so a faulty endpoint can only stall validation, not produce a false attestation. Like the standalone stateless validator, a full node validates the block sequence its peers feed it — it does not derive the canonical chain from L1, and it does not check consistency with the rollup batches posted to L1. -On a full node, the `safe` and `finalized` block tags advance only when an external consensus client drives `engine_forkchoiceUpdated`; sequencer-provided finality markers are ignored. +Once syncing, a full node ignores the sequencer's `safe` and `finalized` markers in block metadata: those tags advance only when an external consensus client drives `engine_forkchoiceUpdated`. +The exception is bootstrap — blocks applied by the bootstrapper persist the sequencer's markers, so a node that has bootstrapped starts its next run with `safe` and `finalized` seeded from the sequencer's view rather than from a consensus client. ## Troubleshooting From cb4bf3e6ee18b5c58602da50293c722af26f45aa Mon Sep 17 00:00:00 2001 From: "liquan.eth" Date: Mon, 27 Jul 2026 11:49:45 +0800 Subject: [PATCH 08/15] Update full-node.md --- docs/node/full-node.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/node/full-node.md b/docs/node/full-node.md index a76f4ba..9b13a46 100644 --- a/docs/node/full-node.md +++ b/docs/node/full-node.md @@ -44,8 +44,10 @@ A full node needs four inputs beyond the binary itself: 4. **Witness endpoint** — one or more RPC URLs serving [`mega_getBlockWitness`](witness.md), passed via `--validator.rpc-urls`. The endpoint needs no other JSON-RPC method: the validator reads blocks and bytecode from the node's own database and fetches only witnesses remotely. -Plan for a fast NVMe SSD and several hundred GB of storage — a full-history MegaETH Mainnet data directory measures roughly 400 GB as of July 2026 and grows with chain history. -A node bootstrapped at the tip (see [Initial sync](#initial-sync)) starts far smaller and grows from its bootstrap block. +Plan for a fast NVMe SSD, sized by how the node syncs — the two paths differ by an order of magnitude. +A node bootstrapped at the tip (see [Initial sync](#initial-sync)) holds only the SALT state at its bootstrap block and everything after: budget at least 500 GB, and size the volume for months of growth rather than for the footprint just after bootstrap. +A node that replays from genesis keeps the chain un-pruned, which on MegaETH Mainnet is roughly 3 TB as of July 2026 (Testnet is roughly 1.6 TB) — provision 4 TB. +Budget for logs separately: they are written outside the data directory and rotate at `--log.file.max-size` × `--log.file.max-files`. Validation throughput scales with CPU cores; the validator defaults to one worker per two physical cores. ## Quick start From 17f186a4655c65395c4f09125b0399f525de8fe3 Mon Sep 17 00:00:00 2001 From: "liquan.eth" Date: Mon, 27 Jul 2026 16:35:10 +0800 Subject: [PATCH 09/15] Update full-node.md --- docs/node/full-node.md | 90 ++++++++++++++++++------------------------ 1 file changed, 39 insertions(+), 51 deletions(-) diff --git a/docs/node/full-node.md b/docs/node/full-node.md index 9b13a46..f281836 100644 --- a/docs/node/full-node.md +++ b/docs/node/full-node.md @@ -30,6 +30,8 @@ The roles available to external operators are: `full-node` and `rpc-node` share the same sync path and data directory layout; the only difference is the embedded validator pipeline that produces attestations. Both types serve `mega_getValidatedChain`, but on an `rpc-node` it only reflects validated tips pushed by an upstream full node via `--validator.report-to` — that node validated nothing itself, and its `anchor` always reads `null`. +An `rpc-node` accepts those pushes only when started with `--rpc.accept-validated-blocks` (env `MEGARETH_RPC_ACCEPT_VALIDATED_BLOCKS`, default `false`). +The flag opens an unauthenticated write path — any client that can reach the RPC port can push validated tips — so enable it only on nodes whose RPC surface is private. To see where attestation actually starts, query the full node that produced the tips. ## Prerequisites @@ -176,32 +178,32 @@ Command-line flags take precedence over environment variables. ### Networking and state sync -| Flag | Env variable | Default | Description | -| -------------------------- | --------------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------- | -| `--trusted-peers` | `MEGARETH_TRUSTED_PEERS` | empty | Comma-separated enode URLs of upstream peers. Effectively required — see [Prerequisites](#prerequisites). | -| `--subscribe-trusted-only` | `MEGARETH_SUBSCRIBE_TRUSTED_ONLY` | `true` | Subscribe to block streams from trusted peers only. | -| `--disable-discovery` | `MEGARETH_DISABLE_DISCOVERY` | `false` | Disable peer discovery. Recommended — MegaETH publishes no bootnodes. | -| `--port` | `MEGARETH_PORT` | `30303` | P2P listen port. | -| `--addr` | `MEGARETH_ADDR` | `0.0.0.0` | P2P listen address. | -| `--min-handshake-peers` | `MEGARETH_MIN_HANDSHAKE_PEERS` | `1` | Peers that must complete the state-sync handshake before syncing starts. | -| `--bootstrap-policy` | `MEGARETH_BOOTSTRAP_POLICY` | `never` | `never` (replay from genesis) or `required` (state bootstrap at the current tip). See [Initial sync](#initial-sync). | +| Flag | Env variable | Default | Description | +| -------------------------- | --------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------- | +| `--trusted-peers` | `MEGARETH_TRUSTED_PEERS` | empty | Comma-separated enode URLs of upstream peers. Effectively required — see [Prerequisites](#prerequisites). | +| `--subscribe-trusted-only` | `MEGARETH_SUBSCRIBE_TRUSTED_ONLY` | `true` | Subscribe to block streams from trusted peers only. The bare flag only sets `true`; pass `false` via the env variable. | +| `--disable-discovery` | `MEGARETH_DISABLE_DISCOVERY` | `false` | Disable peer discovery. Recommended — MegaETH publishes no bootnodes. | +| `--port` | `MEGARETH_PORT` | `30303` | P2P listen port. | +| `--addr` | `MEGARETH_ADDR` | `0.0.0.0` | P2P listen address. | +| `--min-handshake-peers` | `MEGARETH_MIN_HANDSHAKE_PEERS` | `1` | Peers that must complete the state-sync handshake before syncing starts. | +| `--bootstrap-policy` | `MEGARETH_BOOTSTRAP_POLICY` | `never` | `never` (replay from genesis) or `required` (state bootstrap at the current tip). See [Initial sync](#initial-sync). | ### Validator flags Only consulted when `--node-type full-node`; other node types ignore them. -| Flag | Env variable | Default | Description | -| ----------------------------------- | ------------------------------------------ | ------------------ | -------------------------------------------------------------------------------------------------------------- | -| `--validator.rpc-urls` | `MEGARETH_VALIDATOR_RPC_URLS` | — (required) | Witness RPC URLs (comma-separated, round-robin failover). Each must serve `mega_getBlockWitness`. | -| `--validator.mode` | `MEGARETH_VALIDATOR_MODE` | `delta` | Validation strategy: `delta` or `full`. See [Validation modes](#validation-modes). | -| `--validator.workers` | `MEGARETH_VALIDATOR_WORKERS` | physical cores / 2 | Parallel validation workers (minimum 1). Worker count mainly matters during catch-up. | -| `--validator.start-block` | `MEGARETH_VALIDATOR_START_BLOCK` | unset | Trusted block hash to anchor at; validation begins at the next block. Overrides the persisted anchor. | -| `--validator.start-block-wait-secs` | `MEGARETH_VALIDATOR_START_BLOCK_WAIT_SECS` | `0` (indefinite) | How long to wait for the start block to appear locally before the pipeline halts. | -| `--validator.witness-wait-secs` | `MEGARETH_VALIDATOR_WITNESS_WAIT_SECS` | `60` | Per-block witness-fetch deadline; on expiry the block is re-enqueued. | -| `--validator.tip-buffer` | `MEGARETH_VALIDATOR_TIP_BUFFER` | `1` | Blocks of headroom below the local tip before fetching a witness, giving the witness generator time to finish. | -| `--validator.channel-capacity` | `MEGARETH_VALIDATOR_CHANNEL_CAPACITY` | workers × 2 | Target total in-flight witness fetches; values below the worker count are clamped up. | -| `--validator.poll-interval-ms` | `MEGARETH_VALIDATOR_POLL_INTERVAL_MS` | `100` | Tip-refresh interval when the validator is caught up. | -| `--validator.report-to` | `MEGARETH_VALIDATOR_REPORT_TO` | empty (disabled) | RPC-node URLs that receive best-effort `mega_setValidatedBlocks` pushes of the validated tip. | +| Flag | Env variable | Default | Description | +| ----------------------------------- | ------------------------------------------ | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | +| `--validator.rpc-urls` | `MEGARETH_VALIDATOR_RPC_URLS` | — (required) | Witness RPC URLs (comma-separated, round-robin failover). Each must serve `mega_getBlockWitness`. | +| `--validator.mode` | `MEGARETH_VALIDATOR_MODE` | `delta` | Validation strategy: `delta` or `full`. See [Validation modes](#validation-modes). | +| `--validator.workers` | `MEGARETH_VALIDATOR_WORKERS` | physical cores / 2 | Parallel validation workers (minimum 1). Worker count mainly matters during catch-up. | +| `--validator.start-block` | `MEGARETH_VALIDATOR_START_BLOCK` | unset | Trusted block hash to anchor at; validation begins at the next block. Overrides the persisted anchor. | +| `--validator.start-block-wait-secs` | `MEGARETH_VALIDATOR_START_BLOCK_WAIT_SECS` | `0` (indefinite) | How long to wait for the start block to appear locally before the pipeline halts. | +| `--validator.witness-wait-secs` | `MEGARETH_VALIDATOR_WITNESS_WAIT_SECS` | `60` | Per-block witness-fetch deadline; on expiry the block is re-enqueued. | +| `--validator.tip-buffer` | `MEGARETH_VALIDATOR_TIP_BUFFER` | `1` | Blocks of headroom below the local tip before fetching a witness, giving the witness generator time to finish. | +| `--validator.channel-capacity` | `MEGARETH_VALIDATOR_CHANNEL_CAPACITY` | workers × 2 | Target total in-flight witness fetches; values below the worker count are clamped up. | +| `--validator.poll-interval-ms` | `MEGARETH_VALIDATOR_POLL_INTERVAL_MS` | `100` | Tip-refresh interval when the validator is caught up. | +| `--validator.report-to` | `MEGARETH_VALIDATOR_REPORT_TO` | empty (disabled) | RPC-node URLs that receive best-effort `mega_setValidatedBlocks` pushes of the validated tip. Targets must run `--rpc.accept-validated-blocks`. | {% hint style="info" %} When pointing `--validator.rpc-urls` at a rate-limited endpoint, lower `--validator.channel-capacity` to cap concurrent witness fetches — the default of two in-flight fetches per worker is tuned for a dedicated witness provider. @@ -235,7 +237,7 @@ At startup the validator resolves it in this order: 3. Flag unset, fresh database — anchor at the last finalized block, or at the first synced canonical head if nothing is finalized yet. The resolved anchor is persisted, so the choice survives restarts. -The `validator: anchor resolved` log line reports the outcome: `action=Skip` (anchor unchanged), `SeedFresh` (first anchor), or `OverrideWithRollback` (re-anchored via flag). +The `validator: anchor resolved` log line reports the outcome: `action=Skip` (anchor unchanged), `SeedFresh` (first anchor, flag unset), or `OverrideWithRollback` (re-anchored via flag). {% hint style="warning" %} `--validator.start-block` takes a block **hash**, not a number, and re-anchoring clears the validated cursor — the range validated under the old anchor is no longer attested. @@ -317,12 +319,12 @@ curl -sX POST http://localhost:8545 \ | `reth_megaeth_validator_reorg_resets_total` | Counter | Validator cursor rollbacks caused by reorgs. | | `reth_db_table_size{table=
}` | Gauge | On-disk size per database table — watch for disk planning. | -Two more appear only when `--validator.report-to` is set: +Two more track the validated-block publisher: -| Metric | Type | What it tells you | -| ----------------------------------------------------------- | ------- | --------------------------------------------------------------------------- | -| `reth_megaeth_validator_publisher_active` | Gauge | `1` while the validated-block publisher loop is running, `0` once it exits. | -| `reth_megaeth_validator_publish_failures_total{peer=}` | Counter | Failed `mega_setValidatedBlocks` pushes per peer. | +| Metric | Type | What it tells you | +| ----------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------- | +| `reth_megaeth_validator_publisher_active` | Gauge | `1` while the publisher loop is running. Exported on every full node; stays `0` unless `--validator.report-to` is set. | +| `reth_megaeth_validator_publish_failures_total{peer=}` | Counter | Failed `mega_setValidatedBlocks` pushes per peer. The per-peer series appears after that peer's first failure. | A peer that is reachable but refuses a report is not counted as a failure — only transport errors, timeouts, RPC-layer errors, and publish-client panics are. The `peer` label is the configured URL after `Url` normalization, so `http://rpc-a:8545` appears as `http://rpc-a:8545/` and Prometheus matchers must use the trailing-slash form. @@ -381,6 +383,7 @@ WantedBy=multi-user.target ``` `TimeoutStopSec=120` matters: on shutdown the node flushes its databases, and a clean stop can take tens of seconds (5 s for ordinary tasks and 30 s total for critical tasks by default; `MEGARETH_CRITICAL_WAIT_SECS=N` replaces the critical budget with 5 s + N s). +If you raise `MEGARETH_CRITICAL_WAIT_SECS`, raise `TimeoutStopSec` to stay above 5 s + N s — when the stop timeout expires first, systemd escalates to SIGKILL, the hard kill the backup rules below warn against. `MEGARETH_BOOTSTRAP_POLICY=required` belongs in the env file permanently. It applies only to an empty data directory and is ignored once the bootstrap has finished, so it costs nothing on restart and keeps a future re-sync from replaying the chain from genesis. @@ -388,15 +391,15 @@ Adding it to a data directory that was already synced from genesis is the one ca ## Data directory and maintenance -An explicitly passed `--datadir` is used as-is (the OS default adds a `/` level) and contains: +An explicitly passed `--datadir` is used as-is (the OS default adds a `/` level) and contains, among other entries: -| Path | Contents | -| ------------------ | ---------------------------------------------------------------------------------------------------------- | -| `db/main_db/` | RocksDB instance holding headers, transactions, receipts, SALT state and trie, and changesets. | -| `db/index_db/` | RocksDB instance holding history indexes and transaction lookups (created for `full-node` and `rpc-node`). | -| `known-peers.json` | Persisted peer set, written on shutdown. | -| `jwt.hex` | Auto-generated Engine API JWT secret. | -| `discovery-secret` | P2P identity key. | +| Path | Contents | +| ------------------ | ----------------------------------------------------------------------------------------------------------------------------- | +| `db/main_db/` | RocksDB instance holding headers, transactions, receipts, transaction-hash lookups, and the SALT state, trie, and changesets. | +| `db/index_db/` | RocksDB instance holding history indexes and account/storage changesets (created for `full-node` and `rpc-node`). | +| `known-peers.json` | Persisted peer set, written on shutdown. | +| `jwt.hex` | Auto-generated Engine API JWT secret. | +| `discovery-secret` | P2P identity key. | The data directory layout is tied to the node type — reuse a data directory only with a node type that shares its layout (`full-node` and `rpc-node` do; a `sequencer` data directory does not). @@ -420,22 +423,7 @@ mega-reth tables-height \ # Expect: "tables height is consistent true at block height: " ``` -### Unwinding to an earlier block - -`mega-reth recovery` rewinds the database to a target block after verifying consistency: - -```bash -mega-reth recovery \ - --datadir \ - --chain \ - --node-type full-node \ - --target_block_number -``` - -{% hint style="warning" %} -`recovery` rewrites the database destructively — back up the data directory first. -Maintenance subcommands (`tables-height`, `recovery`, `db`) do not raise the process file-descriptor limit the way `node` does; raise it yourself (`ulimit -n 1048576`) before running them against a large database. -{% endhint %} +Maintenance subcommands (`tables-height`, `db`) do not raise the process file-descriptor limit the way `node` does — raise it yourself (`ulimit -n 1048576`) before running them against a large database. ## Trust model From 7e92502b4581a67c9b224afd7881b9cb2171469a Mon Sep 17 00:00:00 2001 From: "liquan.eth" Date: Mon, 27 Jul 2026 17:18:15 +0800 Subject: [PATCH 10/15] doc: note the data directory is chain-bound --- docs/node/full-node.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/node/full-node.md b/docs/node/full-node.md index f281836..fa7783b 100644 --- a/docs/node/full-node.md +++ b/docs/node/full-node.md @@ -402,6 +402,7 @@ An explicitly passed `--datadir` is used as-is (the OS default adds a ` Date: Mon, 27 Jul 2026 17:44:14 +0800 Subject: [PATCH 11/15] Update full-node.md --- docs/node/full-node.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/node/full-node.md b/docs/node/full-node.md index fa7783b..97840d0 100644 --- a/docs/node/full-node.md +++ b/docs/node/full-node.md @@ -26,7 +26,8 @@ The roles available to external operators are: | ----------- | ----------------------------------------------------------------------------------------------------------------------- | | `full-node` | Syncs blocks and state, serves JSON-RPC, and re-validates every block with the embedded stateless validator. This page. | | `rpc-node` | Replica node: syncs blocks and state and serves JSON-RPC without re-execution. | -| `sequencer` | The block producer. Default value — a full node must pass `--node-type full-node` explicitly. | + +The flag defaults to `sequencer` — the block-producer role MegaETH itself operates — so a full node must pass `--node-type full-node` explicitly. `full-node` and `rpc-node` share the same sync path and data directory layout; the only difference is the embedded validator pipeline that produces attestations. Both types serve `mega_getValidatedChain`, but on an `rpc-node` it only reflects validated tips pushed by an upstream full node via `--validator.report-to` — that node validated nothing itself, and its `anchor` always reads `null`. From a3e92b59f345b78d8cea5bca39b31a90ec01515d Mon Sep 17 00:00:00 2001 From: "liquan.eth" Date: Tue, 28 Jul 2026 08:57:15 +0800 Subject: [PATCH 12/15] doc: add production RPC env example and bootstrap timing note --- docs/node/full-node.md | 64 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) diff --git a/docs/node/full-node.md b/docs/node/full-node.md index 97840d0..1fe556e 100644 --- a/docs/node/full-node.md +++ b/docs/node/full-node.md @@ -76,6 +76,7 @@ mega-reth node \ - `--bootstrap-policy required` starts the node at the current chain tip: it fetches a snapshot of the SALT state from the trusted peers and syncs forward from there. The flag defaults to `never`, and a node started without it begins at block 1 and replays the entire chain — see [Initial sync](#initial-sync) for the trade-off. + Expect the first start to run for a while before blocks commit: the snapshot fetch and the CPU-bound trie rebuild both take time on Mainnet-sized state, and an interrupted bootstrap only resumes within 1,800 blocks of the tip — let it finish. - `--disable-discovery` is recommended: MegaETH has no public discovery network, and all sync traffic flows through the trusted peers anyway. - `--max-load` caps how many downstream children this node serves in one streaming tree; it is required and has no default. - `--validator.rpc-urls` must be quoted when the URL carries query parameters — an unquoted `&` splits the command in the shell and the node starts with a truncated URL. @@ -390,6 +391,69 @@ If you raise `MEGARETH_CRITICAL_WAIT_SECS`, raise `TimeoutStopSec` to stay above It applies only to an empty data directory and is ignored once the bootstrap has finished, so it costs nothing on restart and keeps a future re-sync from replaying the chain from genesis. Adding it to a data directory that was already synced from genesis is the one case that fails — see [Troubleshooting](#troubleshooting). +### Serving JSON-RPC in production + +A full node that doubles as an RPC endpoint — the replica workload plus attestation — extends the env file with the server and capacity settings: + +```ini +# Append to /etc/megaeth/full-node.env +# JSON-RPC servers +MEGARETH_HTTP=true +MEGARETH_HTTP_ADDR=127.0.0.1 +MEGARETH_HTTP_PORT=8545 +MEGARETH_HTTP_API=eth,net,web3 +MEGARETH_HTTP_CORS_DOMAIN=* +MEGARETH_HTTP_DISABLE_COMPRESSION=true +MEGARETH_WS=true +MEGARETH_WS_ADDR=127.0.0.1 +MEGARETH_WS_PORT=8546 +MEGARETH_WS_API=eth,net,web3 +MEGARETH_WS_ALLOWED_ORIGINS=* +MEGARETH_RPC_WS_PING_INTERVAL=0 +MEGARETH_IPC_PATH=/var/lib/megaeth/full-node/reth.ipc +# Engine API (used only when an external consensus client is attached) +MEGARETH_AUTH_ADDR=127.0.0.1 +MEGARETH_AUTH_PORT=8551 +# RPC capacity +MEGARETH_RPC_MAX_CONNECTIONS=50000 +MEGARETH_RPC_MAX_TRACING_REQUESTS=200 +MEGARETH_RPC_MAX_LOGS_PER_RESPONSE=50000 +MEGARETH_RPC_MAX_REQUEST_SIZE=150 +MEGARETH_RPC_MAX_RESPONSE_SIZE=1600 +MEGARETH_RPC_GAS_CAP=18446744073709551615 +MEGARETH_RPC_CACHE_MAX_BLOCKS=1000 +MEGARETH_RPC_ETH_PROOF_WINDOW=10000 +# Transaction pool (the *_MAX_SIZE values are megabytes) +MEGARETH_TXPOOL_GAS_LIMIT=2000000000 +MEGARETH_TXPOOL_MAX_ACCOUNT_SLOTS=100 +MEGARETH_TXPOOL_PENDING_MAX_COUNT=200000 +MEGARETH_TXPOOL_PENDING_MAX_SIZE=55 +MEGARETH_TXPOOL_QUEUED_MAX_COUNT=500000 +MEGARETH_TXPOOL_QUEUED_MAX_SIZE=137 +MEGARETH_ROLLUP_DISABLE_TX_POOL_GOSSIP=true +# Storage and pipeline (values shown are the current defaults) +MEGARETH_PIPELINE_PENDING=16 +MEGARETH_BLOCKHASH_CACHE_WINDOW=256 +MEGARETH_CODE_CACHE_SIZE_MB=65536 +# P2P +MEGARETH_MAX_INBOUND_PEERS=100 +# Logging +MEGARETH_LOG_STDOUT_FORMAT=terminal +MEGARETH_LOG_FILE_FILTER=debug +MEGARETH_LOG_FILE_FORMAT=terminal +MEGARETH_LOG_COLOR=never +MEGARETH_LOG_FILE_MAX_SIZE=400 +MEGARETH_LOG_FILE_MAX_FILES=50 +``` + +- Keep the servers on loopback behind a reverse proxy, and widen `MEGARETH_HTTP_API`/`MEGARETH_WS_API` beyond `eth,net,web3` only on private networks — the exposure warning in [Quick start](#quick-start) applies verbatim here. +- `MEGARETH_RPC_MAX_REQUEST_SIZE`, `MEGARETH_RPC_MAX_RESPONSE_SIZE`, and the `MEGARETH_TXPOOL_*_MAX_SIZE` values are megabytes; raise the response cap when serving `debug_trace*` or large `eth_getLogs` responses. +- `MEGARETH_RPC_GAS_CAP=18446744073709551615` (`u64::MAX`) removes the `eth_call`/`eth_estimateGas` gas cap (default 50,000,000) — usual for trace and simulation providers. +- `MEGARETH_RPC_WS_PING_INTERVAL=0` disables server-side WebSocket pings, leaving connection liveness to the reverse proxy. +- A node that answers queries about blocks and state from before its own start needs `MEGARETH_BOOTSTRAP_POLICY=never` and the full-replay disk budget from [Prerequisites](#prerequisites) instead of `required`. +- Unlike a shell command line, an `EnvironmentFile` needs no quoting around a `MEGARETH_VALIDATOR_RPC_URLS` value that carries query parameters. +- `--rpc.max-subscriptions-per-connection` has no `MEGARETH_*` environment variable — pass it on the command line if the default of 1,024 needs changing. + ## Data directory and maintenance An explicitly passed `--datadir` is used as-is (the OS default adds a `/` level) and contains, among other entries: From 265cb020fb453b2ba3375246a974d69fa5202d5f Mon Sep 17 00:00:00 2001 From: "liquan.eth" Date: Tue, 28 Jul 2026 15:46:05 +0800 Subject: [PATCH 13/15] update --- docs/SUMMARY.md | 2 +- docs/node/full-node.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 851dc33..481a2a6 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -99,5 +99,5 @@ - [Stateless Validation](node/stateless-validation.md) - [Validator Architecture](node/validator-architecture.md) - [Get Block Witness](node/witness.md) -- [Run a Full Node](node/full-node.md) +- [Full Node](node/full-node.md) - [Debug Trace Server](node/debug-trace-server.md) diff --git a/docs/node/full-node.md b/docs/node/full-node.md index 1fe556e..12f7ecd 100644 --- a/docs/node/full-node.md +++ b/docs/node/full-node.md @@ -2,7 +2,7 @@ description: Run a MegaETH full node — sync blocks and state from the sequencer stream and re-execute every block with the embedded stateless validator. --- -# Run a full node +# Full node A **full node** is a role of `mega-reth`, the MegaETH node client. Like a replica node, it syncs blocks and state from the sequencer stream and serves the full JSON-RPC surface. From 53236a276f9f7ff33a0fbf3a6224868945e983b5 Mon Sep 17 00:00:00 2001 From: "liquan.eth" Date: Tue, 28 Jul 2026 17:02:21 +0800 Subject: [PATCH 14/15] Update full-node.md --- docs/node/full-node.md | 59 +++++++++++++++--------------------------- 1 file changed, 21 insertions(+), 38 deletions(-) diff --git a/docs/node/full-node.md b/docs/node/full-node.md index 12f7ecd..77caa13 100644 --- a/docs/node/full-node.md +++ b/docs/node/full-node.md @@ -29,11 +29,9 @@ The roles available to external operators are: The flag defaults to `sequencer` — the block-producer role MegaETH itself operates — so a full node must pass `--node-type full-node` explicitly. -`full-node` and `rpc-node` share the same sync path and data directory layout; the only difference is the embedded validator pipeline that produces attestations. -Both types serve `mega_getValidatedChain`, but on an `rpc-node` it only reflects validated tips pushed by an upstream full node via `--validator.report-to` — that node validated nothing itself, and its `anchor` always reads `null`. -An `rpc-node` accepts those pushes only when started with `--rpc.accept-validated-blocks` (env `MEGARETH_RPC_ACCEPT_VALIDATED_BLOCKS`, default `false`). -The flag opens an unauthenticated write path — any client that can reach the RPC port can push validated tips — so enable it only on nodes whose RPC surface is private. -To see where attestation actually starts, query the full node that produced the tips. +`full-node` and `rpc-node` share the same sync path and data directory layout; the difference is the embedded validator pipeline. +The validator produces no witnesses or SALT proofs — it consumes them: it re-executes each block against a witness whose IPA proof authenticates the replay inputs, and records the outcome as the node's validated range. +The node serves that validated range via `mega_getValidatedChain` and, with `--validator.report-to`, pushes the validated tip to other nodes — see [Validator flags](#validator-flags). ## Prerequisites @@ -48,7 +46,8 @@ A full node needs four inputs beyond the binary itself: The endpoint needs no other JSON-RPC method: the validator reads blocks and bytecode from the node's own database and fetches only witnesses remotely. Plan for a fast NVMe SSD, sized by how the node syncs — the two paths differ by an order of magnitude. -A node bootstrapped at the tip (see [Initial sync](#initial-sync)) holds only the SALT state at its bootstrap block and everything after: budget at least 500 GB, and size the volume for months of growth rather than for the footprint just after bootstrap. +A node bootstrapped at the tip (see [Initial sync](#initial-sync)) holds only the SALT state at its bootstrap block and everything after — roughly 5 GB right after bootstrap as of July 2026, which is all a short-lived test node needs. +A long-running node should budget at least 500 GB and size the volume for months of growth rather than for the footprint just after bootstrap. A node that replays from genesis keeps the chain un-pruned, which on MegaETH Mainnet is roughly 3 TB as of July 2026 (Testnet is roughly 1.6 TB) — provision 4 TB. Budget for logs separately: they are written outside the data directory and rotate at `--log.file.max-size` × `--log.file.max-files`. Validation throughput scales with CPU cores; the validator defaults to one worker per two physical cores. @@ -76,15 +75,16 @@ mega-reth node \ - `--bootstrap-policy required` starts the node at the current chain tip: it fetches a snapshot of the SALT state from the trusted peers and syncs forward from there. The flag defaults to `never`, and a node started without it begins at block 1 and replays the entire chain — see [Initial sync](#initial-sync) for the trade-off. - Expect the first start to run for a while before blocks commit: the snapshot fetch and the CPU-bound trie rebuild both take time on Mainnet-sized state, and an interrupted bootstrap only resumes within 1,800 blocks of the tip — let it finish. + Expect the first start to run for a while before blocks commit: the snapshot fetch and the CPU-bound trie rebuild both take time on Mainnet-sized state. - `--disable-discovery` is recommended: MegaETH has no public discovery network, and all sync traffic flows through the trusted peers anyway. - `--max-load` caps how many downstream children this node serves in one streaming tree; it is required and has no default. -- `--validator.rpc-urls` must be quoted when the URL carries query parameters — an unquoted `&` splits the command in the shell and the node starts with a truncated URL. +- `--validator.rpc-urls` must be quoted when the URL carries query parameters, e.g. `--validator.rpc-urls 'https:///rpc?key=&sig='`. + Unquoted, the shell splits the command at the `&` and the node starts with the URL truncated to `https:///rpc?key=`. - `--metrics` binds the Prometheus endpoint; omit it to disable metrics. {% hint style="warning" %} -`--bootstrap-policy required` needs an empty `--datadir`. -Pointing it at a directory that already holds a chain synced from genesis exits with `Bootstrap is required, but the database is not empty`. +`--bootstrap-policy required` triggers a bootstrap only on an empty `--datadir`; restarting a node whose bootstrap already finished is fine — the flag is then a no-op. +The one case that fails is a data directory synced from genesis without a bootstrap: the node exits with `Bootstrap is required, but the database is not empty`. {% endhint %} To also serve JSON-RPC, add the standard server flags: @@ -134,6 +134,7 @@ All flags shown above are operational, not persisted — re-supply them on every Keep `--bootstrap-policy required` in place. Once a bootstrap has finished the flag is a no-op on restart, and leaving it set means a data directory that is ever cleared bootstraps again instead of quietly replaying from genesis. An interrupted bootstrap resumes on the next start on its own, whatever the flag says. +On a healthy restart the `Loaded local state` line reports `bootstrap_status=Finished()`, where `` is the bootstrap base block. The validator resumes from its persisted cursor automatically; only pass `--validator.start-block` when you deliberately want to re-anchor (see [Validator pipeline](#validator-pipeline)). @@ -346,8 +347,7 @@ Read the quantiles directly (`reth_megaeth_validator_block_validation_duration_s ## Deployment -Run the node under a service manager in production. -A minimal systemd setup: +Run the node under a service manager in production, configured through an env file: ```ini # /etc/megaeth/full-node.env @@ -364,29 +364,6 @@ MEGARETH_METRICS=127.0.0.1:9001 MEGARETH_LOG_FILE_DIRECTORY=/var/log/megaeth ``` -```ini -# /etc/systemd/system/megaeth-full-node.service -[Unit] -Description=MegaETH full node -After=network-online.target -Wants=network-online.target - -[Service] -User=megaeth -EnvironmentFile=/etc/megaeth/full-node.env -ExecStart=/usr/local/bin/mega-reth node -Restart=on-failure -RestartSec=5 -LimitNOFILE=1048576 -TimeoutStopSec=120 - -[Install] -WantedBy=multi-user.target -``` - -`TimeoutStopSec=120` matters: on shutdown the node flushes its databases, and a clean stop can take tens of seconds (5 s for ordinary tasks and 30 s total for critical tasks by default; `MEGARETH_CRITICAL_WAIT_SECS=N` replaces the critical budget with 5 s + N s). -If you raise `MEGARETH_CRITICAL_WAIT_SECS`, raise `TimeoutStopSec` to stay above 5 s + N s — when the stop timeout expires first, systemd escalates to SIGKILL, the hard kill the backup rules below warn against. - `MEGARETH_BOOTSTRAP_POLICY=required` belongs in the env file permanently. It applies only to an empty data directory and is ignored once the bootstrap has finished, so it costs nothing on restart and keeps a future re-sync from replaying the chain from genesis. Adding it to a data directory that was already synced from genesis is the one case that fails — see [Troubleshooting](#troubleshooting). @@ -451,7 +428,7 @@ MEGARETH_LOG_FILE_MAX_FILES=50 - `MEGARETH_RPC_GAS_CAP=18446744073709551615` (`u64::MAX`) removes the `eth_call`/`eth_estimateGas` gas cap (default 50,000,000) — usual for trace and simulation providers. - `MEGARETH_RPC_WS_PING_INTERVAL=0` disables server-side WebSocket pings, leaving connection liveness to the reverse proxy. - A node that answers queries about blocks and state from before its own start needs `MEGARETH_BOOTSTRAP_POLICY=never` and the full-replay disk budget from [Prerequisites](#prerequisites) instead of `required`. -- Unlike a shell command line, an `EnvironmentFile` needs no quoting around a `MEGARETH_VALIDATOR_RPC_URLS` value that carries query parameters. +- Unlike a shell command line, an env file needs no quoting around a `MEGARETH_VALIDATOR_RPC_URLS` value that carries query parameters. - `--rpc.max-subscriptions-per-connection` has no `MEGARETH_*` environment variable — pass it on the command line if the default of 1,024 needs changing. ## Data directory and maintenance @@ -472,6 +449,8 @@ It is also bound to its chain: opening it with a different `--chain` fails at st ### Graceful shutdown and backups Stop the node with SIGTERM or SIGINT and wait for the process to exit before touching the data directory. +A clean stop flushes the databases and can take tens of seconds: 5 s for ordinary tasks and 30 s total for critical tasks by default, and `MEGARETH_CRITICAL_WAIT_SECS=N` replaces the critical budget with 5 s + N s. +If a service manager supervises the node, keep its stop timeout above that budget so it does not escalate to SIGKILL mid-flush. {% hint style="warning" %} Never SIGKILL the node or copy the data directory while it runs. @@ -524,6 +503,10 @@ Preserve the logs around the halt and report the block number to the MegaETH tea Either witness fetches are stalling (witness endpoint slow, rate-limited, or persistently behind the local tip — consider raising `--validator.tip-buffer`) or validation is CPU-bound (raise `--validator.workers` up to the physical core count). The `validation_witness_verification_seconds`, `validation_block_replay_seconds`, and `validation_salt_update_seconds` quantiles break the per-block cost into fetch-verify, replay, and SALT stages; `block_validation_duration_seconds` is the total. +**The log shows an occasional `Witness not found` error near the tip.** +Normal: witness generation runs slightly behind the chain head, so a fetch can land before the witness exists — the validator backs off and retries, and `--validator.tip-buffer` sets how much headroom it leaves the generator. +It needs attention only when the same blocks keep failing and the validation lag grows — the previous item. + **A fallback counter climbs steadily.** Delta mode is degrading to the slower full path on most blocks. Validation results remain correct; expect reduced throughput. @@ -537,8 +520,8 @@ Which counter is moving says why: Switching to `--validator.mode full` removes the fallback churn but not the underlying cost. **Startup fails with `Bootstrap is required, but the database is not empty`.** -`--bootstrap-policy required` only works on an empty data directory. -Either clear the data directory to bootstrap, or drop the flag to keep the existing database. +`--bootstrap-policy required` can only start a bootstrap on an empty data directory, and this one already holds a chain synced from genesis. +Either clear the data directory to bootstrap at the tip, or drop the flag to keep the existing full-history database. ## Related pages From 31a17e774d403063a1058be9cbdff413c4d99cb2 Mon Sep 17 00:00:00 2001 From: "liquan.eth" Date: Tue, 28 Jul 2026 17:43:02 +0800 Subject: [PATCH 15/15] Update full-node.md --- docs/node/full-node.md | 72 +++++++++++++++++++----------------------- 1 file changed, 33 insertions(+), 39 deletions(-) diff --git a/docs/node/full-node.md b/docs/node/full-node.md index 77caa13..757d1b9 100644 --- a/docs/node/full-node.md +++ b/docs/node/full-node.md @@ -68,7 +68,7 @@ mega-reth node \ --disable-discovery \ --bootstrap-policy required \ --max-load 100 \ - --validator.rpc-urls \ + --validator.rpc-urls '' \ --metrics 127.0.0.1:9001 \ --log.file.directory ``` @@ -87,7 +87,7 @@ mega-reth node \ The one case that fails is a data directory synced from genesis without a bootstrap: the node exits with `Bootstrap is required, but the database is not empty`. {% endhint %} -To also serve JSON-RPC, add the standard server flags: +To also serve JSON-RPC, append the standard server flags (add a trailing `\` to the last line above first): ```bash --http --http.addr 127.0.0.1 --http.port 8545 \ @@ -98,9 +98,7 @@ To also serve JSON-RPC, add the standard server flags: The `mega_*` methods are not part of this selection — they are merged into every enabled transport regardless of `--http.api`, so `mega_getValidatedChain` works with the namespace set above. {% hint style="warning" %} -Bind to `127.0.0.1` unless the node is deliberately a public RPC endpoint. -Serving `0.0.0.0` with `admin`, `debug`, `trace`, or `txpool` enabled exposes `admin_addPeer` (unauthenticated peer manipulation), `debug_trace*` (unmetered remote CPU and memory), and `txpool_content` (pending-transaction leakage) to anyone who can reach the port. -To serve traffic publicly, keep the node on loopback and put a reverse proxy in front of it that terminates TLS, rate-limits, and forwards only the namespaces you intend to expose. +Bind to `127.0.0.1` unless the node is deliberately a public RPC endpoint — before exposing anything beyond loopback, see [Serving JSON-RPC in production](#serving-json-rpc-in-production). {% endhint %} On start, the node: @@ -131,8 +129,7 @@ INFO validator: blocks validated and advanced from=... to=... count=... All flags shown above are operational, not persisted — re-supply them on every run. -Keep `--bootstrap-policy required` in place. -Once a bootstrap has finished the flag is a no-op on restart, and leaving it set means a data directory that is ever cleared bootstraps again instead of quietly replaying from genesis. +Keep `--bootstrap-policy required` in place — a data directory that is ever cleared then bootstraps again instead of quietly replaying from genesis. An interrupted bootstrap resumes on the next start on its own, whatever the flag says. On a healthy restart the `Loaded local state` line reports `bootstrap_status=Finished()`, where `` is the bootstrap base block. @@ -142,15 +139,12 @@ The validator resumes from its persisted cursor automatically; only pass `--vali `--bootstrap-policy` selects how an empty data directory reaches the chain tip: -- **`required`** — fetch a snapshot of the current SALT state from the trusted peers instead of replaying history, then sync blocks forward from there. - The node bootstraps at the tip its peers report at startup, so it is caught up and serving current state without replaying the chain. +- **`required`** — fetch a snapshot of the current SALT state from the trusted peers at the tip they report, then sync forward from there — caught up and serving current state without replaying history. Bootstrap requires an empty data directory, and on a full node it does not complete until the node has also applied the next 256 blocks past that base block. A bootstrapped node cannot serve history from before its bootstrap block and cannot unwind below it. -- **`never` (default)** — fetch and apply every historical block from the trusted peers, starting at genesis. - The node serves full chain history, but initial sync replays the entire chain. +- **`never` (default)** — fetch and apply every historical block starting at genesis; the node serves full chain history at the cost of replaying the entire chain. Pass `required` unless the node has to answer queries about blocks and state that predate its own start. -The flag defaults to `never`, so omitting it is what makes a fresh node start at block 1. Bootstrapped state is verified, not taken on trust: when the fetched buckets are complete the node rebuilds the SALT and withdrawal tries from scratch and compares the recomputed state root against the header's, failing the bootstrap on a mismatch. The rebuild is CPU-bound and can take a while on a large state. @@ -193,7 +187,7 @@ Command-line flags take precedence over environment variables. ### Validator flags -Only consulted when `--node-type full-node`; other node types ignore them. +Only consulted when `--node-type full-node`. | Flag | Env variable | Default | Description | | ----------------------------------- | ------------------------------------------ | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | @@ -209,7 +203,7 @@ Only consulted when `--node-type full-node`; other node types ignore them. | `--validator.report-to` | `MEGARETH_VALIDATOR_REPORT_TO` | empty (disabled) | RPC-node URLs that receive best-effort `mega_setValidatedBlocks` pushes of the validated tip. Targets must run `--rpc.accept-validated-blocks`. | {% hint style="info" %} -When pointing `--validator.rpc-urls` at a rate-limited endpoint, lower `--validator.channel-capacity` to cap concurrent witness fetches — the default of two in-flight fetches per worker is tuned for a dedicated witness provider. +When pointing `--validator.rpc-urls` at a rate-limited endpoint, lower `--validator.channel-capacity` to cap concurrent witness fetches — the default is tuned for a dedicated witness provider. {% endhint %} ### Logging flags @@ -220,6 +214,7 @@ When pointing `--validator.rpc-urls` at a rate-limited endpoint, lower `--valida | `--log.stdout.format` | `MEGARETH_LOG_STDOUT_FORMAT` | `terminal` | Console format: `terminal`, `log-fmt`, or `json`. | | `--log.file.directory` | `MEGARETH_LOG_FILE_DIRECTORY` | OS cache dir (`.../reth/logs`) | Directory for rotated log files; the chain ID is appended. | | `--log.file.filter` | `MEGARETH_LOG_FILE_FILTER` | `debug` | Log filter for file output. | +| `--log.file.format` | `MEGARETH_LOG_FILE_FORMAT` | `terminal` | File log format: `terminal`, `log-fmt`, or `json`. | | `--log.file.max-size` | `MEGARETH_LOG_FILE_MAX_SIZE` | `200` | Max log file size (MB) before rotation. | | `--log.file.max-files` | `MEGARETH_LOG_FILE_MAX_FILES` | `5` | Number of rotated log files to keep. | | `--color` | `MEGARETH_LOG_COLOR` | `always` | ANSI color: `always`, `auto`, or `never`. | @@ -234,7 +229,7 @@ A validator failure never tears down the node — RPC and sync keep running. The anchor is the trusted block validation starts from; blocks are validated from `anchor + 1` onward. At startup the validator resolves it in this order: -1. `--validator.start-block ` set — anchor at that block, overriding any persisted anchor and clearing the validated cursor. +1. `--validator.start-block ` set — anchor at that block, overriding any persisted anchor. The node waits (bounded by `--validator.start-block-wait-secs`) for the hash to appear locally via state sync. 2. Flag unset, persisted anchor exists — resume from the persisted cursor. 3. Flag unset, fresh database — anchor at the last finalized block, or at the first synced canonical head if nothing is finalized yet. @@ -244,7 +239,7 @@ The `validator: anchor resolved` log line reports the outcome: `action=Skip` (an {% hint style="warning" %} `--validator.start-block` takes a block **hash**, not a number, and re-anchoring clears the validated cursor — the range validated under the old anchor is no longer attested. -Once the anchor is persisted, restarts with the same flag are skipped without rewriting it, but remove the flag anyway so a copied unit file or later restart does not silently pin validation to an old block. +Once the anchor is persisted, restarts with the same flag are skipped without rewriting it, but remove the flag anyway so a copied env file or later restart does not silently pin validation to an old block. {% endhint %} ### Validation modes @@ -255,20 +250,14 @@ Both modes IPA-verify the witness and re-execute the block with the `stateless-v - **`full`** — recompute the SALT trie root from the replay output and compare it with the header's state root. Slower, but independent of stored changesets and parent anchors. -Delta mode never runs unanchored: a block falls back to `full` behavior for that block alone when either input is missing, and each cause has its own counter. - -| Missing input | Counter | -| ---------------------------------------------------------------- | ------------------------------------------------------ | -| Stored changeset row | `reth_megaeth_validator_changeset_fallback_full_total` | -| Parent-anchor pair (reorg mid-fetch, or a pruned/bootstrap edge) | `reth_megaeth_validator_anchor_fallback_full_total` | - +Delta mode never runs unanchored: a block falls back to `full` behavior for that block alone when either input is missing, and each cause has its own counter — `reth_megaeth_validator_changeset_fallback_full_total` for a missing changeset row, `reth_megaeth_validator_anchor_fallback_full_total` for a missing parent-anchor pair (reorg mid-fetch, or a pruned/bootstrap edge). Both are counted once per validated block, not per fetch attempt. Validation results are identical either way — only throughput differs. ### Mismatch handling A deterministic validation failure — a state-root, receipts-root, logs-bloom, or gas mismatch, or a changeset mismatch — halts the validation pipeline. -The node logs `megaeth validator pipeline halted` at error level, sets the `reth_megaeth_validator_halted` gauge to 1, and keeps serving RPC and syncing. +The node logs `megaeth validator pipeline halted` at error level and sets the `reth_megaeth_validator_halted` gauge to 1. The stalled validated height is the operator alert: it means the sequencer produced a block this node could not reproduce. On a reorg, the validator restarts from its persisted cursor — it does not search for the divergence point — and re-validates forward on the new canonical chain; `reth_megaeth_validator_reorg_resets_total` counts these events. @@ -291,7 +280,7 @@ reth_megaeth_validator_cursor 6907396 The difference is the validation lag in blocks. A small, stable lag is normal; a growing lag means validation cannot keep pace (see [Troubleshooting](#troubleshooting)). -Full nodes also serve `mega_getValidatedChain`, which returns the anchor and the validated tip — available on any enabled transport, whatever `--http.api` selects: +Full nodes also serve `mega_getValidatedChain`, which returns the anchor and the validated tip — available on any enabled transport, whatever `--http.api` selects; the call below assumes the `--http` server flags from [Quick start](#quick-start): ```bash curl -sX POST http://localhost:8545 \ @@ -332,8 +321,10 @@ Two more track the validated-block publisher: A peer that is reachable but refuses a report is not counted as a failure — only transport errors, timeouts, RPC-layer errors, and publish-client panics are. The `peer` label is the configured URL after `Url` normalization, so `http://rpc-a:8545` appears as `http://rpc-a:8545/` and Prometheus matchers must use the trailing-slash form. +{% hint style="warning" %} The `_seconds` metrics are recorded as histograms internally but exported as Prometheus **summaries** — pre-computed quantile series plus `_sum` and `_count`, with no `_bucket` series. Read the quantiles directly (`reth_megaeth_validator_block_validation_duration_seconds{quantile="0.99"}`); `histogram_quantile()` returns nothing for them. +{% endhint %} ### Key log lines @@ -364,13 +355,12 @@ MEGARETH_METRICS=127.0.0.1:9001 MEGARETH_LOG_FILE_DIRECTORY=/var/log/megaeth ``` -`MEGARETH_BOOTSTRAP_POLICY=required` belongs in the env file permanently. -It applies only to an empty data directory and is ignored once the bootstrap has finished, so it costs nothing on restart and keeps a future re-sync from replaying the chain from genesis. -Adding it to a data directory that was already synced from genesis is the one case that fails — see [Troubleshooting](#troubleshooting). +`MEGARETH_BOOTSTRAP_POLICY=required` belongs in the env file permanently — it is a no-op once the bootstrap has finished (see [Subsequent runs](#subsequent-runs)), and it keeps a future re-sync from replaying the chain from genesis. ### Serving JSON-RPC in production -A full node that doubles as an RPC endpoint — the replica workload plus attestation — extends the env file with the server and capacity settings: +A full node that doubles as an RPC endpoint — the replica workload plus attestation — extends the env file with server and capacity settings. +The block below is a typical configuration, not a requirement: every key is optional and falls back to its default when omitted, and the values are a starting point to adapt: ```ini # Append to /etc/megaeth/full-node.env @@ -386,17 +376,19 @@ MEGARETH_WS_ADDR=127.0.0.1 MEGARETH_WS_PORT=8546 MEGARETH_WS_API=eth,net,web3 MEGARETH_WS_ALLOWED_ORIGINS=* +# 0 disables server-side WebSocket pings; the reverse proxy owns liveness MEGARETH_RPC_WS_PING_INTERVAL=0 MEGARETH_IPC_PATH=/var/lib/megaeth/full-node/reth.ipc # Engine API (used only when an external consensus client is attached) MEGARETH_AUTH_ADDR=127.0.0.1 MEGARETH_AUTH_PORT=8551 -# RPC capacity +# RPC capacity (the *_SIZE values are megabytes) MEGARETH_RPC_MAX_CONNECTIONS=50000 MEGARETH_RPC_MAX_TRACING_REQUESTS=200 MEGARETH_RPC_MAX_LOGS_PER_RESPONSE=50000 MEGARETH_RPC_MAX_REQUEST_SIZE=150 MEGARETH_RPC_MAX_RESPONSE_SIZE=1600 +# u64::MAX removes the eth_call/eth_estimateGas gas cap (default 50,000,000) MEGARETH_RPC_GAS_CAP=18446744073709551615 MEGARETH_RPC_CACHE_MAX_BLOCKS=1000 MEGARETH_RPC_ETH_PROOF_WINDOW=10000 @@ -423,17 +415,19 @@ MEGARETH_LOG_FILE_MAX_SIZE=400 MEGARETH_LOG_FILE_MAX_FILES=50 ``` -- Keep the servers on loopback behind a reverse proxy, and widen `MEGARETH_HTTP_API`/`MEGARETH_WS_API` beyond `eth,net,web3` only on private networks — the exposure warning in [Quick start](#quick-start) applies verbatim here. -- `MEGARETH_RPC_MAX_REQUEST_SIZE`, `MEGARETH_RPC_MAX_RESPONSE_SIZE`, and the `MEGARETH_TXPOOL_*_MAX_SIZE` values are megabytes; raise the response cap when serving `debug_trace*` or large `eth_getLogs` responses. -- `MEGARETH_RPC_GAS_CAP=18446744073709551615` (`u64::MAX`) removes the `eth_call`/`eth_estimateGas` gas cap (default 50,000,000) — usual for trace and simulation providers. -- `MEGARETH_RPC_WS_PING_INTERVAL=0` disables server-side WebSocket pings, leaving connection liveness to the reverse proxy. +{% hint style="warning" %} +Serving `0.0.0.0` with `admin`, `debug`, `trace`, or `txpool` enabled exposes `admin_addPeer` (unauthenticated peer manipulation), `debug_trace*` (unmetered remote CPU and memory), and `txpool_content` (pending-transaction leakage) to anyone who can reach the port. +Keep the servers on loopback behind a reverse proxy that terminates TLS, rate-limits, and forwards only the namespaces you intend to expose; widen `MEGARETH_HTTP_API`/`MEGARETH_WS_API` beyond `eth,net,web3` only on private networks. +{% endhint %} + +- Raise `MEGARETH_RPC_MAX_RESPONSE_SIZE` when serving `debug_trace*` or large `eth_getLogs` responses. - A node that answers queries about blocks and state from before its own start needs `MEGARETH_BOOTSTRAP_POLICY=never` and the full-replay disk budget from [Prerequisites](#prerequisites) instead of `required`. - Unlike a shell command line, an env file needs no quoting around a `MEGARETH_VALIDATOR_RPC_URLS` value that carries query parameters. - `--rpc.max-subscriptions-per-connection` has no `MEGARETH_*` environment variable — pass it on the command line if the default of 1,024 needs changing. ## Data directory and maintenance -An explicitly passed `--datadir` is used as-is (the OS default adds a `/` level) and contains, among other entries: +The data directory contains, among other entries: | Path | Contents | | ------------------ | ----------------------------------------------------------------------------------------------------------------------------- | @@ -458,9 +452,11 @@ Never SIGKILL the node or copy the data directory while it runs. The node repairs an inconsistent directory in place on the next start, but a backup taken from a running or killed node bakes the inconsistency in. {% endhint %} -To back up: stop the node, copy the data directory, and verify the copy: +To back up: stop the node, copy the data directory, and verify the copy. +Maintenance subcommands (`tables-height`, `db`) do not raise the process file-descriptor limit the way `node` does — raise it before running them against a large database: ```bash +ulimit -n 1048576 mega-reth tables-height \ --datadir \ --chain \ @@ -468,8 +464,6 @@ mega-reth tables-height \ # Expect: "tables height is consistent true at block height: " ``` -Maintenance subcommands (`tables-height`, `db`) do not raise the process file-descriptor limit the way `node` does — raise it yourself (`ulimit -n 1048576`) before running them against a large database. - ## Trust model A full node removes the sequencer's execution from its trusted computing base in two layers: @@ -479,7 +473,7 @@ A full node removes the sequencer's execution from its trusted computing base in The witness endpoint does not need to be trusted for correctness: witness contents are cryptographically verified, so a faulty endpoint can only stall validation, not produce a false attestation. -Like the standalone stateless validator, a full node validates the block sequence its peers feed it — it does not derive the canonical chain from L1, and it does not check consistency with the rollup batches posted to L1. +Like the [standalone stateless validator](stateless-validation.md#trust-model), a full node validates the block sequence its peers feed it — it does not derive the canonical chain from L1, and it does not check consistency with the rollup batches posted to L1. Once syncing, a full node ignores the sequencer's `safe` and `finalized` markers in block metadata: those tags advance only when an external consensus client drives `engine_forkchoiceUpdated`. The exception is bootstrap — blocks applied by the bootstrapper persist the sequencer's markers, so a node that has bootstrapped starts its next run with `safe` and `finalized` seeded from the sequencer's view rather than from a consensus client.