Skip to content

doc: add full node operator guide - #87

Draft
flyq wants to merge 16 commits into
mainfrom
liquan/doc/add_full_node
Draft

doc: add full node operator guide#87
flyq wants to merge 16 commits into
mainfrom
liquan/doc/add_full_node

Conversation

@flyq

@flyq flyq commented Jul 26, 2026

Copy link
Copy Markdown
Member

Summary

Adds docs/node/full-node.md, an operator guide for running mega-reth as a full node — a replica node plus an embedded asynchronous stateless validator that re-executes every committed block against a SALT witness and halts on mismatch. Registered in docs/SUMMARY.md under Node Operation, before Debug Trace Server.

Contents

  • Node types — what --node-type full-node is versus rpc-node and sequencer, and how a full node publishes its validated range (mega_getValidatedChain, --validator.report-to).
  • Prerequisites & quick start — genesis file, sequencer public key, trusted peers, witness endpoint; a copy-pasteable first-run command and the log lines that mark a healthy startup.
  • Initial sync--bootstrap-policy never vs required, and the 1,800-block resume limit for an interrupted bootstrap.
  • Command-line reference — core, networking/state-sync, validator, and logging flags with their MEGARETH_* environment variables and defaults.
  • Validator pipeline — anchor resolution order, delta vs full validation modes, and mismatch/reorg handling.
  • Monitoring — sync-tip vs validated-cursor lag, mega_getValidatedChain, and the metrics and log lines worth alerting on.
  • Deployment & maintenance — env-file deployment with a production RPC-serving example (server, capacity, txpool, and logging settings with source-verified MEGARETH_* names), data directory layout, graceful shutdown and backup rules, and tables-height backup verification.
  • Trust model & troubleshooting — what a full node does and does not remove from its trusted computing base, plus the common startup and lag failure modes.

Verification

Flag names, env vars, defaults, metric names, log lines, and error strings are verified against mega-reth develop @ ab60376 (stateless-validator library v2.0.15). Additionally verified empirically with a release build: mega_* methods are served on every enabled transport regardless of the --http.api list, an existing datadir opened with a different --chain is refused at startup (GenesisHashMismatch), and the log directory gains the chain-ID suffix in both env-only and argv configurations. The quoted startup error --node-type full requires at least one --validator.rpc-urls is intentional — the stale full spelling is what the current build emits (crates/megaeth/validator/src/args.rs:221).

Notes

mega-reth source and binaries are permissioned; the page states this up front and points readers at the access-request path, so publishing commits to that path being in place.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ 4 findings

  • Summary: Doc-only PR adding docs/node/full-node.md (414 lines) and one docs/SUMMARY.md entry. Coverage, tone, and cross-links all read well.
  • Inline comments: 4 anchorable findings — one Major (quickstart RPC bind exposes privileged namespaces on 0.0.0.0), two Minor (metric type inconsistency; likely --target_block_number typo), one Nit (placeholder convention).

Pre-mortem (unverified):

  • docs/node/full-node.md:241 — Verify that mega_* methods really are merged into every enabled transport regardless of --http.api on the actual mega-reth build. If not, monitoring against the recommended --http.api set (which does not list mega) silently returns -32601 and validator-halt alerts never fire.
  • docs/node/full-node.md:132,320 — Verify mega-reth behavior when --datadir is explicitly set and opened against a different --chain. The current wording ("used as-is when set") could invite cross-chain reuse; if the binary does not refuse, add a warning.

Verification reminder: the PR description acknowledges that flags, defaults, metric names, and log strings from the permissioned mega-reth repo are unverified. Please record the mega-reth commit SHA the tables were compared against before merging — a stale metric name silently kills Prometheus alerts, and a stale flag name breaks the copy-pasteable quickstart.

Still open from earlier reviews: none.

Comment thread docs/node/full-node.md Outdated
Comment thread docs/node/full-node.md Outdated
Comment thread docs/node/full-node.md Outdated
Comment thread docs/node/full-node.md Outdated
- Bind the quickstart JSON-RPC example to loopback with eth,net,web3 and
  warn about exposing admin/debug/trace/txpool on a public interface
- Label the validator duration metrics as Prometheus summaries and explain
  that histogram_quantile() does not apply to them
- Add replay-full and verifier to the node types table
- Note that index_db is created for full-node, rpc-node, and replay-full
- Use an angle-bracket placeholder for the reth_db_table_size label
@flyq
flyq marked this pull request as ready for review July 26, 2026 06:29
@flyq
flyq requested a review from Troublor as a code owner July 26, 2026 06:29

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ 1 finding

  • Summary: Reviewed the added full-node operator guide and its docs/SUMMARY.md entry.
  • Inline comments: 1 anchorable finding.
  • Prior thread on the reth_db_table_size label placeholder has been addressed and was resolved.

Pre-mortem (unverified):

  • docs/node/full-node.md:83 and :250 — Verify against the current mega-reth build that mega_* methods really are merged into every enabled RPC transport regardless of --http.api. The quick-start --http.api eth,net,web3 and the mega_getValidatedChain monitoring recipe both depend on this. If the current build gates mega_* on the namespace list, the monitoring probe will silently return -32601 and alerting recipes built from this page will not detect a stalled validator.
  • docs/node/full-node.md:397 — Verify the exact wording of the startup error. The doc quotes --node-type full requires at least one --validator.rpc-urls, but the documented --node-type values are full-node/rpc-node/etc. — there is no full. If the real error string uses full-node, operators grepping the exact string will miss this troubleshooting entry.

Comment thread docs/node/full-node.md Outdated
flyq added 6 commits July 26, 2026 14:51
The replayer, replay-full and verifier roles are internal and not open to
external operators, so drop them from the table and say plainly that it
lists the roles operators can actually run. Also stop claiming index_db is
exclusive to full-node and rpc-node, and apply prettier formatting.
Re-verified the page against mega-reth ba30a29a6
(liquan/perf/fullnode-state-sync-validator). Flags, defaults, log lines and
anchor resolution are unchanged; the metrics surface and delta-mode
fallbacks grew.

- Document the metrics added on the branch: validation_salt_update_seconds,
  anchor_fallback_full_total, failures_panic_total, canonical_lag_seconds,
  and the report-to pair publisher_active / publish_failures_total{peer}
- Delta mode has two fallback causes now (missing changeset row, missing
  parent anchor), each with its own counter; note it never runs unanchored
- Split the fallback troubleshooting entry to say which counter means what
- Note that mega_getValidatedChain reports anchor=null on an rpc-node and
  tip=null until the first attestation lands
- Fix the initial-sync anchor sentence, which claimed the validator anchors
  at the first synced head rather than the last finalized block
"latest" resolved to 0.24.2 on CI and 0.23.0 locally, and include_fragments
changed type in 0.24 (bool -> string), so lychee.toml could only parse on
one of them. Pin 0.23.0: it is the newest release whose macOS assets still
match the aqua registry, so it installs on both CI and Apple Silicon.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧭 Re-review update

  • Reviewed incremental changes since 12b9243 (metrics additions, delta-mode fallback split, node-types table trim, systemd MEGARETH_BOOTSTRAP_POLICY=required, .mise.toml lychee pin + lychee.toml fragments type flip).
  • No new actionable findings on this pass. Cross-links resolve, terminology is consistent, and the anchor-resolution / delta-fallback rewrites read cleanly.

Still open from earlier reviews:

  • --target_block_number should be kebab-case — the snake_case spelling is still present at docs/node/full-node.md:432, and every other flag on the page is kebab-case. If the binary rejects the unknown token, the operator hits it on mega-reth recovery, which the warning right below flags as destructive. Verify against mega-reth recovery --help and update.

Pre-mortem (unverified):

  • docs/node/full-node.md:377,383 — verify the sample TimeoutStopSec=120 still bounds the worst-case flush window on a large Mainnet datadir. The same paragraph notes MEGARETH_CRITICAL_WAIT_SECS=N extends the critical-task budget to 5 + N seconds, so an operator raising N past ~115 for a heavy datadir would silently overshoot TimeoutStopSec, triggering systemd SIGKILL escalation mid-flush while the doc separately states RocksDB runs with the WAL disabled. tables-height on the copy catches the resulting inconsistency, but the sample unit itself does not tie the two knobs together. Consider recommending TimeoutStopSec=infinity (or wiring it to MEGARETH_CRITICAL_WAIT_SECS) in the example.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ Review complete — 2 open question(s)

Reviewed head 17f186a4.

Doc PR adds docs/node/full-node.md, one SUMMARY.md entry, pins lychee to 0.23.0, and reverts include_fragments to the 0.23-era bool syntax.

Open questions:
Open question · Medium confidence

  • Do mega_* RPC methods really surface on every enabled transport regardless of the --http.api namespace list on the current mega-reth build?
  • Why it matters: The quick-start uses --http.api eth,net,web3 and the monitoring section instructs operators to call mega_getValidatedChain over that same transport. If the current build gates mega_* behind the --http.api list (as vanilla reth does for other namespaces), the documented monitoring recipe and validator-halt alerting silently return -32601 and operators never see a stalled validator.
  • How to verify: In the mega-reth source, check where the mega_* RPC module is registered on the HTTP/WS server (look for RpcModule::merge/server_builder calls around the mega namespace) and confirm it is added unconditionally, not gated on the Selection derived from --http.api. Alternatively, start a node with --http.api eth,net,web3 and confirm a curl of mega_getValidatedChain returns a result rather than method-not-found.

Open question · Low confidence

  • Does mega-reth node refuse to open an existing data directory when --chain points at a different network than the one that populated it?
  • Why it matters: docs/node/full-node.md:174 says an explicitly passed --datadir is used as-is (only the OS default appends the chain ID), and the systemd example hard-codes /var/lib/megaeth/full-node. If the binary does not refuse cross-chain reuse, an operator who reruns the same unit against a different --chain (e.g. swapping Testnet genesis for Mainnet) could silently corrupt or misinterpret the on-disk database. The Data directory section warns about node-type reuse but not chain reuse.
  • How to verify: Check the mega-reth startup sequence for a chain-ID/genesis-hash comparison between the loaded chain spec and the on-disk state (look for something like expected_chain_id != stored_chain_id bail in the datadir open path). If no such guard exists, the doc should add a warning that an explicit --datadir must be scoped per chain (e.g. /var/lib/megaeth/<chain>/full-node).

@flyq

flyq commented Jul 27, 2026

Copy link
Copy Markdown
Member Author

Answers to the two open questions from the last review round. Verified against mega-reth develop @ ab60376 in source, and empirically with a release build whose code on the tested paths is byte-identical to that commit (git diff is empty for crates/megaeth/node/src/add_ons.rs, crates/rpc/rpc-builder/, crates/storage/db-common/src/init.rs, and crates/megaeth/chainspec/src/genesis_check.rs).

1. mega_* methods vs --http.api — confirmed: merged unconditionally on every enabled transport.

Source: MegaethMegaApi is merged via modules.merge_configured(...) (crates/megaeth/node/src/add_ons.rs:496-502), and merge_configured merges into HTTP, WS, and IPC unconditionally (crates/rpc/rpc-builder/src/lib.rs:1770-1778) — unlike merge_if_module_configured, it is not gated on the --http.api selection.

Empirical, on a node started with --http.api eth,net,web3:

  • mega_getValidatedChain{"jsonrpc":"2.0","id":1,"result":{"anchor":null,"tip":null}}
  • txpool_content (control, also outside the list) → {"error":{"code":-32601,"message":"Method not found"}}

The namespace list gates ordinary namespaces but not mega_*, so the monitoring recipe on the page works as documented.

2. Cross-chain reuse of an existing --datadir — confirmed: refused at startup, no silent corruption.

Source: genesis init bails with GenesisHashMismatch (crates/storage/db-common/src/init.rs:55,116), plus mega-reth's own check in crates/megaeth/chainspec/src/genesis_check.rs.

Empirical: opening a datadir initialized with chain ID 424242 using --chain dev shuts down immediately with

Error: genesis hash in the storage does not match the specified chainspec: chainspec is 0x12ef98d6…, database is 0xb411b61f…

The page now documents this in the Data directory section: "It is also bound to its chain: opening it with a different --chain fails at startup with genesis hash in the storage does not match the specified chainspec." (commit 7e92502).

@flyq
flyq marked this pull request as draft July 28, 2026 07:46
@mega-maxwell

mega-maxwell Bot commented Jul 28, 2026

Copy link
Copy Markdown

Claude review status

Living comment — rewritten in place. The review workflow keeps this single comment up to date instead of posting a new one each round, so it always describes the latest reviewed commit and the earlier text is intentionally gone. No reply is needed here; answer findings and questions in the review threads it links to.

✅ Review clean

Last reviewed: head 31a17e77 · updated 2026-07-28T09:47:31+00:00

New this round: 0 finding(s), 0 question(s) · Resolved this round: 0 · Open questions: 0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant