Skip to content

docs(validators): add node state-recovery and rollup-outage runbooks#435

Open
MuncleUscles wants to merge 1 commit into
mainfrom
docs/node-runbooks
Open

docs(validators): add node state-recovery and rollup-outage runbooks#435
MuncleUscles wants to merge 1 commit into
mainfrom
docs/node-runbooks

Conversation

@MuncleUscles

Copy link
Copy Markdown
Member

Description

Adds two node-operator runbook pages under pages/validators/, plus navigation and cross-links.

1. state-recovery.mdx — Node State Recovery & Re-sync
The operator's guide to recovering a node that has fallen behind or lost state. It documents the two paths that actually exist today:

  • Snapshot seeding (fast): create a snapshot on a healthy node via the ops-port POST /snapshot endpoint, then seed a target node from the archive. The archive is a consistent checkpoint of the Pebble chain DB (SSTables / MANIFEST / WAL).
  • Full re-sync from genesis (trustless): start with an empty database and let the node re-scan the rollup and re-execute all history through GenVM, rebuilding local state block by block.

Honesty note (per the audit): the snapshot endpoint is create-only — there is no first-class restore / import command or endpoint. Restoring from a snapshot is a manual filesystem drop-in performed while the node is stopped, and the page says so explicitly and flags where the tooling is genuinely thin, rather than inventing a restore command.

2. rollup-outage.mdx — Rollup / Sequencer Outage: Incident Response
What a node operator does when the underlying rollup / sequencer (the GenLayer Chain L2) is unreachable, structured as detect → communicate → wait & verify → resume → post-incident:

  • Detection: doctor GenLayer-Chain RPC/WS failures, the /health zksync-connectivity check, and the transport circuit breaker (observable via metrics, not logs).
  • No false slashing: slashing and quarantine are decided on-chain by the consensus contracts; the node has no self-penalty path and skips its duties rather than mis-voting when it can't reach the chain. It waits and retries (transient classification, no replacement-budget burn, no nonce gaps).
  • Not-active vs quarantine: the page distinguishes the node-side not-active state from on-chain quarantine (idleness/tribunal-driven, which presupposes an advancing chain), and describes automatic resume (priming retries on its own).
  • Grounded in the Bradbury outage of 2026-07-01 as the precedent.

Both pages are wired into pages/validators/_meta.json and cross-link the setup guide, monitoring, upgrade, staking (priming/quarantine), and slashing pages. monitoring.mdx and upgrade.mdx gain reciprocal links.

All technical claims were verified against genlayer-node@origin/v0.6-dev (snapshot API, sync/circuit-breaker, health/doctor checks, transient-retry behavior) and the genlayer-consensus contracts (quarantine/slashing are on-chain). Validated with scripts/list-docs-routes.js, scripts/generate-full-docs.js, and scripts/check-llm-exports.js (209 pages, exports OK).

Release Notes

Added two validator runbooks to docs.genlayer.com: Node State Recovery & Re-sync (snapshot seeding and trustless full re-sync from genesis) and Rollup / Sequencer Outage: Incident Response (detect, wait, and resume without false slashing).

Types of Changes

  • docs: Documentation changes

Checklist

  • My code follows the style guidelines of this project (matches existing pages/validators/*.mdx Nextra/MDX conventions)
  • I have performed a self-review of my own changes
  • My changes generate no new warnings (generate-full-docs.js and check-llm-exports.js pass; both routes register)
  • Any dependent changes have been merged and published
  • No fabricated facts — all claims verified against the genlayer-node and genlayer-consensus repositories, and thin tooling is called out rather than invented

Add two operator runbook pages under pages/validators/:

- state-recovery.mdx: recovering a node that has fallen behind or lost
  state. Documents the two real paths — seeding a new node from a
  POST /snapshot archive (create-only endpoint; restore is a manual
  filesystem drop-in, no first-class restore command), and a trustless
  full re-sync from genesis that re-executes history through GenVM.
  Honestly flags where restore tooling is thin.
- rollup-outage.mdx: incident response when the underlying rollup /
  sequencer is unreachable. Detection (doctor + /health zksync-connectivity
  + transport circuit-breaker metrics), why validators are not falsely
  slashed (slashing/quarantine are on-chain; the node skips duties rather
  than mis-voting), status-comms, and automatic resume. Grounded in the
  Bradbury outage of 2026-07-01 as precedent.

Wire both into pages/validators/_meta.json and cross-link setup-guide,
monitoring, upgrade, staking, and slashing.
@netlify

netlify Bot commented Jul 3, 2026

Copy link
Copy Markdown

Deploy Preview for genlayer-docs ready!

Name Link
🔨 Latest commit 0c6a70d
🔍 Latest deploy log https://app.netlify.com/projects/genlayer-docs/deploys/6a47939fddad5c0008866650
😎 Deploy Preview https://deploy-preview-435--genlayer-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai

coderabbitai Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@MuncleUscles, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 48 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 7bb83fec-7f11-4c24-9c07-4c4af18a6c9d

📥 Commits

Reviewing files that changed from the base of the PR and between 19558ee and 0c6a70d.

📒 Files selected for processing (5)
  • pages/validators/_meta.json
  • pages/validators/monitoring.mdx
  • pages/validators/rollup-outage.mdx
  • pages/validators/state-recovery.mdx
  • pages/validators/upgrade.mdx
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch docs/node-runbooks

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant