feat(yearn): alert when the Envio indexer falls behind

[spalen0]

Envio was down for 24+ hours this week and none of our monitoring noticed. The monitors that read from the indexer — large flows, timelock alerts, 3jane borrower watch — go quiet rather than loud when it stalls: GraphQL keeps answering, it just stops returning new rows, so an outage is indistinguishable from a quiet day.

This adds an hourly freshness check that makes that silence loud.

How it works

1. Query chain_metadata at ENVIO_GRAPHQL_URL for each chain's latest_processed_block.
2. Fetch that block's timestamp via ChainManager and compare it to wall-clock time.
3. Alert the errors channel when a chain is more than --max-lag-minutes (default 60) stale, when an expected chain reports no sync state at all, or when the endpoint itself is unset/unreachable/empty.

Step 2 is what makes the check trustworthy. Envio parks chain_metadata.block_height at the last processed block once a chain looks caught up, so comparing those two fields reports a stalled indexer as zero blocks behind — the same trap called out in the indexer's own monitoring dashboard source.

Step 3 covers the inverse trap, caught in review: an empty result set is not good news. If a chain drops out of the indexer's config, or comes back from a restart with no processed block, it stops appearing in chain_metadata — and a check that only looks at what it was given reports every remaining chain fresh while that chain's monitors sit blind. EXPECTED_CHAINS is the authority on what must be present, and anything absent from it alerts.

It already caught a live incident

First dry run this morning found Mainnet 206 days behind, mid re-sync. Half an hour later the indexer had been wiped and restarted from scratch — every chain reset to its start block and is now crawling forward:

⚠️ [yearn] Envio indexer problem on 6 chain(s) — events may be missing from monitoring alerts.

- Mainnet (chain 1): 1358d 20h behind, last block 15918582
- Optimism (chain 10): 1149d 9h behind, last block 104673357
- Polygon (chain 137): 844d 20h behind, last block 55436408
- Base (chain 8453): 388d 2h behind, last block 32455806
- Arbitrum (chain 42161): 520d 12h behind, last block 308855365
- Katana (chain 747474): 140d 16h behind, last block 26336482

Threshold: 1h
Dashboard: https://envio-monitoring.yearn.dev/

Every one of those chains reported itself as caught up via block_height. Worth someone confirming the re-index is intentional.

Notes for review

• EXPECTED_CHAINS is spelled out, not derived from the Chain enum. The enum is the repo's list of supported chains, not a statement about what Envio indexes — deriving from it would start alerting that the indexer is "missing" a chain nobody asked it to index the moment someone adds an enum member for an unrelated protocol. Add a chain to EXPECTED_CHAINS when its events start feeding a monitor.
• Re-alert cooldown (default 6h, --alert-cooldown-hours). Without it, a multi-day re-sync like the one above posts an identical alert every hour for weeks. Each chain alerts on entering trouble, then at most once per window, then once on recovery. Tracked per chain, so one lagging chain never suppresses another's first alert. Endpoint-down alerts bypass the cooldown — that's the acute outage case.
• Raw eth_getBlockByNumber instead of eth.get_block. web3 rejects the 97-byte PoA extraData on Polygon and pre-Bedrock Optimism blocks, and old blocks are exactly what a lagging indexer points at — this silently blanked those two chains until fixed. Added Web3Client.make_request for it, so provider rotation and retries still apply.
• A chain whose RPC is unreachable is skipped, never alerted on — a broken provider is not a stale indexer.

Alerts route to the existing errors channel (TELEGRAM_*_ERRORS) labelled [yearn]. Scheduled first in the hourly profile so the freshness signal lands before the monitors that depend on it.

Also moves the duplicated format_duration helper into utils/formatting.py.

Testing

• 22 unit tests: staleness thresholds, missing/unsynced chains, per-chain cooldown independence, recovery, and the 5xx / connection-refused / timeout / non-JSON-body transport paths. Full suite passes (693 passed, 4 skipped).
• ruff format / ruff check clean. mypy adds no new errors (repo-wide run fails on a pre-existing stale build/ duplicate-module error).
• Verified end-to-end against the live indexer with Telegram sends disabled — output above is from that run.

🤖 Generated with Claude Code