Isolate per-route cert failures and hot-reload rotated cert files

[kriszyp]

Summary

Makes symphony resilient to TLS cert rotation. Three fixes in one PR:

1. Per-route fault isolation (src/router.rs + ts/server.ts) — build_route_table no longer aborts the whole table on the first route that fails to build. A route whose cert can't be built (e.g. rustls KeyMismatch) is logged (eprintln!, matching the proxy.rs route-warning convention) and skipped; every other route on the port survives. This feeds both initial construction and the updateConfig hot-swap.

   • On a hot-swap, the last-good route is carried forward for that SNI instead of dropped — a cert+key pair rotates as two non-atomic files, so a transient mismatch mid-rotation is normal, and the SNI keeps serving the still-valid old cert until the next reconcile heals it. On initial build (no prior route) the SNI is simply dropped.
   • The server-side per-proxy guard in doReconcile is widened to cover SymphonyProxy construction + start(), so a failing port-set can no longer skip the remaining ones. seen still holds the key, so a skipped existing proxy is left running, not torn down.
   • The isolation extends to cert file-read failures: resolveRoute() catches a per-route ENOENT/unreadable cert file and hands Rust an unbuildable cert, so build_route_table drops/carries-forward just that SNI rather than throwing out of toProxyConfig and aborting the whole port-set (a review finding from gemini-code-assist).

2. Cert-file hot reload (ts/server.ts) — the standalone server now watches the cert/key files referenced by the config (grouped by parent dir, deduped, re-derived + torn down every reconcile so watchers don't leak; 'error' listener on each) and reconciles on change. A running symphony picks up an on-disk renewal without a config.json write or restart. Debounced (300ms) and coalesced into the already-serialized reconcile().

3. status.json ownership guard (ts/server.ts) — stop() now only deletes status.json when its pid matches process.pid (best-effort; a missing/garbage file must not throw). During a version upgrade the replacement starts first (SO_REUSEPORT overlap) and rewrites status.json with its own pid before the incumbent retires; an unconditional unlink would clobber the successor's file and make host-manager's health check read null → respawn a duplicate.

Why

A single tenant's LE cert rotated on disk, but symphony's config.json still inlined that tenant's old cert chain while referencing its live key by path → leaf pubkey ≠ key → rustls KeyMismatch. Two symphony behaviors turned a one-tenant rotation into a host-wide TLS outage (every TLS listener — 443/9925/8883/9933 — failed to bind, only :80 stayed up): the reconcile aborted on the first bad route, and cert changes were only observed on a config.json write/restart, so it detonated days later on the next restart.

Coordination (host-manager — land in order)

Layered story: hm #132 stops emitting a bad pair; Fix 1 tolerates one if it arrives; Fix 2 gives live rotation pickup; hm #133 + Fix 3 make the upgrade handoff fail-safe.

• host-manager #132 (public cert emitted by file instead of inlining the chain) is the separate root-cause fix. This PR does not assume all certs are by-file — the internal self-signed cert stays inline; Fix 2 only hot-reloads file-referenced certs, Fix 1 is the net if a mismatched inline/file pair still reaches symphony.
• host-manager #133 (start-before-stop upgrade handoff via SO_REUSEPORT overlap) is gated on Fix 3 — #133 must not merge/deploy ahead of this PR, or the retiring process will delete the successor's status.json.

Where to look / open items for the reviewer

• build_route_table last-good retention (src/router.rs) — the hot-swap now carries the previous route forward on a transient build failure. Note the persistent-failure behavior: a route that stays broken keeps serving its last-good (still on-disk) cert indefinitely (logged each reconcile), trading staleness for availability. This was the sharpest cross-model finding (a partial-table hot-swap would otherwise drop the SNI where the pre-change code retained the last-good table).
• listenerSig now hashes resolved cert contents (ts/server.ts) — a listener-level defaultCert/mTLS rotation only applies if it forces a proxy recreate (the Rust default_listener_tls is frozen at construction; updateConfig only swaps routes). Keying the signature off the resolved listeners (cert bytes included) makes a rotation change the signature. Covered by a test.
• Cert-watcher teardown/dedup (ts/server.ts) — one fs.watch per distinct parent dir, filtered to referenced basenames against a live certFilesByDir set so a config repoint is honored without recreating watchers; stale dirs are closed each reconcile.
• Known limitation (follow-up): basename-filtered dir watching handles in-place / rename rotation (what host-manager does). Kubernetes projected-volume / cert-manager ..data symlink-swap rotation isn't covered (the real basename never gets a direct event) — a follow-up if symphony ever runs under that model.
• Out of scope (pre-existing): SymphonyProxy.start() resolves before the actual bind (the bind + its errors live in the spawned Rust listener task), so the widened guard gives no protection against an async bind failure. Unchanged by this diff; worth tracking separately.
• Unrelated pre-existing red: sni::tests::test_is_grease fails on main (the is_grease bitmask only matches 0x0A0A, not 0x1A1A/0xFAFA); and the Integration Test – Harper UDS proxy CI check fails on main too (the harper dev-dep can't resolve its own ../core/* paths). Neither is touched here.

Tests

• src/router.rs: bad_route_is_skipped_not_fatal, transient_failure_retains_last_good_on_hot_swap.
• __test__/server.spec.ts: route-cert rotation, listener-defaultCert rotation, per-route isolation (healthy co-tenant survives a broken route on the same port + skip log), route cert file-read isolation (co-tenant missing cert file keeps last-good while another route rotates), and status.json ownership guard (owned file removed on stop; successor-owned file untouched).

npm test (33 pass) and cargo test (new tests pass) green. Harper integration leg needs a bootable Harper instance (env-gated + pre-existing-red on main); relying on CI for the rest.

Review

Cross-model review run (thorough): Codex + Harper-domain adjudication. The Gemini/agy leg failed to run (stochastic print-mode hang, 3 attempts) — no second-model perf/maintainability coverage from Gemini; correctness/concurrency were covered by Codex + the domain pass. Confirmed findings (listener-cert rotation gap, hot-swap last-good retention, route file-read isolation) are all addressed above.

🤖 Generated by an LLM (Claude Opus 4.8, 1M context). Reviewed by Kris before merge.

[kriszyp]

CI note for reviewers: the red Integration Test – Harper UDS proxy is pre-existing and environmental, not a regression from this PR — it fails identically on main (latest main run: the harper dev-dependency's own source can't resolve its ../core/* module paths in the integration harness — TS2307 across harper's replication/, security/ modules — before symphony's proxy is even exercised). This diff touches only symphony's router.rs/server.ts; nothing harper-related. All 5 platform builds and the unit suites (ubuntu-20/22, macos-14) are green; macos-13 was runner-backlogged at flip time (same suite, passing elsewhere).

Also unrelated: sni::tests::test_is_grease is a pre-existing red on main (the is_grease bitmask only matches 0x0A0A, not 0x1A1A/0xFAFA) — worth a separate issue, untouched here.

(comment generated by an LLM — Claude Opus 4.8)

[Devin-Holland]

Reviewed the full change — per-route cert isolation, carry-forward-last-good on hot-swap, and the server.ts cert-file watching. Approving: this is a solid, well-tested fix. Coverage is thorough (cert-file hot reload, per-route KeyMismatch isolation, listener defaultCert reload forcing a recreate, and the co-tenant ENOENT-mid-rotation case, plus Rust unit tests exercising the non-atomic key-then-chain write window).

Two minor, non-blocking notes:

1. Carry-forward retains the whole old route, not just the cert. On a hot-swap where a route's upstreams change and its cert transiently fails to build, the old upstreams also persist for that SNI until the next reconcile heals it. It self-heals (the cert watcher fires a reconcile), so this is fine in practice — worth a one-line comment noting it's intentional so a future reader doesn't mistake it for a bug.

2. Log volume on a persistently-bad cert. The eprintln! ("retaining last-good" / "skipping route") fires on every reconcile, so a cert that stays mismatched long-term (not just mid-rotation) will log on each cert-file event. Consider logging only on the good→bad transition to keep steady-state noise down.

[kriszyp]

Thanks for the thorough read, @Devin-Holland — much appreciated, and great callouts. Both notes are addressed in 21c090c:

1. Whole-route retention — added a comment right at the carry-forward making it explicit that the previous route is kept whole (cert and upstreams) by design, and that if the upstreams also changed in the same reconcile they persist until the next (healing) reconcile — so a future reader won't mistake it for a bug.

2. Steady-state log volume — RouteTable now tracks the set of SNIs that failed to build and carries it across the hot-swap, so the eprintln! fires only on the good→bad transition rather than on every rebuild while a cert stays broken. A route stuck mismatched long-term now logs once, not on each cert-file event. Added a unit assertion on the tracked failing set.

All green locally (33 JS tests + cargo test). Left the JS-side resolveRoute file-read log ("failed to resolve cert material") firing per reconcile for now, since a missing cert file is an actionable filesystem condition worth re-surfacing — happy to give it the same transition-dedup if you'd prefer symmetry.

(reply generated by an LLM — Claude Opus 4.8; Kris will confirm before merge)