Skip to content

feat(retire-falco): F5 — per-node agent-liveness contract + honest corroboration-coverage UX (JEF-308)#172

Merged
thejefflarson merged 1 commit into
mainfrom
thejefflarson/jef-308-retire-falco-f5-per-node-agent-liveness-contract-honest
Jul 5, 2026
Merged

feat(retire-falco): F5 — per-node agent-liveness contract + honest corroboration-coverage UX (JEF-308)#172
thejefflarson merged 1 commit into
mainfrom
thejefflarson/jef-308-retire-falco-f5-per-node-agent-liveness-contract-honest

Conversation

@thejefflarson

Copy link
Copy Markdown
Owner

Closes JEF-308

Why

"Corroboration blind on node X" is not honestly derivable today: RuntimeObservation carries no node, the agent DaemonSet doesn't expose its node, and readiness is fleet-aggregate. A per-node view would invent data — violating the honesty invariant. And a Ready agent whose eBPF probes failed to load is still BLIND (the exact Falco failure mode). This makes liveness signal-flow, not pod-Ready, which is the resilience story that justifies dropping the 2nd sensor.

Data contract (behavior/src + agent)

  • RuntimeObservation gains node; new AgentReport beacon {node, probes_loaded, probes_total, signals_emitted} POSTed each window to a new engine ingest /agent-liveness — behind the same bearer auth + per-peer rate limit + body cap as /behavior.
  • The agent gets its node via the downward API (K8S_NODE = spec.nodeName), stamps it onto every observation, and beacons every window even when quiet (quiet≠blind). probes_loaded=0 (no-eBPF build, or an attach failure) reports the node blind.

Engine liveness (state/agent_liveness.rs)

  • AgentLivenessStore (TTL'd + bounded against a distinct-node flood) and the expected-node set derived from the agent DaemonSet's own pods in the informer (JEF-131) — the scheduler already honoured the agent's nodeSelector/tolerations (JEF-295 arm64), so a node it isn't scheduled on is out-of-scope, not blind.
  • derive_runtime_coverage classifies each expected node healthy / degraded (partial probes) / blind (not-reporting | probes-failed) / out-of-scope, stamped into the findings handle each pass.

Honest UX

  • Readiness collapses the two falco + ebpf-agent rows into one agent-sourced "Runtime corroboration" row with an honesty ladder — healthy / degraded (blind on N, named) / blind (no live signal) / Falco-still-feeding (cutover) — plus a server-rendered per-node <table> inside <details> (no JS).
  • Quiet ≠ sensor-down: a node reporting 0 signals with probes loaded reads healthy-quiet.
  • Strip: the corroboration chip goes degraded the moment any expected node is blind.
  • A latent / uncorroborated finding on a blind node carries a caveat ("no live sensor on this node; absence of a signal is not evidence of safety") so it never renders as reassuringly calm.

Invariants held

  • Zero-egress: liveness is derived from the beacons the agent already POSTs + the in-cluster informer — the agent's OTLP/metrics endpoint is never consumed.
  • Untrusted node names are auto-escaped at render (maud default; no PreEscaped) — covered by a render test.
  • Presentation is a view, never a gate (ADR-0016) — this adds a derived liveness signal + display, no actuation.
  • falco row still feeds during cutover (both-sources rung) until F8 retires the adapter.

Tests

New coverage: expected-set-from-informer respecting nodeSelector; healthy / quiet≠blind / not-reporting-blind / probes-failed-blind / partial-degraded / out-of-scope / stale-prunes / store-bound; the collapsed readiness row + ladder; strip chip degrades on a blind node; latent-on-blind caveat; per-node table render + node-name escaping; the AgentReport/node wire contract; the agent beacon builder + reporter endpoint.

Gates (both workspaces)

cargo fmt · cargo test (engine 670 + agent 45, all green) · cargo clippy --all-targets (0 warnings) · cargo build --example dashboard_preview · helm template ... --set agent.enabled=true renders the K8S_NODE downward-API env. No file >1000 lines (extracted EngineMetrics, the readiness tests, and the eBPF observer tests to sibling files).

Scope notes / decisions

  • DECISION: the expected-node set is derived from the agent's own DaemonSet pods' spec.nodeName (the scheduler already applied nodeSelector/tolerations), rather than re-matching node labels against a plumbed-through selector. This is drift-free (one source of truth: the chart schedules the pods) and needs no Node watch / extra RBAC.
  • The eBPF observer's probe-count wiring compiles only under --features ebpf (not built in CI gates), so it's covered by review, not an automated test; the pure beacon-builder and the no-eBPF (blind) path are tested.

🤖 Generated with Claude Code

…rroboration-coverage UX (JEF-308)

Make "corroboration blind on node X" honestly derivable, so dropping the 2nd
sensor is defensible. Liveness is SIGNAL-FLOW, not pod-Ready: a Ready agent whose
eBPF probes failed to attach is still BLIND.

Data contract (behavior/src + agent):
- RuntimeObservation gains `node`; new AgentReport beacon {node, probes_loaded,
  probes_total, signals_emitted} POSTed each window to a new engine ingest
  `/agent-liveness` (behind the same bearer + rate-limit + body cap as /behavior).
- Agent gets its node via the downward API (K8S_NODE = spec.nodeName), stamps it
  onto every observation, and beacons EVERY window even when quiet (quiet≠blind);
  probes_loaded=0 (no-eBPF build or attach failure) reports itself blind.

Engine liveness (state/agent_liveness.rs):
- AgentLivenessStore (TTL'd, bounded) + expected-node set derived from the agent
  DaemonSet's own pods in the informer (the scheduler already honoured the agent's
  nodeSelector/tolerations, so a node it isn't scheduled on is out-of-scope, not
  blind). derive_runtime_coverage classifies healthy / degraded / blind
  (not-reporting | probes-failed) / out-of-scope; stamped each pass.

Honest UX:
- Readiness collapses the two `falco`+`ebpf-agent` rows into ONE agent-sourced
  "Runtime corroboration" row with a ladder — healthy / degraded (blind on N,
  named) / blind (no live signal) / Falco-cutover — plus a server-rendered
  per-node <table> inside <details> (no JS). Quiet≠blind; missing data never
  reassuring.
- Strip: the corroboration chip goes degraded the moment any expected node is blind.
- A latent/uncorroborated finding on a blind node carries a caveat ("no live
  sensor on this node; absence of a signal is not evidence of safety").

Zero-egress preserved (derived from POSTed beacons + informer, never OTLP). Node
names auto-escaped at render. Extracted EngineMetrics, readiness tests, and the
eBPF observer tests to sibling files to stay under the 1,000-line cap.

Closes JEF-308

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP
@thejefflarson
thejefflarson force-pushed the thejefflarson/jef-308-retire-falco-f5-per-node-agent-liveness-contract-honest branch from 0c41af9 to 0ff81f8 Compare July 5, 2026 03:22
@thejefflarson
thejefflarson merged commit 11b0c7a into main Jul 5, 2026
7 checks passed
@thejefflarson
thejefflarson deleted the thejefflarson/jef-308-retire-falco-f5-per-node-agent-liveness-contract-honest branch July 5, 2026 04:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant