feat(retire-falco): F5 — per-node agent-liveness contract + honest corroboration-coverage UX (JEF-308)#172
Merged
thejefflarson merged 1 commit intoJul 5, 2026
Conversation
…rroboration-coverage UX (JEF-308)
Make "corroboration blind on node X" honestly derivable, so dropping the 2nd
sensor is defensible. Liveness is SIGNAL-FLOW, not pod-Ready: a Ready agent whose
eBPF probes failed to attach is still BLIND.
Data contract (behavior/src + agent):
- RuntimeObservation gains `node`; new AgentReport beacon {node, probes_loaded,
probes_total, signals_emitted} POSTed each window to a new engine ingest
`/agent-liveness` (behind the same bearer + rate-limit + body cap as /behavior).
- Agent gets its node via the downward API (K8S_NODE = spec.nodeName), stamps it
onto every observation, and beacons EVERY window even when quiet (quiet≠blind);
probes_loaded=0 (no-eBPF build or attach failure) reports itself blind.
Engine liveness (state/agent_liveness.rs):
- AgentLivenessStore (TTL'd, bounded) + expected-node set derived from the agent
DaemonSet's own pods in the informer (the scheduler already honoured the agent's
nodeSelector/tolerations, so a node it isn't scheduled on is out-of-scope, not
blind). derive_runtime_coverage classifies healthy / degraded / blind
(not-reporting | probes-failed) / out-of-scope; stamped each pass.
Honest UX:
- Readiness collapses the two `falco`+`ebpf-agent` rows into ONE agent-sourced
"Runtime corroboration" row with a ladder — healthy / degraded (blind on N,
named) / blind (no live signal) / Falco-cutover — plus a server-rendered
per-node <table> inside <details> (no JS). Quiet≠blind; missing data never
reassuring.
- Strip: the corroboration chip goes degraded the moment any expected node is blind.
- A latent/uncorroborated finding on a blind node carries a caveat ("no live
sensor on this node; absence of a signal is not evidence of safety").
Zero-egress preserved (derived from POSTed beacons + informer, never OTLP). Node
names auto-escaped at render. Extracted EngineMetrics, readiness tests, and the
eBPF observer tests to sibling files to stay under the 1,000-line cap.
Closes JEF-308
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP
thejefflarson
force-pushed
the
thejefflarson/jef-308-retire-falco-f5-per-node-agent-liveness-contract-honest
branch
from
July 5, 2026 03:22
0c41af9 to
0ff81f8
Compare
thejefflarson
deleted the
thejefflarson/jef-308-retire-falco-f5-per-node-agent-liveness-contract-honest
branch
July 5, 2026 04:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes JEF-308
Why
"Corroboration blind on node X" is not honestly derivable today:
RuntimeObservationcarries no node, the agent DaemonSet doesn't expose its node, and readiness is fleet-aggregate. A per-node view would invent data — violating the honesty invariant. And a Ready agent whose eBPF probes failed to load is still BLIND (the exact Falco failure mode). This makes liveness signal-flow, not pod-Ready, which is the resilience story that justifies dropping the 2nd sensor.Data contract (behavior/src + agent)
RuntimeObservationgainsnode; newAgentReportbeacon{node, probes_loaded, probes_total, signals_emitted}POSTed each window to a new engine ingest/agent-liveness— behind the same bearer auth + per-peer rate limit + body cap as/behavior.K8S_NODE = spec.nodeName), stamps it onto every observation, and beacons every window even when quiet (quiet≠blind).probes_loaded=0(no-eBPF build, or an attach failure) reports the node blind.Engine liveness (
state/agent_liveness.rs)AgentLivenessStore(TTL'd + bounded against a distinct-node flood) and the expected-node set derived from the agent DaemonSet's own pods in the informer (JEF-131) — the scheduler already honoured the agent's nodeSelector/tolerations (JEF-295 arm64), so a node it isn't scheduled on is out-of-scope, not blind.derive_runtime_coverageclassifies each expected node healthy / degraded (partial probes) / blind (not-reporting | probes-failed) / out-of-scope, stamped into the findings handle each pass.Honest UX
falco+ebpf-agentrows into one agent-sourced "Runtime corroboration" row with an honesty ladder — healthy / degraded (blind on N, named) / blind (no live signal) / Falco-still-feeding (cutover) — plus a server-rendered per-node<table>inside<details>(no JS).Invariants held
PreEscaped) — covered by a render test.falcorow still feeds during cutover (both-sources rung) until F8 retires the adapter.Tests
New coverage: expected-set-from-informer respecting nodeSelector; healthy / quiet≠blind / not-reporting-blind / probes-failed-blind / partial-degraded / out-of-scope / stale-prunes / store-bound; the collapsed readiness row + ladder; strip chip degrades on a blind node; latent-on-blind caveat; per-node table render + node-name escaping; the
AgentReport/nodewire contract; the agent beacon builder + reporter endpoint.Gates (both workspaces)
cargo fmt·cargo test(engine 670 + agent 45, all green) ·cargo clippy --all-targets(0 warnings) ·cargo build --example dashboard_preview·helm template ... --set agent.enabled=truerenders theK8S_NODEdownward-API env. No file >1000 lines (extractedEngineMetrics, the readiness tests, and the eBPF observer tests to sibling files).Scope notes / decisions
spec.nodeName(the scheduler already applied nodeSelector/tolerations), rather than re-matching node labels against a plumbed-through selector. This is drift-free (one source of truth: the chart schedules the pods) and needs no Node watch / extra RBAC.--features ebpf(not built in CI gates), so it's covered by review, not an automated test; the pure beacon-builder and the no-eBPF (blind) path are tested.🤖 Generated with Claude Code