spec: Isolated v1 hardening — mandatory egress routing, credentials, audit events & metrics

[imneov]

Companion spec to #2, distilled from the isolated-runtime wayfinder map (2026-07-16, .scratch/isolated-runtime-spec/) after all five decision tickets were resolved and the T03/T04 resolutions passed fresh-context adversarial review. Spec #2 defined the Isolated mode core and deferred "egress policy (part B)" to its own spec — this is that spec, deliberately scoped down to v1: mandatory egress routing (not policy), the credential model, the audit-event and metrics catalogs, and the PR-ordering correction. Domain terms follow CONTEXT.md; ADR-0001/0002/0003 apply.

Problem Statement

The Isolated tier's whole reason to exist is hijack-level isolation (CONTEXT.md): the platform assumes the agent inside a 小龙虾 will be prompt-injected and must contain the blast radius to that one instance. Spec #2 gives the instance its own Kubernetes isolation unit, but four load-bearing gaps remain, and until they close the tier's promise is hollow:

1. Egress is injected but not enforced. Today HTTP_PROXY env is always injected, yet the "proxy" is the ClawManager server itself with no enforcement semantics; an instance env override can silently clobber the proxy variables; and nothing fails closed when the proxy is unreachable. A hijacked Isolated instance can talk to anything.
2. No decided credential story for a sandboxed instance. Operators cannot say what a compromised Isolated instance holds, and agent-sandbox itself offers no identity surface — it is entirely ClawManager's job.
3. No audit trail and no metrics. The backend has no structured logging and no Prometheus surface at all today. A tier built for "we expect this thing to be hijacked" is not operable without an audit record of lifecycle and fail-closed refusals, and without basic per-pod runtime metrics.
4. The RuntimeBackend refactor stalled halfway. Only Lite is behind the seam; Pro still runs through legacy fallback branches. Every uniform behavior this spec adds (egress, audit) would otherwise have to be threaded through both the backend layer and the legacy code paths, twice.

Solution

Ship the smallest set of decisions that make the Isolated tier's contract real, in the existing PR sequence:

• Complete PR1 first: Pro lifecycle is extracted behind RuntimeBackend (covering instance_mode=pro for BOTH desktop and shell runtime types, so the legacy fallback branch dies), before any Isolated code lands. All three modes then sit behind one seam.
• Mandatory egress routing ships with sandboxBackend (PR2): Isolated instances get HTTP(S)_PROXY injected unconditionally — no operator toggle, because mandatory proxy IS the tier contract (ADR-0003). The proxy env keys are reserved for Isolated: user env overrides cannot clobber them. Creation and start fail closed with a stable error when the proxy is unreachable; the runtime image fails closed mid-run per ADR-0003. Egress policy (allow-lists, authenticated proxy, filtering UX) stays deferred.
• Credentials reuse the existing model: the per-instance access token and agent bootstrap token, injected via the Sandbox pod template env. No new identity system.
• A thin, net-new audit pipeline: a server-side JSON-lines-to-stdout audit logger with a platform-global switch (default ON), emitting a small mode-agnostic catalog of lifecycle, refusal, credential, agent-command, and skill events, each attributed with instance_mode.
• Pod-side metrics as a documented contract: a scrapeable GET /metrics on the runtime agent's existing port, with a cardinality-safe catalog; spec'd in the agent-runtime contract now, delivered when the runtime-agent track owner is confirmed.

User Stories

1. As a security engineer, I want every Isolated instance's outbound HTTP/HTTPS traffic forced through the platform egress proxy, so that a hijacked agent cannot exfiltrate directly to arbitrary endpoints.
2. As a security engineer, I want the proxy environment variables to be reserved keys that instance env overrides cannot set for Isolated instances, so that per-instance configuration cannot silently disable the tier's egress contract.
3. As a security engineer, I want Isolated instance creation and start to fail closed with a stable, identifiable error when the egress proxy is unreachable, so that no Isolated instance ever runs without its egress path.
4. As a security engineer, I want a compromised Isolated instance to hold only its own per-instance credentials, so that the blast radius of a hijack stays within that instance's scope.
5. As a security engineer, I want fail-closed refusals recorded as audit events, so that enforcement decisions are visible after the fact, not just at the moment they happen.
6. As a platform administrator, I want Isolated mode reported unavailable with an explicit reason when no egress proxy URL is computable from deploy-time configuration, so that the tier never silently ships without its defining control.
7. As a platform administrator, I want no new configuration surface for egress enforcement, so that enabling the Isolated tier does not add another toggle to misconfigure — deploy-time proxy config is the single source of truth.
8. As a platform administrator, I want a platform-global audit-log switch that defaults to on, so that audit exists out of the box but can be silenced in environments that collect nothing.
9. As a platform administrator, I want audit events emitted as structured JSON lines on stdout, so that my existing log collector (Loki, ELK, CloudWatch, …) ingests them without a new integration.
10. As a platform administrator, I want audit events for all instance modes with an instance_mode attribute, so that I can filter for the Isolated tier without the platform maintaining two audit code paths.
11. As an SRE, I want sandbox lifecycle transitions (ready, finished with reason, recreated) recorded as audit events, so that I can reconstruct what happened to an instance during an incident.
12. As an SRE, I want an Isolated instance whose Pod died of a transient failure to be recreated automatically by the platform, so that upstream agent-sandbox's lack of auto-respawn does not become user-visible downtime.
13. As an SRE, I want each runtime pod to expose a scrapeable Prometheus /metrics endpoint with gateway-up, command, duration, skill, and resource gauges, so that fleet dashboards and alerts work with standard tooling.
14. As an SRE, I want metric identity to come from pod labels rather than per-series labels, so that metric cardinality stays bounded as instances churn.
15. As an agent operator, I want credential.minted, agent.registered, and agent.command.started/completed events in the audit stream, so that I can trace an instance from provisioning to command execution.
16. As an agent operator, I want skill install/uninstall requests audited, so that capability changes to an agent are attributable.
17. As a ClawManager maintainer, I want Pro's lifecycle (desktop and shell) genuinely moved behind RuntimeBackend before Isolated lands, so that egress and audit behavior thread through one seam instead of a seam plus legacy branches.
18. As a ClawManager maintainer, I want the RuntimeBackend interface left un-expanded by the Pro extraction (restart orchestration, proxy endpoint fallback, and background sync stay where they are), so that PR1-completion remains a reviewable, behavior-preserving move.
19. As a ClawManager maintainer, I want the audit emitter to be net-new and thin (no new DB writes, no replacement of the existing AI Gateway audit inserts), so that v1 observability does not turn into a logging-platform project.
20. As a ClawManager maintainer, I want egress.* audit event names reserved but not emitted in v1, so that the catalog anticipates proxy hardening without pretending the trust-all proxy can attribute traffic today.
21. As an upstream maintainer, I want the Sandbox pod template to carry only pass-through Kubernetes fields and env, so that the Isolated tier works on any conformant cluster without CNI-specific egress machinery (ADR-0003 rejected NetworkPolicy as primary mechanism for portability).
22. As an upstream maintainer, I want the agent-sandbox dependency pinned to a known version with an explicit upgrade note, so that CRD/API drift is a managed event rather than a surprise.

Implementation Decisions

PR ordering (supersedes the "thread through both paths" constraint)

• PR1 is completed before PR2: Pro is extracted into a backend implementing RuntimeBackend, covering instance_mode=pro for BOTH desktop and shell runtime types, so the legacy fallback branch is deleted (or reduced to an explicit error). Mode→backend dispatch stays clean; runtime-type branching lives inside the pro backend, mirroring the legacy structure.
• The extraction is behavior-preserving and mechanical (~600–800 moved lines, single PR). It does NOT expand the RuntimeBackend interface — restart orchestration, the proxy-service endpoint fallback, and the background sync loop stay where they are — and does NOT bundle any signature tightening.
• After PR1 completion, all three modes sit behind the RuntimeBackend seam; everything below assumes that.

Egress enforcement (T03; ADR-0003)

• v1 egress = HTTP(S)_PROXY env injection in the Sandbox pod template, and nothing else. The direct Sandbox CR has no NetworkPolicy field (that lives on SandboxTemplate, an extensions path this spec defers). Per-Sandbox NetworkPolicy is v1.x defense-in-depth where the CNI supports it — ADR-0003 explicitly rejected NetworkPolicy as the primary mechanism.
• No new config surface: the proxy URL comes from the existing deploy-time egress proxy configuration (env var / service derivation). No operator toggle — Isolated enforces unconditionally; mandatory proxy is the tier contract. If no proxy URL is computable, the Isolated tier is unavailable with an explicit error (mirrors the missing-CRDs pattern).
• Injection happens in the isolated backend's pod-template env build via the existing shared env-builders. No admission webhook.
• Proxy env vars are reserved keys for Isolated: env overrides containing them are rejected at Create/Update with a validation error, and the isolated env build re-applies proxy env after the override merge (today overrides merge after proxy injection and could clobber it — that ordering bug must not survive into the isolated path).
• Fail-closed, three layers: (1) server-side precheck at Create AND Start — URL parse + TCP dial with 2s timeout, one retry; refusal uses the stable error code egress_proxy_unreachable; (2) image-level fail-closed per ADR-0003 covers mid-run proxy outages — no active server-side reaction in v1; (3) optionally a startupProbe in the pod template as the Kubernetes-visible expression of the same check.
• Egress policy — FQDN allow-lists, authenticated proxy, content filtering, blocked-traffic UX — stays deferred (RFC §5). Named v1.x follow-on: harden the trust-all proxy (per-instance auth, allow-list, request logging); it is the prerequisite for egress.* audit events.

Credential model (T01/T02)

• agent-sandbox provides no identity/token surface; credentials are entirely ClawManager's. Isolated reuses the existing per-instance access token + agent bootstrap token model, injected via the Sandbox pod template env — same env-builders as other gateway instances.
• Known, accepted v1 gap: the cluster-wide shared runtime-agent report token also lands in every runtime pod env; under the hijack threat model that is scope-bleed. Not a v1 blocker (the tier is still a strict improvement over lite/pro); promoted to a v1.x follow-on ticket when the core ships.

Sandbox lifecycle observation (T01)

• Lifecycle is observed directly from Sandbox.Status.Conditions (Ready | Suspended | Finished{reason}); no parallel state machine in ClawManager.
• agent-sandbox does NOT auto-recreate on Finished(PodFailed) (upstream issue #729): the isolated backend observes it and issues Delete+recreate for transient failures, emitting a sandbox.recreated audit event.
• Version pin: agent-sandbox v0.5.1 (v1beta1); upgrade to v0.5.2+ when released per upstream migration guidance.

Audit pipeline (T04)

• Net-new server-side audit logger emitting one JSON object per line to stdout. No new DB writes in v1. The existing AI Gateway gateway.request.* DB inserts stay exactly as they are, explicitly OUTSIDE the new switch and pipeline until a v1.x unification.
• Platform-global switch: CLAWMANAGER_AUDIT_LOG_ENABLED, deploy-time env var, default ON. (No generic system-settings store exists; deploy-time env is the consistent v1 surface, matching the egress config decision.)
• Emission is mode-agnostic — events fire for lite/isolated/pro alike — with instance_mode attribution on every event. instance_id is nullable in the line contract (null on instance.create.refused).
• v1 event catalog:
  • instance lifecycle: instance.create/start/stop/delete outcomes including .refused variants carrying the stable refusal code (e.g. egress_proxy_unreachable, capability-gate refusals)
  • sandbox substrate: sandbox.ready, sandbox.finished (with condition reason), sandbox.recreated
  • credentials: credential.minted
  • agent plane: agent.registered, agent.command.started, agent.command.completed
  • skills: skill.install.requested, skill.uninstall.requested
  • reserved, NOT emitted in v1: egress.* (blocked on proxy hardening)

Metrics (T04)

• Pod-side only. The runtime agent adds GET /metrics (Prometheus text format) on its existing agent port. No server-side Prometheus endpoint, no ServiceMonitor in v1.
• Catalog: agent build-info, per-gateway up gauge, command counters and duration histograms, skill install counters, and cpu/mem/disk usage gauges.
• Identity comes from pod labels (scrape-config relabeling), not per-series labels — cardinality-safe as instances churn.
• Scrape discovery via prometheus.io/* annotations on the Sandbox pod template.
• Delivery status: this is a spec'd agent-runtime-contract addition. The runtime-agent implementation owner is not yet confirmed, so v1 commits to the documented contract + pod-template annotations; the endpoint implementation ships on the runtime-agent track once its owner confirms.

Testing Decisions

• Tests assert external behavior at the highest existing seam. The RuntimeBackend seam (from spec spec: Isolated instance mode on agent-sandbox (RuntimeBackend refactor) #2) is where mode behavior is tested; this spec adds exactly ONE new seam: the audit logger interface (so emission can be asserted via an injected sink and silenced in other tests). Metrics add no code seam in this repo in v1 (contract doc only).
• Pro extraction is locked by the existing k8s-layer and service-layer suites passing unchanged — same rule as the Lite move: no new tests for moved-but-unchanged logic, no test edits beyond mechanical constructor/import updates.
• Egress behavior is tested at the backend seam against a fake dynamic client (prior art: existing fake-clientset k8s service tests): created Sandbox spec contains proxy env after an override merge that attempted to clobber it; Create/Update rejects overrides containing reserved keys; Create/Start refuse with egress_proxy_unreachable when the precheck dials a dead endpoint; Isolated reported unavailable when no proxy URL is computable.
• Audit emission is tested through the logger seam: lifecycle operations on a fake backend produce the expected event sequence as parsed JSON lines (event type, instance_mode attribution, nullable instance_id on create-refusal); the switch OFF produces silence; no assertion on log formatting internals.
• Sandbox recreate-on-PodFailed is tested at the backend seam by feeding condition transitions through the fake client and asserting Delete+recreate plus the sandbox.recreated event.
• e2e stays as spec'd in spec: Isolated instance mode on agent-sandbox (RuntimeBackend refactor) #2 (Playwright, agent-sandbox installed as environment requirement); this spec adds an API-level fail-closed check (create with unreachable proxy → stable error), not a new browser flow.

Out of Scope

• Egress policy: FQDN allow-lists, authenticated proxy, content filtering, blocked-traffic UX (RFC §5).
• Proxy hardening (per-instance auth, allow-list, request logging) and therefore egress.* audit events — named v1.x follow-on.
• Unifying the existing gateway.request.* DB audit inserts into the stdout pipeline — v1.x.
• Server-side Prometheus endpoint, ServiceMonitor objects, or any dashboard/alerting content.
• Fixing the shared runtime-agent report-token scope-bleed — v1.x follow-on ticket at core-ship time.
• Per-Sandbox NetworkPolicy defense-in-depth — v1.x, CNI-dependent.
• Suspend/Resume as a product feature; warm pools; cross-tier migration; Pro migration to agent-sandbox (all per spec: Isolated instance mode on agent-sandbox (RuntimeBackend refactor) #2 / ADR-0001 / RFC §5).
• MCP/Skills management surface for the Isolated tier (whitelist, install policy, scanner integration) — deliberately deferred to its own follow-on map after core ships.
• Cost model / per-tier pricing.

Further Notes

• Decision provenance: .scratch/isolated-runtime-spec/map.md (wayfinder), with recon reports for agent-sandbox v0.5.1 capabilities, the current backend state (env injection, dead AttachPolicy payload, half-done refactor), audit/metrics infrastructure (none exists today), and the Pro extraction inventory. The T03/T04 resolutions each passed a fresh-context adversarial review; the reserved-keys requirement was a review blocker folded into the decision.
• ADR-0003 framing note: NetworkPolicy was rejected there as the primary egress mechanism for CNI portability. Per-Sandbox NetworkPolicy later is defense-in-depth, not "the real enforcement waiting to happen".
• Relationship to existing tickets: refactor: migrate Pro behind RuntimeBackend and unify dispatch #6 (Pro extraction) gains the T05 scope rulings; feat: sandboxBackend — Isolated instance lifecycle on agent-sandbox #8 (sandboxBackend) gains the egress + recreate scope; audit and metrics get their own tickets. Ticket updates follow this spec.