spec: Isolated instance mode on agent-sandbox (RuntimeBackend refactor)

[imneov]

Distilled from #1 (part A only) through a design interview on 2026-07-15. Egress policy (part B of #1) is explicitly out of scope and will get its own spec. Domain terms follow CONTEXT.md; architectural decisions are recorded in docs/adr/0001 and docs/adr/0002.

Problem Statement

Users running headless Hermes/OpenClaw agents (CI-style coding agents, browserless automation, tool-execution sandboxes, per-project isolated workspaces) have to choose between two bad fits: Lite mode is cheap but co-locates many agents in one shared runtime Pod, so Kubernetes-level isolation between agents is weak; Pro mode gives every agent its own Pod but drags in a desktop streaming stack (Webtop/KasmVNC/Selkies) the workload never uses, wasting CPU/memory and money.

Solution

A third Instance Mode, Isolated (displayed as "Isolated Gateway" / 独立网关): every instance gets its own Kubernetes isolation unit running only the Hermes/OpenClaw gateway process — no desktop stack — with a persistent workspace, per-instance resource limits, and access exclusively through the ClawManager gateway. The isolation unit is a kubernetes-sigs/agent-sandbox Sandbox; agent-sandbox is a required cluster dependency for this mode (ADR-0001). Internally, all three modes are served through a single RuntimeBackend interface with three full implementations (lite, pro desktop, sandbox).

User Stories

1. As an agent operator, I want to create an OpenClaw instance in Isolated mode, so that my agent gets Kubernetes-level isolation without paying for a desktop stack.
2. As an agent operator, I want to create a Hermes instance in Isolated mode, so that my headless agent runs in its own sandbox.
3. As an agent operator, I want to pick the mode from a three-way selector (Lite / Isolated Gateway / Pro), so that I can trade off isolation and cost per instance.
4. As an agent operator, I want my Isolated instance's workspace to survive stop/start, so that long-running agent state is not lost.
5. As an agent operator, I want to start, stop, and delete an Isolated instance with the same controls as Lite/Pro, so that I don't need to learn a new lifecycle.
6. As an agent operator, I want to reach my Isolated instance's gateway through the ClawManager access URL and token, so that access works the same way as for my other instances.
7. As an agent operator, I want the instance detail view to show that no desktop is available for an Isolated instance, so that I am not offered a desktop button that cannot work.
8. As an agent operator, I want my instance's model configuration injected by ClawManager, so that the agent uses governed model access without manual setup.
9. As an agent operator, I want per-instance CPU/memory/disk limits enforced on my Isolated instance, so that one agent cannot starve others.
10. As an agent operator, I want a clear error when I request Isolated mode on a cluster that cannot provide it, so that I know to ask my administrator instead of debugging a silent failure.
11. As a platform administrator, I want Isolated mode to be advertised as unavailable (with the reason) when the agent-sandbox CRDs are not installed, so that users cannot create instances that will never schedule.
12. As a platform administrator, I want each Isolated instance to be its own Pod/Sandbox in the cluster, so that Kubernetes-native tooling (quotas, NetworkPolicy, monitoring) applies per agent.
13. As a platform administrator, I want Isolated instances to stay private by default (no public Service/Ingress per instance), so that the only way in is the audited ClawManager gateway.
14. As a platform administrator, I want instance records to show which runtime backend and which Sandbox object serve an instance, so that I can correlate product state with cluster state during incidents.
15. As a platform administrator, I want mode-specific resource limit ceilings for Isolated mode, so that I can cap what a single sandboxed agent may request.
16. As a platform administrator, I want existing Lite and Pro instances to behave exactly as before this feature ships, so that the rollout is risk-free for current users.
17. As a cluster operator, I want ClawManager to detect the agent-sandbox CRDs at startup and expose that capability status, so that I can verify my installation without creating a test instance.
18. As a cluster operator (enterprise integrator), I want the Isolated backend to pass through generic placement fields (node selector, runtime class, tolerations), so that an external NodeGroup/scope system can steer instances to the right nodes without ClawManager knowing about groups (ADR-0002).
19. As a security engineer, I want the runtime backend interface to expose a policy attachment point, so that per-instance egress policy (part B) can later be bound without another core refactor.
20. As a ClawManager maintainer, I want Lite and Pro lifecycle logic moved fully behind the same RuntimeBackend interface as Isolated, so that instance dispatch has one seam instead of mode-specific branches.
21. As a ClawManager maintainer, I want the mode/runtime-type derivation functions removed in favor of explicit mode, so that the type system stops implying that gateway instances are always Lite.
22. As an upstream maintainer (Yuan-lab-LLM/ClawManager), I want this feature proposed as an RFC and delivered as reviewable, stage-sliced PRs, so that I can evaluate the agent-sandbox dependency and the refactor separately.

Implementation Decisions

• Domain model: instance_mode becomes a three-value enum lite | isolated | pro (isolation/cost tier). runtime_type (desktop | shell | gateway) stays the workload-shape axis. The two axes are orthogonal; the existing bidirectional derivation helpers (InstanceModeForRuntimeType / RuntimeTypeForInstanceMode) are removed and every call site passes mode explicitly. Isolated currently only combines with runtime_type=gateway.
• Naming: enum value isolated; display name "Isolated Gateway" (独立网关). The word "sandbox" refers exclusively to the agent-sandbox CRD object, never to the product mode.
• Substrate (ADR-0001): Isolated mode runs exclusively on kubernetes-sigs/agent-sandbox CRDs via a single sandboxBackend. No native Pod fallback. The control plane probes for the Sandbox CRD; when absent, Isolated mode is reported unavailable with an explicit reason and creation requests fail loudly.
• Backend architecture: introduce a RuntimeBackend interface (Create/Start/Stop/Delete/GetStatus/GetEndpoint/AttachPolicy, with Suspend reserved) and do the full refactor: Lite and Pro lifecycle logic genuinely moves into liteBackend and proDesktopBackend; instance dispatch goes through the interface for all three modes. AttachPolicy is defined but unimplemented (seam for part B). Suspend is not exposed as a product state; the product keeps the running/stopped model, and "stopped" maps to whichever Sandbox lifecycle action frees compute while preserving the workspace volume (settled during the spike).
• Placement passthrough (ADR-0002): the backend spec carries nodeSelector (matchLabels + matchExpressions, structurally aligned with edge-apiserver's NodeSelector), runtimeClassName, and tolerations. No nodegroup concept enters ClawManager.
• Workspace: per-instance persistence via the Sandbox volume claim template, following the existing per-instance workspace path conventions.
• Access: reuse the existing per-instance JWT access token service and gateway routing; Sandbox Services stay cluster-private; no per-instance public exposure.
• Model/config injection: reuse the existing gateway env/model catalog injection used by gateway runtime types.
• Upstream strategy: an RFC distilling this spec is opened on Yuan-lab-LLM/ClawManager before PRs; the mandatory agent-sandbox dependency is presented as a core proposal point (if upstream rejects it, the decision reopens — no silent compromise). Exploratory development proceeds on this fork in parallel. PR slicing: (1) pure RuntimeBackend refactor with zero behavior change, (2) isolated mode + sandboxBackend, (3) UI/API surface.
• Spike deliverable: the agent-sandbox spike produces a substrate gap list; every gap is labeled either "upstream PR to agent-sandbox" or "solved in ClawManager + justification".

Testing Decisions

• Tests assert external behavior at the highest existing seam, not implementation details. One new seam is introduced: the RuntimeBackend interface.
• Refactor safety: the existing k8s-layer and service-layer test suites (runtime deployment, instance deployment, instance service) must pass unchanged — they are the behavior lock for the Lite/Pro migration. No new tests are written for moved-but-unchanged logic.
• sandboxBackend is tested at the backend seam against a fake dynamic client: asserting the created Sandbox object (spec contents, volume claim template, placement passthrough, absence of desktop processes/env), lifecycle action mapping, and status read-back mapping. Prior art: fake.NewSimpleClientset() usage in the k8s service tests.
• Instance service seam: mode semantics tests — three-value enum validation, isolated requests dispatching to sandboxBackend, capability gating error when the CRD is absent, explicit-mode behavior after removing the derivation helpers.
• e2e: a new Playwright spec modeled on the existing lite/pro modes spec: create an Isolated instance → no desktop offered → gateway reachable through ClawManager access URL → workspace survives stop/start. The e2e cluster installs agent-sandbox as an environment requirement (per ADR-0001), not a skip condition.

Out of Scope

• Enterprise egress policy (part B of feat: add isolated gateway runtime mode and enterprise egress policy #1): policy schema, NetworkPolicy/Cilium enforcement, audit of blocked events. Only the AttachPolicy interface placeholder ships now.
• Nodegroup as a product concept (tenant admission, capacity profiles, runtime-class declarations) — enterprise layer concern (ADR-0002).
• SandboxWarmPool / warm capacity (issue feat: add isolated gateway runtime mode and enterprise egress policy #1 Phase 5) — revisit with real startup-latency data; roadmap mention in the RFC only.
• Migrating Pro to agent-sandbox; Kasm-influenced Pro UX changes. Pro and Lite product behavior is unchanged.
• A user-facing "suspended" instance state.

Further Notes

• Depends on the shared understanding recorded in CONTEXT.md (Runtime Type, Instance Mode, Lite/Isolated/Pro, Sandbox) and docs/adr/0001, docs/adr/0002 — added in this repo alongside this spec.
• Upstream Yuan-lab-LLM/ClawManager is active and this fork is currently unforked in content (0 ahead); keeping the refactor PR behavior-neutral is what makes the staged upstream landing realistic.