feat: add isolated gateway runtime mode and enterprise egress policy

[imneov]

Summary

ClawManager currently has two runtime shapes:

• Lite Runtime: multiple agent/gateway subprocesses share one runtime Pod. It is cheap and fast, but the isolation boundary is weak because Kubernetes only sees one Pod.
• Pro Runtime: every agent gets an independent desktop Pod with Webtop/KasmVNC/Selkies-style browser desktop access. It has stronger isolation, but higher CPU/memory cost and unnecessary UI overhead for headless agents.

We need a third runtime shape between them:

Isolated Gateway Runtime: every agent gets an independent Kubernetes sandbox/Pod, runs Hermes/OpenClaw gateway only, has no desktop UI, and is accessed through ClawManager's unified gateway and policy layer.

After evaluating kubernetes-sigs/agent-sandbox, the recommendation is not to build this Kubernetes primitive from scratch. ClawManager should evaluate adopting agent-sandbox as the substrate for the new headless isolated runtime, while ClawManager remains the product control plane for tenant, nodegroup, gateway, model, audit, and enterprise policy.

Current Runtime Model

Lite
  shared runtime Pod
  high density, low cost
  weak per-agent isolation
  no desktop

Isolated Gateway    <-- new
  independent sandbox/Pod per agent
  no desktop
  medium cost
  strong Kubernetes-level isolation

Pro
  independent Pod per agent
  desktop streaming enabled
  highest cost
  strongest interactive experience

Why This Mode Is Needed

Enterprise users need stronger isolation than Lite but do not always need a full remote desktop.

Common cases:

• headless Hermes/OpenClaw agents
• CI-style coding agents
• browserless automation agents
• tool execution sandboxes
• per-tenant or per-project isolated workspaces
• agents with constrained internet or intranet access
• warm pooled agents for lower startup latency

Using Pro for these cases wastes resources because the desktop stack is not part of the actual workload.

Evaluation: Kasm / Desktop Runtime

Kasm is more mature for the desktop workspace layer. It is a good reference point for Pro Runtime because it already focuses on browser-accessible workspaces, remote desktop/session management, and KasmVNC-based streaming.

Relevant references:

• Kasm Workspaces
• KasmVNC

Design implication:

• Keep the Pro Runtime path aligned with mature desktop workspace patterns.
• Treat desktop streaming as a Pro capability, not as the default runtime for every agent.
• Do not force headless agents to pay the cost of Webtop/KasmVNC/Selkies.

Evaluation: kubernetes-sigs/agent-sandbox

agent-sandbox is a Kubernetes SIG Apps project for isolated, stateful, singleton workloads, explicitly targeting AI agent runtimes.

Observed project metadata at time of evaluation:

• Repo: kubernetes-sigs/agent-sandbox
• License: Apache-2.0
• Latest release observed: v0.5.1, published 2026-07-09
• Repo description: isolated, stateful, singleton workloads for AI agent runtimes

Relevant references:

• agent-sandbox repository
• agent-sandbox documentation
• OpenClaw sandbox example
• Hermes agent example
• Cilium egress demo
• Network policy composition example

What agent-sandbox Already Provides

agent-sandbox already covers much of the infrastructure layer needed by Isolated Gateway Runtime:

• Sandbox CRD for a single stateful Pod with stable identity.
• Persistent storage through volume claim templates.
• Lifecycle management including running/suspend-style flows.
• Optional Service creation for a Sandbox.
• SandboxTemplate for reusable runtime definitions.
• SandboxClaim for claiming prepared sandboxes.
• SandboxWarmPool for pre-warmed capacity.
• Optional stronger isolation via runtimes such as gVisor or Kata.
• Examples for OpenClaw and Hermes-style agent runtimes.
• Examples showing per-identity egress allowlisting with Cilium FQDN policies.

What agent-sandbox Does Not Replace

agent-sandbox is a Kubernetes primitive/controller, not a full ClawManager replacement.

ClawManager still needs to provide:

• tenant and user model
• nodegroup placement policy
• instance lifecycle API and UI
• unified access gateway
• per-instance access tokens and auth
• model gateway integration
• secret injection policy
• egress policy product model
• audit logs and blocked-event reporting
• runtime capacity views
• compatibility with existing Lite and Pro runtimes

Gateway Assessment

agent-sandbox has partial gateway/routing building blocks, but it does not provide the full enterprise gateway required here.

Already present upstream:

• Sandbox-level Service support.
• Example router/gateway patterns.
• OpenClaw/Hermes containers expose their own gateway/API ports.
• Optional gateway routing examples around sandbox-router.

Still required in ClawManager:

• unified ClawManager-managed instance URL
• per-agent bearer/session token
• tenant-aware authorization
• nodegroup-aware routing
• egress policy binding
• model gateway binding
• audit trail
• private-by-default external exposure model

Conclusion: agent-sandbox can provide the runtime substrate, but ClawManager still needs the product gateway and policy layer.

Proposed Architecture

ClawManager API/UI
  |
  |-- Tenant / Nodegroup / Runtime Mode / Policy model
  |
  |-- Runtime Adapter Interface
        |
        |-- Lite Adapter
        |     shared runtime Pod + gateway subprocesses
        |
        |-- Isolated Gateway Adapter
        |     agent-sandbox Sandbox or native Pod backend
        |     Hermes/OpenClaw gateway only
        |     no desktop
        |
        |-- Pro Desktop Adapter
              independent desktop Pod
              Webtop/KasmVNC/Selkies-style UI

ClawManager Gateway
  |
  |-- authn/authz
  |-- per-instance routing
  |-- model gateway injection
  |-- egress policy association
  |-- audit/logging

Isolated Gateway Runtime Requirements

The new runtime mode should satisfy:

• each agent/instance runs in an independent Kubernetes isolation unit
• no Webtop/Selkies/desktop process is started
• only Hermes/OpenClaw gateway/agent process is started
• workspace is persisted per instance
• resources are limited per instance
• secrets are injected according to tenant policy
• model configuration comes from ClawManager
• access is only through ClawManager gateway by default
• runtime placement respects tenant and nodegroup constraints
• egress policy can be enforced per agent

agent-sandbox Adoption Strategy

Add an abstraction instead of hard-coding directly to Deployments or directly to Sandbox everywhere.

Suggested backend interface:

RuntimeBackend
  CreateInstance(ctx, spec)
  StartInstance(ctx, id)
  StopInstance(ctx, id)
  SuspendInstance(ctx, id)
  DeleteInstance(ctx, id)
  GetStatus(ctx, id)
  GetEndpoint(ctx, id)
  AttachPolicy(ctx, id, policy)

Initial implementations:

• liteBackend: existing shared gateway runtime.
• proDesktopBackend: existing desktop runtime.
• sandboxBackend: new backend using agent-sandbox CRDs.

Fallback option:

• If agent-sandbox is not installed, support a minimal native Pod/Service backend for development clusters.
• Production should prefer agent-sandbox once validated.

Enterprise Egress Gateway

Current proxy-env injection is not enough for enterprise isolation. The product should model egress as first-class policy.

Required capabilities:

• per-agent / per-tenant / per-nodegroup policy
• allowlist and denylist
• default-deny profile
• predefined profiles for common resources:
  • model providers
  • GitHub
  • package registries such as PyPI/npm
  • internal services
  • no-internet mode
• audit logs for allowed and denied requests
• blocked-event reporting in API/UI
• forced proxy path where possible
• NetworkPolicy/Cilium/eBPF enforcement where proxy env can be bypassed

Runtime-Specific Enforcement

Lite
  Shared Pod makes Kubernetes NetworkPolicy too coarse for per-agent policy.
  Needs proxy identity, gateway token, runtime-level enforcement, UID/GID separation,
  iptables owner-match, or eBPF enforcement.

Isolated Gateway
  Per-agent Pod/Sandbox makes per-Pod NetworkPolicy natural.
  Cilium FQDN policy can provide domain-level allowlisting.
  ClawManager still owns policy definition and audit UX.

Pro
  Same network enforcement model as Isolated Gateway, plus desktop-specific exposure rules.

Nodegroup / Multi-Tenant Integration

The runtime must integrate with existing ClawManager nodegroup and tenant concepts.

Required model:

• tenant can be allowed or denied access to specific nodegroups
• nodegroup can advertise supported runtime modes
• nodegroup can define capacity profiles
• nodegroup can define allowed runtime classes such as default/gVisor/Kata
• tenant policy decides model access, egress profile, secret visibility, and internal network access
• instance creation carries tenant, nodegroup, runtime mode, image/runtime template, resource profile, and egress profile
• admin view can filter by tenant, nodegroup, runtime mode, policy, and risk state

Implementation Plan

Phase 1: Runtime Model

• Add isolated_gateway or equivalent runtime mode in instance/runtime schema.
• Define status mapping for Lite, Isolated Gateway, and Pro.
• Introduce runtime backend interface in the backend service layer.
• Preserve current Lite and Pro behavior.

Phase 2: agent-sandbox Spike

• Install agent-sandbox in a development cluster.
• Create one Hermes Sandbox from ClawManager.
• Create one OpenClaw Sandbox from ClawManager.
• Persist workspace through PVC.
• Route access through ClawManager gateway, not direct external Pod exposure.
• Verify start/stop/delete/status sync.

Phase 3: Gateway Integration

• Add gateway route resolution for Sandbox-backed instances.
• Generate per-instance access token/session.
• Keep Sandbox services private by default.
• Add health probing and status reconciliation.
• Record gateway access events.

Phase 4: Egress Policy

• Add egress policy schema and API.
• Bind egress policy at tenant/nodegroup/instance levels.
• Implement default-deny plus allowlisted profiles.
• For Sandbox/Pro, enforce with NetworkPolicy first.
• Evaluate Cilium FQDN policy for domain-level policy.
• Keep ClawManager egress proxy for audit and app-level policy.

Phase 5: Warm Pool and Capacity

• Evaluate SandboxWarmPool for faster startup.
• Map warm pools to nodegroups and runtime templates.
• Expose warm/cold capacity in admin UI.
• Add quota controls per tenant.

Phase 6: UI/API

• Add runtime mode selector: Lite / Isolated Gateway / Pro.
• Add egress profile selector.
• Add nodegroup placement selector.
• Show runtime backend, pod/sandbox identity, gateway endpoint, and policy status.
• Show denied egress events and audit history.

Open Questions

• Should the new mode be named isolated_gateway, agent_pod, pod_gateway, or sandbox?
• Should agent-sandbox be mandatory in production or optional by cluster capability?
• Which CNI/enforcement path is required for enterprise FQDN allowlisting: Kubernetes NetworkPolicy only, Cilium, or proxy plus eBPF?
• How should Lite per-agent egress isolation be represented if multiple agents continue sharing one Pod?
• Should Pro Runtime also migrate to agent-sandbox for lifecycle/PVC/warm-pool consistency, or remain on the current Deployment model?
• How much of Kasm's workspace/session model should influence Pro Runtime UX?

Acceptance Criteria

• A user can create an Isolated Gateway Hermes agent with no desktop UI.
• A user can create an Isolated Gateway OpenClaw agent with no desktop UI.
• Each Isolated Gateway instance gets an independent Kubernetes isolation unit.
• Workspace data survives restart/suspend where configured.
• Instance access goes through ClawManager gateway by default.
• No direct public Pod/Service exposure is required for normal use.
• Tenant and nodegroup placement constraints are enforced.
• Per-instance resource limits are enforced.
• Per-instance egress policy can be attached.
• Unauthorized egress is blocked and visible in audit/events.
• Pro and Lite continue to work unchanged.
• The implementation documents whether agent-sandbox is used directly, optionally, or rejected after spike, with evidence.