Tracking: harden Maestro remote servers for hosted runtimes

[haasonsaas]

Current Status

• 2026-04-23: #79 is now complete via merged PR #133 (0f9609e2f4e78966dcca8c3ebaff5bd06c6f89b0). Hosted session scope and headless runtime scope now bind to verified JWT/shared-token/API-key principals, reuse scope across JWT rotation for the same user/workspace, preserve explicit local anon mode, and keep full long principal identifiers in the hash input so the hashed scope-key path does not collide across tenants.
• 2026-04-23: #81 is complete and should be read as landed public behavior, not open implementation work. Public main has the hosted runner entrypoint/config surface, required workspace-root and runner-session validation, hosted readyz/draining behavior, and the single-session/runtime boundary.
• 2026-04-23: #83 is closed after re-validating current public main; Maestro-core drain/export/restore is already present, including persisted snapshot-manifest.v1, workspace export-path recording, Rust restore-manifest loading, restored_from_snapshot replay, and not-ready partial-restore behavior.
• 2026-04-23: #84 is complete as a runtime decision/parity-contract issue: Rust is the target runtime kernel, TypeScript remains the compatibility bridge, and the public conformance/runtime surface reflects that split.
• 2026-04-23: #85 is complete as a conformance issue: public CI already runs rust-hosted-conformance, and local validation on current public main is green for both hosted-runner Rust tests and the hosted headless conformance suite.
• 2026-04-23: #77 is now complete via merged PR #135 (d3f9bbe6f7a8d57f85e73b172fb06768aa1aa6e2). Hosted/headless sessions now carry an explicit workspaceRoot, agent/runtime context uses that effective root instead of the web-server process cwd, and utility command/read/search/watch calls are enforced against the assigned workspace boundary.
• Remaining work in this umbrella is the actual remote-server hardening still tracked below: fleet-safe routing/ownership across replicas (#78) and deploy/runtime-image production hardening (#80). Cross-repo orchestration for drain/restore now lives in evalops/platform#838 and evalops/deploy#1126.

Agent handoff: read this first

This is the top-level tracker for making Maestro remote/headless servers safe to run as hosted infrastructure, not just as a local web process.

Use this order:

1. Bind headless remote sessions to explicit workspace sandboxes #77 — Bind every remote/headless runtime to an explicit workspace sandbox. Without this, utility commands/file search/file reads/file watches and MCP/LSP/git/team-memory context are grounded in the server process cwd or caller-provided cwd.
2. Make headless runtime fleet-safe across Maestro web replicas #78 — Make the headless runtime fleet-safe across web replicas. Without this, the Helm chart's replicaCount: 2 fights the process-local HeadlessRuntimeService map and SSE/POST session affinity.
3. Bring Maestro Helm and runtime image up to hosted remote-server requirements #80 — Bring Helm/runtime-image/web-server production settings up to hosted remote requirements.

Keep this linked with the Platform integration spine:

• Durable ledger: Use Platform agent-runtime as Maestro's durable run ledger #72 and evalops/platform#671
• Platform-side Maestro worker adapter: evalops/platform#780
• Overall integration tracker: evalops/platform#765
• Events/correlation: Publish Maestro telemetry as typed CloudEvents onto platform's event bus #49, evalops/platform#762, evalops/platform#763
• Policy/tooling: Route Maestro approvals, sandbox, and firewall decisions through Platform policy services #73, Consume Platform-governed MCP skill endpoints as Maestro tool sources #74

Current source read

Reviewed heads on 2026-04-22:

• evalops/maestro@ae6e88a22d3a36d34d07c5c507375cfd7fdcd372
• evalops/platform@7b003938349939052f90ae41241599993a7c06c9

Relevant Maestro source:

• src/web-server.ts creates the app-wide web/headless runtime services and builds agents from process.cwd().
• src/server/headless-runtime-service.ts owns the process-local runtime map, replay buffer, subscribers, heartbeats, controller leases, and utility operations.
• src/server/handlers/headless-sessions.ts exposes /api/headless/* create/subscribe/events/messages/heartbeat/disconnect endpoints.
• packages/tui-rs/src/headless/remote_transport.rs is the Rust HTTP/SSE client for remote headless sessions.
• deploy/helm/maestro/* currently deploys two replicas with only basic probes/env.

Why this is separate from the Platform runtime issues

#72 and evalops/platform#780 are about adopting Platform's durable AgentRun ledger and driving Maestro headless from Platform. This tracker is about Maestro's own remote-server boundary: workspace isolation, multi-replica behavior, identity scoping, and production deploy shape. Both tracks are needed; neither replaces the other.

Acceptance Criteria

• Each sub-issue above is cross-linked from this tracker.
• A hosted Maestro remote server has a defined workspace/sandbox root per session and cannot operate on arbitrary server filesystem paths.
• Running more than one web replica is either explicitly disabled or backed by sticky routing plus durable runtime-owner/session routing semantics.
• Session scope and authorization use verified principals and workspace/team claims.
• The Helm chart documents and enforces the minimum production contract for API auth, Redis/Postgres/session storage, probes, graceful draining, and SSE/headless routing.

---

Remote Runner Follow-Up Issues Added 2026-04-22

These issues connect this Maestro hardening tracker to the account-scoped remote runner build in evalops/platform#781:

• Add hosted runner entrypoint for single-session Maestro runtime pods #81 — Add hosted runner entrypoint for single-session Maestro runtime pods
• Add maestro remote commands backed by EvalOps runner sessions #82 — Add maestro remote commands backed by EvalOps runner sessions
• Add runner workspace snapshot, drain, and resume hooks #83 — Add runner workspace snapshot, drain, and resume hooks
• evalops/platform#782 — remote-runner session lifecycle APIs and durable store
• evalops/platform#783 — remote-runner gateway for Maestro headless attach routing
• evalops/platform#784 — Kubernetes provisioner for one Maestro runtime pod per runner session
• evalops/platform#785 — account policy, entitlements, and metering
• evalops/deploy#1024 — GitOps runtime plane for remote Maestro runner sessions
• evalops/deploy#1025 — GCP foundations for remote-runner identity and workspace artifacts

Tech split correction for agents picking this up: Maestro is already Rust-first for the native TUI/agent path, including tool execution and remote attach transport. The current hosted web/headless Docker path still runs Node/TypeScript, so v1 may bridge through that surface, but the runner contract should not assume Node is the target. Prefer a Rust-native hosted runner once it can expose the same headless HTTP/SSE/session contract. Keep Platform remote-runner control plane/gateway/provisioner in Go.

Related: #84 — Rust core runtime parity contract

---

Runtime Parity / Conformance Links

• Define Rust core runtime parity contract for hosted Maestro #84 — Rust core runtime parity decision for hosted Maestro.
• Add headless runtime conformance suite for Rust parity #85 — headless runtime conformance suite that should gate Rust-hosted runner parity.

Use #85 as the protocol-level acceptance bar: lifecycle, subscribe/events/messages/heartbeat/disconnect, viewer/controller roles, permission request/response, utility operations, workspace-root enforcement, cursor replay/reset, and drain/resume behavior should match across the current TypeScript web/headless bridge and the target Rust runtime.

[haasonsaas]

Cross-linking the remote-server hardening issue graph:

• Bind headless remote sessions to explicit workspace sandboxes #77 — workspace/sandbox root for every headless remote session
• Make headless runtime fleet-safe across Maestro web replicas #78 — fleet-safe runtime routing across web replicas
• Use verified principals for hosted remote session scope and authorization #79 — verified principal/session scoping for hosted remote sessions
• Bring Maestro Helm and runtime image up to hosted remote-server requirements #80 — Helm/runtime image production hardening

Related integration spine:

• Use Platform agent-runtime as Maestro's durable run ledger #72 and evalops/platform#671 for the durable AgentRun ledger
• evalops/platform#780 for the Platform-side Maestro headless worker adapter
• evalops/platform#765 for the canonical Maestro ↔ Platform tracker
• Publish Maestro telemetry as typed CloudEvents onto platform's event bus #49 / evalops/platform#762 / evalops/platform#763 for typed events and correlation
• Route Maestro approvals, sandbox, and firewall decisions through Platform policy services #73 / Consume Platform-governed MCP skill endpoints as Maestro tool sources #74 for policy and governed tool-plane integration

[haasonsaas]

Adding the product/control-plane follow-up from the remote runner discussion:

• evalops/platform#781 proposes the account-scoped remote runner control plane where users can start/attach/stop timed Maestro sessions against their EvalOps account.

Relationship: this tracker (#76) hardens Maestro as the hosted remote runtime. evalops/platform#781 is the Platform product/control plane that provisions, meters, routes, and tears down those runtimes.

[haasonsaas]

Correction for future agents: Rust is already the primary native Maestro agent/TUI path; Node is still the current web/headless server image. Please do not implement this tracker as “keep runtime Node, keep Rust client-only.” The better target is a hosted-runner contract that can be served by the current Node bridge first and by a Rust-native runner as it matures.

[haasonsaas]

Cross-linking #84: the remote hardening tracker should steer toward a Rust core runtime with rough parity against the current TypeScript web/headless surface. The downloaded source snapshot reinforced that the durable remote/session layer and the runtime kernel need a clear protocol contract between them.