feat(multi-host-live-mount): one workspace, many hosts, proof the mirror is current - #94
Conversation
…ror is current Adds a skill for taking a fresh machine to a state where it mounts an existing Relayfile workspace, joins the fleet, and hosts a placed agent working inside the live-mounted tree with nothing cloned. This is the composition of three separately-documented primitives that lie when combined. `relayfile workspace join` -- the actual second-host primitive -- was absent from both setting-up-relayfile and orchestrating-agent-relay. The load-bearing claim, measured on a live production mount: lag: 0, pending: 0, a live daemon pid, and a .relay/state.json whose mtime is ticking right now are all simultaneously compatible with days-stale content. Every one of those signals measures the daemon's own activity, not the freshness of the bytes it serves. Only a content-level assertion against a fact known true now proves currency. Observed: daemon up 3h06m, lag 0s, all four provider state.json files rewritten within ~4 minutes, 9 content files touched in 3h -- and digests/today.md two days old while its cloud revision advanced rev_1553031 -> rev_1553124. The github projection was separately behind the provider (newest projected cloud issue #2935, #2949 known present and absent), which no local-vs-cloud assertion can detect. linear and notion left explicitly uncertified rather than assumed. Two defects documented: - stale projection with every health signal green - relayfile tree pagination: higher --depth returns FEWER rows (/linear depth20=100, depth3=325, depth3 --json=596, actual 3072); only --json exposes nextCursor, and --cursor is not implemented. A single tree call is a page, not a tree -- so a naive currency assertion passes by omission. The shipped assertion walks --depth 1 per directory and prints mandatory coverage. Also notes that broker-spawned agents receive workspace keys and agent tokens as argv, readable by any local user via ps. Ships 10 named assertions; all embedded bash and python blocks are syntax- checked and both assertion scripts were executed against a live workspace (PASS on /github/_agents 51/51, FAIL catching the real staleness on /digests). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
Warning Review limit reached
Next review available in: 50 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThe PR adds the ChangesMulti-host live mount skill
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Operator
participant Broker
participant RelayfileWorkspace
participant TargetHost
Operator->>Broker: complete enrollment and preflight
Broker->>RelayfileWorkspace: join and mount workspace scope
RelayfileWorkspace-->>Broker: pass currency and projection checks
Broker->>TargetHost: discover matching node and place agent
TargetHost-->>Operator: confirm execution in mounted tree
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b865792d69
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@skills/multi-host-live-mount/SKILL.md`:
- Around line 336-344: Update the bounded polling assertion and the
corresponding assertion around the lines 383-384 so success exits with status 0,
while the final failed check prints its failure message and exits with status 1.
Ensure the loop does not sleep or fall through after the final attempt, and
avoid returning the status of echo in either branch.
- Line 33: Update the observed-failure link in SKILL.md to use the matching
heading fragment
`#the-observed-failure-every-health-signal-green-content-days-stale` instead of
the stale daemon-alive fragment.
- Around line 281-292: Update the mirror verification logic around the per-file
comparison and final ok calculation to validate file content using an
authoritative digest or relayfile read, rather than size alone or the unused rev
value. Collect the complete cloud path set and compare it with local paths under
mirror + scope, excluding reserved .relay state; mark extra local files as
errors or otherwise make ok false, while preserving the existing missing, stale,
truncated, and error reporting.
- Around line 330-334: Update the HOST A write probe around MARK so it writes
valid JSON conforming to the target resource’s .schema.json, placing the marker
in an appropriate schema-defined field instead of writing plain text. Preserve
the marker’s uniqueness and continue reporting writeback status and the marker.
- Around line 455-460: Export RELAY_ENROLLMENT_TOKEN, RELAY_ENROLLMENT_URL, and
RELAY_NODE_NAME before the chained sandbox-node-bootstrap.sh preflight and
enroll commands so both phases inherit them; retain the existing token cleanup
trap.
- Around line 469-471: Update the fleet spawn command in the live-mount workflow
to pass --cwd "$MIRROR" before --task, ensuring the agent starts inside the
mounted tree while preserving the existing no-clone task instruction.
- Around line 462-468: Update the multi-host mount gate following the relayfile
mount command and assert mirror freshness for every mounted scope the agent can
read: /digests, /linear, /github, and /notion. Ensure the mount command
explicitly uses the documented scope-aligned options, including --remote-path,
--creds-file, --state-dir, and --local-layout scoped, before any placement
occurs.
- Around line 379-382: Update the `known-true-now` assertion in the `NEWEST`
lookup block to match the documented `by-id` filename shape, checking for
`${KNOWN}__*.json` rather than the literal `$KNOWN.json`; preserve the existing
`NEWEST` calculation and diagnostic output.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 68b33386-94c5-4cc2-b0f4-b727d756095b
📒 Files selected for processing (3)
README.mdprpm.jsonskills/multi-host-live-mount/SKILL.md
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 3 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
…on-demand surfaces Three corrections found by checking the skill against a second mount surface that appeared on the same host. 1. Real bug in the shipped assertion. The coverage counter used os.walk with the default followlinks=False, which reports 0 files for a symlinked mount or symlinked provider dir -- and mounts are routinely symlinks (observed: .integrations -> ~/.agentworkforce/pear/relayfile/workspaces/<uuid>). PASS/FAIL was unaffected because per-file os.path.exists follows symlinks, but the coverage line -- the thing added to stop a truncated page reading as a clean pass -- silently measured nothing. Now passes followlinks=True. 2. One repo can hold several mounts from DIFFERENT workspaces. Observed senses/github (rw_7ccfea89) and .integrations/github (bare UUID 50587328-...) side by side, both exposing a github/ subtree. Certification is per (workspace, mirror root) pair, never per machine. Both rw_* and bare UUID workspace ids are live, sometimes on one host. 3. Not every mount is a full projection. On-demand/event-scoped surfaces do not download history by design, so a cloud file with no local counterpart is expected, not stale -- bulk MISSING there means nothing. Establish projection mode before calling MISSING a defect; STALE remains real in both modes. Adds named assertions mount-identified and projection-mode-known, three troubleshooting rows, and fixes a stale internal anchor left by the earlier heading rename. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follow-up commit
|
| shape | os.walk default |
followlinks=True |
|---|---|---|
| direct dir | 2 files | 2 files |
| symlinked root | 2 files | 2 files |
| symlinked subdir | 0 files | 2 files |
PASS/FAIL was unaffected (per-file os.path.exists follows symlinks), but the coverage: line — added specifically to stop a truncated page reading as a clean pass — silently measured nothing. Fixed.
2. One repo can hold several mounts, from different workspaces. Observed side by side, both exposing a github/ subtree:
chief/senses/github/… ← workspace rw_7ccfea89
chief/.integrations/github/… ← workspace 50587328-… (bare UUID, different workspace)
Certification is per (workspace, mirror root) pair, never per machine. Note both rw_* and bare-UUID workspace ids are live, sometimes on one host — the artifact previously implied rw_* only.
3. Not every mount is a full projection. On-demand / event-scoped surfaces deliberately do not download history, so a cloud file with no local counterpart is expected. Bulk MISSING there means nothing, and reading it as staleness is a false positive at scale. Establish projection mode before calling MISSING a defect; STALE (present but byte-divergent) stays real in both modes.
Adds named assertions mount-identified and projection-mode-known (12 total), three troubleshooting rows, and fixes a stale internal anchor from the earlier heading rename.
Re-verified: 10/10 bash blocks bash -n, 2/2 python heredocs ast.parse, all internal anchors resolve, 0 live credentials. Still do not merge.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@skills/multi-host-live-mount/SKILL.md`:
- Around line 198-203: Update the Assertion A procedure to accept projection
mode explicitly and report the selected mode with its result. For
on-demand/event-scoped projections, ignore only missing entries; keep stale,
truncated, and errors fatal, while full projections retain the existing missing
check. Apply the same failure-condition change to the related Assertion A
content at the additional referenced section.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: c8a70876-b6b9-4e7b-9e50-2ef291f7fdd1
📒 Files selected for processing (1)
skills/multi-host-live-mount/SKILL.md
There was a problem hiding this comment.
All reported issues were addressed across 1 file (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
There was a problem hiding this comment.
Actionable comments posted: 11
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
skills/multi-host-live-mount/SKILL.md (1)
654-659: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick winUse a private temporary file for fleet inventory.
/tmp/nodes.rawis predictable, may be world-readable under a permissiveumask, and remains after the command. A pre-existing symlink or concurrent invocation can also redirect or overwrite the output.Use
mktemp, setumask 077, pass the generated path to Python, and remove it with anEXITtrap.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@skills/multi-host-live-mount/SKILL.md` around lines 654 - 659, Update the fleet inventory flow around the agent-relay command and Python reader to create a private temporary file with mktemp under umask 077, pass the generated path to Python, and register an EXIT trap to remove it. Replace the predictable /tmp/nodes.raw path while preserving the existing redirected output and JSON parsing behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@skills/multi-host-live-mount/SKILL.md`:
- Around line 533-538: Update the failure-output example near
assert-mirror-current.sh so it matches the current checker: retain the 80 cloud
and 84 local path counts and include the corresponding “EXTRA LOCAL PATHS”
diagnostic with the four extra paths, or revise the sample counts to match the
shown diagnostics.
- Around line 597-599: Update the relayfile mount invocation in the multi-host
live-mount workflow so --state-dir is unique for each mount, deriving it from
the workspace ID, mirror root, and scope set rather than only <node>. Preserve
separate state directories across mounts on the same host and avoid reusing the
node-only path.
- Around line 592-604: Validate that MOUNT_SCOPES is non-empty before
constructing MOUNT_ARGS or mounting; fail the workflow when it is empty, unless
the implementation explicitly switches to and asserts a full-root projection.
Preserve the existing per-scope assertion loop for configured scopes.
- Around line 407-409: Update the mirror-matches-cloud assertion output to
include workspace, mirror, and scope alongside the result and counts, using the
existing identity values. Apply the same fields to the cross-host and
known-current assertion outputs so every result identifies its exact mount.
- Around line 287-292: Make the assert-mirror-current.sh implementation in the
SKILL.md workflow executable by creating the script file before the placement
workflow invokes it, and set its executable permissions before the existing
./assert-mirror-current.sh call around the placement steps. Prefer shipping it
as a package file if that matches the repository’s conventions; otherwise add
explicit extraction and chmod steps, preserving the current script arguments and
invocation.
- Around line 606-614: Update the fleet spawn flow around `RELAY_AGENT_TOKEN`
and `agent-relay fleet spawn` to define and validate a control-host manifest
containing the target-local mirror path and exact mount scopes, rather than
reusing target-host variables `WS`, `MIRROR`, and `MOUNT_SCOPES`. Pass the
validated manifest values explicitly to `--cwd` and the task text, preserving
the requirement to work only within the mounted scopes.
- Around line 447-465: Update the cross-host probe in the documented workflow to
use an explicitly declared writable scope and exact resource path: include the
chosen scope in both hosts’ --remote-path configuration, grant write permission
on the source host, and target the corresponding resource under that scope.
Replace the broad grep over MIRROR_B with polling for the exact expected
wb-$MARK.json path. For read-only hosts, test and assert write rejection rather
than expecting reverse-direction visibility.
- Around line 503-508: Update the known-true-now assertion around known_files so
it succeeds only when at least one glob match is a regular file. Iterate over
the matched paths and use [[ -f "$file" ]] before printing the PASS result;
retain the existing failure behavior when matches are absent or only
directories.
- Line 254: Complete the pre-placement gate in the multi-host mount procedure by
running all three assertions, including known-true-now and each write direction
permitted by the permission model, before fleet spawn. Update the assertion
sequence around “Three assertions. Run all three.” and lines 601-604 so it does
not run only mirror-matches-cloud; for intentionally read-only hosts, validate
the forbidden write direction as a permission denial rather than requiring
visibility.
- Around line 398-405: Update the pass/fail guard around fatal and ok so an
on-demand check cannot pass without comparing cloud content: require cloud_paths
to be nonempty and at least one cloud file to be classified as match or stale,
unless a caller-provided local anchor explicitly permits the pass. Preserve the
existing full-mode missing-file failure and all other error conditions.
- Around line 381-388: Update the local-path validation around lp to use
os.path.lexists(lp), preserving broken symlinks as present entries rather than
classifying them as MISSING. Before open(lp, "rb"), validate that the path is a
regular file and record non-regular entries, including broken symlinks and
FIFOs, in errors without opening them.
---
Outside diff comments:
In `@skills/multi-host-live-mount/SKILL.md`:
- Around line 654-659: Update the fleet inventory flow around the agent-relay
command and Python reader to create a private temporary file with mktemp under
umask 077, pass the generated path to Python, and register an EXIT trap to
remove it. Replace the predictable /tmp/nodes.raw path while preserving the
existing redirected output and JSON parsing behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 6b887e3a-3169-462c-8c55-6c8111f1cdf5
📒 Files selected for processing (2)
prpm.jsonskills/multi-host-live-mount/SKILL.md
There was a problem hiding this comment.
All reported issues were addressed across 2 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 5 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@skills/multi-host-live-mount/scripts/assert-cross-host-write-visible.sh`:
- Around line 15-21: The script’s source-file precheck incorrectly requires HOST
A’s local mirror on HOST B. Remove the SOURCE_EXPECTED/SOURCE_MIRROR dependency
from assert-cross-host-write-visible.sh and retain the cloud propagation and
target-file validation using TARGET_EXPECTED; perform source validation
separately on HOST A if needed.
In `@skills/multi-host-live-mount/scripts/assert-known-true-now.sh`:
- Around line 15-21: Validate KNOWN before constructing known_files, rejecting
pathname separators and shell glob metacharacters so it can only identify the
intended anchor. Preserve the existing PASS output and exit behavior for valid
KNOWN values, while invalid values must not be used in the glob pattern.
In `@skills/multi-host-live-mount/SKILL.md`:
- Around line 692-693: Update the rejection probe around the jq redirection and
relayfile read validation so an immediate permission-denied failure from the
read-only mount is accepted; capture the local write result, fail on unexpected
write errors, and then confirm relayfile read cannot find REJECT_FILE, failing
if the cloud path exists.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 590f58f8-faaa-4fab-863d-54a62b992973
📒 Files selected for processing (5)
prpm.jsonskills/multi-host-live-mount/SKILL.mdskills/multi-host-live-mount/scripts/assert-cross-host-write-visible.shskills/multi-host-live-mount/scripts/assert-known-true-now.shskills/multi-host-live-mount/scripts/assert-mirror-current.sh
🚧 Files skipped from review as they are similar to previous changes (1)
- prpm.json
There was a problem hiding this comment.
All reported issues were addressed across 5 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
3 participants
Adds
skills/multi-host-live-mount/SKILL.md— how to take a fresh machine to one Relayfile workspace mounted across multiple hosts, with a placed agent working in a live-mounted tree and nothing cloned.Do not merge. Opened for review only.
Why a new skill
This is the composition of three primitives that are documented separately and lie to you when combined:
relayfile setup/ mountsetting-up-relayfileagent-relay fleetenroll / spawnorchestrating-agent-relayworkspace-layoutrelayfile workspace join— the actual second-host primitive, including its opt-in--writepermission model — appears in neither existing skill. That was the gap.The load-bearing claim
Defect 1 — stale projection, every health signal green
Measured on a live production mount (
rw_7ccfea89):relayfile statusmode: poll,lag: 0s,pending writebacks: 0.relay/state.jsondigests/today.mdrev_1553031→rev_1553123→rev_1553124while local size never moved off 10112githubprojection#2935;#2949known to exist and absent — the cloud projection is behind the providerlinear,notionThe
githubrow is the important one: it is a projection gap, so a local-vs-cloud assertion passes — local and cloud agree, on stale data. Only theknown-true-nowassertion catches that class.relayfile supervisor statusreported the service not found while the mount ran fine — supervisor state is independent of mount health.Defect 2 —
relayfile treepagination / CLI cursor gapThe listing is paginated and the pagination is unusable:
tree /linear --depth 20tree /linear --depth 3tree /linear --depth 3 --jsonnextCursor--depthreturns fewer rows.--jsonexposesnextCursor.--cursoris not implemented (error: flag provided but not defined: -cursor).A single
treecall is a page, not a tree. The first draft of the currency assertion in this PR would have passed by omission on a truncated page. The shipped version walks--depth 1per directory and prints a mandatorycoverage:line.Fleet node listing
Default
fleet nodesreturned 3 records while omittingsf-mini—status: online,live: true, advertisingspawn:claude|codex|gemini|opencode. A live spawn-capable node, invisible in the default view.--allreturned exactly 400 on 3/3 runs with no--limitflag (server-side cap); a capability-filtered query returned 33. The id set was stable across three consecutive runs — nondeterministic subsets were not reproduced; what varied was the live count (20 → 20 → 21).Credential handling
Broker-spawned agents receive workspace keys and agent tokens as argv, readable by any local user via
ps auxww. Live values were observed on the test host and are deliberately not reproduced here or in the skill; they have been reported for rotation out-of-band. The skill documents this as a per-node credential hazard and recommends narrowly-scoped per-node delegated credentials.Named assertions shipped
workspace-joined-not-created,scope-declared,mirror-matches-cloud,listing-coverage-reported,known-true-now,uncertified-scopes-named,cross-host-write-visible,write-permission-matches-intent,placement-target-live,placement-executed,nothing-cloned.Verification of this PR's own contents
bashblocks passbash -nast.parse/github/_agents(checked=51, match=51, coverage 51 cloud = 51 local, exit 0) and FAIL on/digests(checked=80, match=78, stale=2, exit 1) catching the real defectprpm.jsonandREADME.mdchanges are purely additiveVersions:
relayfile0.10.39,agent-relay11.4.2.🤖 Generated with Claude Code