fix(mount): fail stalled bootstraps loudly

[khaliqgant]

Closes #419

Summary

• make bootstrap stall-limit and directory-traversal-limit failures terminal in the CLI daemon, preserving the failed checkpoint across supervisor/process restarts
• report mount health independently from provider health, with explicit bootstrapping and stalled phases, the blocked path/page checkpoint, pending directory count, and actionable reason
• add an authoritative caller-visible totalFiles tree response so progress never renders an invalid N/0 denominator; suppress it persistently when reserved runtime pruning makes the generic total unreachable
• cap the bounded-tree frontier at 50,000 distinct directories; operators can inspect the named subtree and deliberately raise the bound to resume the persisted checkpoint
• keep the OpenAPI contract and TypeScript/Python SDK response types synchronized
• restore the published Linux x64 mount package's missing 0.10.42 lock entry so root npm ci and PR checks can execute

Root cause

The mount syncer already produced a typed hard failure after the checkpoint-stable cycle limit, but the CLI daemon treated it as an ordinary failed poll and retried forever. A per-cycle deadline branch could also mask the typed stall because the error wrapped context.DeadlineExceeded. Separately, the CLI snapshot writer replaced the mountsync public state and lost its structured bootstrap error, so provider health remained the only green status visible to operators.

The reported empty cursor is not evidence that the saved traversal was discarded: the directory frontier and page cursor were already persisted and restored. An empty cursor is valid at a directory boundary. Status and errors now include the current directory, page cursor/offset, and pending frontier together, removing that ambiguity.

Validation

• go test ./...
• go vet ./...
• go test -race ./internal/mountsync ./cmd/relayfile-cli -run 'Test(BootstrapStallCycleGuardPersistsAndFailsHard|BootstrapDirectoryTraversalLimitFailsLoudlyAndPersistsPath|CLIMountLoopStopsAndPersistsStatusOnBootstrapStall|StatusDistinguishesStalledBootstrapFromHealthyProvider)' -count=1
• scripts/check-contract-surface.sh
• node scripts/check-sdk-parity.mjs
• npm run build --workspace=packages/core
• npm run typecheck --workspace=packages/sdk/typescript
• npm run build --workspace=packages/sdk/typescript
• npm run test --workspace=packages/sdk/typescript (239 tests)
• npm ci
• client codegen diff check, build, typecheck, and test (22 passed, 3 skipped)
• uv run --project packages/sdk/python --extra dev pytest -q packages/sdk/python/tests (96 tests)

All mount-loop regression tests use scratch directories and an httptest server. The live shared workspace was not restarted, stopped, re-homed, or used for reproduction.

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[coderabbitai]

Warning

Review limit reached

@khaliqgant, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 35 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: eaa434a6-67db-469b-b673-cc0b03e8da81

📥 Commits

Reviewing files that changed from the base of the PR and between 36a6dad and 8907b27.

📒 Files selected for processing (3)

• cmd/relayfile-cli/main.go
• internal/mountsync/syncer.go
• internal/mountsync/syncer_test.go

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0feb4bbc-1bcf-43d2-9924-9998e8625d40

📥 Commits

Reviewing files that changed from the base of the PR and between eb7e882 and 36a6dad.

⛔ Files ignored due to path filters (1)

• package-lock.json is excluded by !**/package-lock.json

📒 Files selected for processing (19)

• cmd/relayfile-cli/main.go
• cmd/relayfile-cli/main_test.go
• cmd/relayfile-cli/mount_up_path_priority_test.go
• cmd/relayfile-mount/main.go
• docs/environment-variables.md
• internal/httpapi/server.go
• internal/httpapi/server_test.go
• internal/mountsync/bootstrap_test.go
• internal/mountsync/syncer.go
• internal/mountsync/syncer_test.go
• internal/relayfile/store.go
• internal/relayfile/store_test.go
• openapi/relayfile-v1.openapi.yaml
• packages/cli/CHANGELOG.md
• packages/sdk/python/src/relayfile/types.py
• packages/sdk/python/tests/test_client.py
• packages/sdk/typescript/CHANGELOG.md
• packages/sdk/typescript/src/client.test.ts
• packages/sdk/typescript/src/types.ts

---

📝 Walkthrough

Walkthrough

The change adds caller-visible tree totals, bounded bootstrap traversal, persisted terminal stall states, richer progress metadata, and CLI reporting for stalled or unknown-total bootstrap phases.

Changes

Bootstrap observability and convergence

Layer / File(s)	Summary
Authoritative tree totals internal/relayfile/store.go, internal/httpapi/server.go, openapi/..., packages/sdk/..., internal/.../*_test.go	Tree responses now expose stable pre-pagination totals after access filtering. API schemas, SDK types, and tests cover the new field.
Bounded bootstrap traversal internal/mountsync/syncer.go, internal/mountsync/*_test.go	Bootstrap tracks directory and file progress, enforces a configurable directory limit, preserves blocked paths, handles unreliable totals, and resets traversal metadata after completion or migration.
Terminal error persistence internal/mountsync/syncer.go, cmd/relayfile-cli/main.go	Stall and traversal-limit errors persist across restarts, prevent repeated cloud retries, support re-arming after configuration changes, and publish detailed stalled status.
CLI status and lifecycle reporting cmd/relayfile-cli/*, cmd/relayfile-mount/main.go, docs/environment-variables.md, packages/cli/CHANGELOG.md	Mount loops stop on terminal bootstrap errors. Status output reports mount state, progress, pending directories, current path, and actionable reasons. Unknown totals no longer render as zero denominators.

Estimated code review effort: 4 (Complex) | ~60 minutes

Merge Risk: ⚪ Minimal · up to 36a6d

The changes make stalled bootstrap failures explicit, preserve progress state, and synchronize API surfaces; no actionable merge-blocking risk remains beyond normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant CLI mount loop
  participant Syncer
  participant Remote tree API
  participant Persisted state
  CLI mount loop->>Syncer: start bootstrap
  Syncer->>Remote tree API: request paginated tree
  Remote tree API-->>Syncer: tree page and caller-visible total
  Syncer->>Persisted state: save progress or terminal reason
  Syncer-->>CLI mount loop: continue or return terminal error
  CLI mount loop->>Persisted state: publish mount snapshot

Loading

Possibly related PRs

• AgentWorkforce/relayfile#361: Extends earlier bootstrap stall handling.
• AgentWorkforce/relayfile#352: Also changes mount-loop and initial bootstrap synchronization behavior.
• AgentWorkforce/relayfile#416: Also modifies bootstrap progress state in internal/mountsync/syncer.go.

Suggested reviewers: willwashburn, kjgbot

Poem

I’m a rabbit who watches the paths,
Counting each file through the grass.
When progress stalls, I leave a clear sign,
With the blocked path and reason in line.
No false zero, no endless run—
Bootstrap stops when its work is done.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 13.64% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the primary change: terminal handling for stalled mount bootstraps.
Description check	✅ Passed	The description directly explains the bootstrap, status, progress, checkpoint, API, and validation changes.
Linked Issues check	✅ Passed	The changes address issue #419 by persisting terminal failures, separating stalled health, fixing totals, preserving checkpoints, bounding traversal, and adding regression tests.
Out of Scope Changes check	✅ Passed	The reviewable changes support issue #419 through bootstrap handling, status reporting, progress totals, traversal limits, and synchronized API contracts.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/issue-419-mount-bootstrap-stall

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8c356aa172

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

Relayfile Eval Review

Run: .relayfile/evals/runs/2026-08-15T00-58-02-919Z-HEAD-provider
Mode: provider
Git SHA: 458ddb0

Passed: 4 | Needs human: 0 | Reviewable: 0 | Missing output: 0 | Failed: 0 | Skipped: 0

Human Review Cases

No reviewable human-review cases captured Relayfile output.

[cubic-dev-ai]

All reported issues were addressed

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 6 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[khaliqgant]

Required: verify against the built CLI, not just unit tests

Do not mark this done on green unit tests alone. Build this repo and exercise the real relayfile CLI.

Reproduce → fix → re-verify

npm ci && npm run build      # or the repo's build script
relayfile status             # against a workspace with an incomplete bootstrap

The live reproduction is on workspace rw_7ccfea89:

bootstrapping: 27392 files (started 164h+ ago)     <- 6.8 days, never completes
mount sync cycle failed: bootstrap stalled for 31 consecutive checkpoint-stable cycles (limit 20, cursor "")
github       healthy  lag 0s                        <- reports healthy anyway

Prove, with actual terminal output for both the failing and passing run:

1. A stalled bootstrap no longer reports healthy — this is the more damaging half of the bug.
2. Exceeding the stall limit escalates rather than failing one cycle and retrying forever.
3. Restarting the mount does not discard bootstrap progress.
4. The N/M files denominator expresses real completion (currently 27392/0).

Check the cursor discrepancy first: the stall message says cursor "" while the resume message says from persisted cursor. If the persisted cursor is not honoured on resume, that likely explains zero forward progress and may be the whole fix.

Constraints

• Do not relayfile stop, re-home, or restart workspace rw_7ccfea89. It is live and shared by ~16 running agents. Reproduce against a scratch workspace.
• mergePolicy: never — do not merge. Stop at review.

[khaliqgant]

Review — issue #419 mount bootstrap stall (reviewer: ar-419-review-relayfile)

Reviewed against origin/main (eb7e882). Verified locally: go build ./..., go test ./internal/mountsync/ ./cmd/relayfile-cli/ ./internal/httpapi/ ./internal/relayfile/ all pass, and scripts/check-contract-surface.sh reports SDK parity check passed / contract check passed. CI on this PR is green across all 9 checks.

Branch reconciliation (please resolve before merge)

There are two heads for issue #419:

• factory/419-agentworkforce-relayfile-a4efde3a @ bcc5614 — pushed, no PR opened
• codex/issue-419-mount-bootstrap-stall @ c934d5f — this PR

This PR is a strict superset of bcc5614 and contains a fix the factory branch is missing, so this PR is the one to keep. Worth confirming explicitly so the factory branch doesn't get opened as a competing PR later.

The delta that matters: on bcc5614, page.TotalFiles is adopted as the denominator, but the server total counts every caller-visible file including reserved mount-runtime (.relay/…) descendants, which pullRemoteFullTree deliberately prunes. FilesSynced therefore can never reach FilesTotal, so N/M shows an unreachable target for the whole bootstrap — a different wrong denominator than the 27392/0 in the issue. This PR fixes it properly via the persisted BootstrapFilesTotalUnavailable latch, with TestBootstrapProgressSuppressesRuntimeInclusiveTotalAcrossRestart covering the across-restart case (the important one, since a fresh process revisits the root page with the server's raw total and must not restore the suppressed denominator).

What I verified against the issue's definition of done

• Stall escalates rather than retrying forever — IsBootstrapTerminalError is now honoured in cmd/relayfile-mount/main.go and in both runMountLoopWithAuthLock paths (initial cycle and the reconcile ticker), so the typed hard stop terminates the runner instead of being absorbed as one failed cycle. ✅
• Terminal across restart — persistedBootstrapTerminalError returns before any cloud request, and TestBootstrapStallCycleGuardPersistsAndFailsHard asserts a restart issues zero new tree requests. That is the right assertion. ✅
• status no longer reads healthy while stalled — savePublicState promotes LastError.Kind == "bootstrap_stalled" to status/phase stalled, canonicalViewStatus propagates the phase, and runStatus prints a distinct mount: stalled line. Provider rows still show their own health, which is correct — the providers are healthy; the mount is not. ✅
• Restart does not discard progress — cursor/directories/pageOffset are persisted on the traversal-limit path before returning, and the re-arm test proves raising the bound resumes the saved checkpoint to completion rather than restarting. ✅
• Regression test for the escalation path — present and genuinely load-bearing. ✅
• Contract sync — totalFiles added to openapi/relayfile-v1.openapi.yaml, the store, both SDKs, and handleTree. Per CLAUDE.md this was required; it's done. ✅

I also confirmed the handleTree claim that totalFiles is stable across pagination: visibleFiles is accumulated by paginating QueryFiles over the whole PathPrefix subtree, independent of the tree page, so len(visibleFiles) really is the subtree-wide caller-visible count. The comment is accurate.

On the issue's suspicion that the persisted cursor wasn't being honoured on resume — I don't think that was ever the bug. The traversal checkpoint is a triple (cursor + BootstrapDirectories + BootstrapPageOffset), and an empty page cursor with directories still queued is the normal "finished a directory, starting the next" state. The old error message only printed the page cursor, which is what made it look like state loss. Widening BootstrapStalledError to carry path/page-offset/directories-pending is the correct fix, and it's the change that would have made the original 6.8-day wedge diagnosable.

Non-blocking notes

1. Rollout will hard-stop the live mount immediately. rw_7ccfea89 already has BootstrapStallCycles = 31 ≥ 20 and code: bootstrap_stall_cycle_limit persisted from the old build, so the first cycle after upgrade hits persistedBootstrapTerminalError and the daemon exits — for a mount shared by ~19 agents. That is exactly the requested "fail loudly", but it is a behaviour change on first contact, not on next failure. Please put the re-arm procedure (raise RELAYFILE_BOOTSTRAP_STALL_CYCLES / RELAYFILE_BOOTSTRAP_MAX_DIRECTORIES, or re-bootstrap) in the PR body or release notes. Both knobs are documented in docs/environment-variables.md, which is good; the deploy sequencing is what's missing.

2. Writeback and outbox flush are skipped on a terminal stall. The early return sits at the top of syncReserved, above pushLocal and flushOutboxRecords, so queued local writes never drain before the hard stop. Defensible for a hard stop (the runner is terminating anyway), but a drain-then-stop would lose less agent work. Worth a follow-up rather than a change here.

3. Denominator stays absent once traversal advances past the root. BootstrapFilesTotal is only assigned when currentDirectory == s.remoteRoot. A bootstrap resumed from a checkpoint whose frontier has moved past root (the live mount's /neon/advisors/by-project case) never revisits root and so never acquires a total. The output degrades honestly — "N files synced (total not reported by the saved checkpoint)" rather than a misleading N/0 — so this is an accepted limitation, not a regression. Flagging it because issue ask feat: Nango bridge + Python SDK + SDK improvements #3 is only partly satisfied: for workspaces containing reserved runtime subtrees the denominator is suppressed by design anyway.

4. BootstrapDirectoriesDiscovered can drift upward across resumes. queuedDirectories is seeded only from the pending queue, so an already-processed directory re-encountered after a resume is re-queued and increments the counter again. The monotonic max-seed bounds it and there's ~3x headroom at 14822 real directories vs. the 50000 default, so this is unlikely to trip a false traversal limit — but the counter is not a true distinct-directory count, and the error message presents it as one.

Feature map

.agentworkforce/features/manifest.yaml is not present on main or on either head (only .agentworkforce/trajectories/ and .agentworkforce/workforce/ exist), and it has never existed in this repo's history. The factory featuremap check gate therefore does not apply to this PR.

Not approving or merging — mergePolicy: never for this task, and the branch-reconciliation question above is for a human to settle. The code itself is in good shape.

[cubic-dev-ai]

All reported issues were addressed across 3 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>