[factory] Dispatch is not atomic — a crash mid-dispatch leaves a running agent with no reviewer and no claim

[khaliqgant]

This issue was rewritten on 2026-08-15. The original description claimed dispatch state was "dropped before enqueue". That diagnosis was wrong. The corrected mechanism is below. If you began work against the original text, re-scope against this.

Summary

Dispatch is not atomic. A dispatch consists of at least three effects — spawn the implementer, spawn the reviewer, and record the claim on GitHub (the factory:in-progress label plus the dispatch comment). If the Factory process dies partway through, the dispatch is left half-committed: the implementer runs, the reviewer never spawns, and GitHub shows the issue as untouched.

A repair path exists, but it only runs at startup. So a half-committed dispatch stays broken for as long as Factory keeps running — indefinitely.

Evidence

Six issues were dispatched on 2026-08-14. They split cleanly by which Factory instance dispatched them:

Issue	Dispatched	Claim written to GitHub	Latency
factory#242	15:33:12Z	15:33:25Z	13s
factory#243	15:35:33Z	15:36:33Z	~60s
factory#244	~15:39Z	15:39:19Z	seconds
factory#240	~14:01Z	21:47:02Z	7h 45m
factory#241	~14:01Z	21:46:14Z	7h 45m
relayfile#419	~14:01Z	21:45:29Z	7h 44m

The discriminator is not load, repo, or issue shape. The first group was dispatched by a healthy instance and wrote back in seconds. The second group was dispatched at ~14:01Z by an instance that subsequently died. Their claims were never written.

For those 7¾ hours, all three were half-committed:

• implementer running (ar-240-impl-factory et al., verified via ps)
• no reviewer — ar-240-review-factory, ar-241-review-factory, ar-419-review-relayfile did not exist
• no label, no dispatch comment on GitHub

At 21:45Z the daemon was upgraded 0.1.20 → 0.1.58 and restarted. Within ~2 minutes all three claims were written and all three missing reviewers spawned. The same restart also adopted two entirely unrelated stale issues (cloud#2185, cloud#2872), taking the fleet from 15 to 23 agents.

So the startup reconcile in 0.1.58 does repair this. Nothing repairs it while Factory is running.

Why the original diagnosis was wrong

relayfile writeback status reported pending: 0, failed: 0, dead-lettered: 0, which I read as "the write was dropped before enqueue". It was not: the write never reached relayfile's queue because it died inside Factory, between effects. Anyone debugging this should not expect evidence in relayfile's writeback subsystem.

Why it matters

The factory:in-progress label is the dispatch claim, and it fails closed. A half-committed dispatch is the worst of both states: an agent is consuming resources and editing a repo, while every observer — an operator, a human reading the issue, any tooling keyed on the label — sees unclaimed work. Re-triggering it duplicates the dispatch onto work already in flight.

During this incident that ambiguity produced a chain of confident wrong conclusions: that intake was stalled, that the event pipeline had stopped, and that creating an issue with the label pre-applied does not dispatch. All three were false, and all three followed from trusting GitHub state while agents were already running.

Asked-for behaviour

1. Make dispatch atomic, or explicitly recoverable. Either record a durable intent before spawning anything and drive it to completion, or treat a partial dispatch as a first-class state that is detected and repaired. A crash mid-dispatch must not leave a running agent with no claim.
2. Do not depend on a restart to repair it. The 0.1.58 startup reconcile is the right idea firing at the wrong time. A half-committed dispatch must be detected on a periodic sweep while the daemon runs.
3. Reconcile must cover all three effects, not just the label — a dispatch missing only its reviewer is still broken, and that is exactly what happened here for 7¾ hours.
4. Surface it. A dispatch that could not complete its claim must produce an error-level log naming the issue and the missing effect.

Verification — required, against the built CLI

Do not close this on green unit tests. Build the repo and exercise the real CLI.

npm ci && npm run build
node bin/factory.mjs --help                       # sanity
node bin/factory.mjs dispatch <ISSUE_KEY> --config <path>
node bin/factory.mjs status --config <path>

Prove all of the following, with real terminal output:

• Kill Factory mid-dispatch (after implementer spawn, before the claim is written). On restart and on a running-daemon sweep, the claim is completed and the reviewer is spawned.
• A dispatch whose GitHub write fails is retried and surfaced, not silently abandoned.
• factory status reports in-flight work accurately.

Note on factory status: it currently returns "inFlight": [] while 23 agents are running. I have not established whether that is a reporting defect or simply a fresh CLI process holding no state — separate those two before treating it as a bug.

Context you need

• The daemon in production was @agent-relay/factory@0.1.20 (installed Jul 17) while this repo was 0.1.57. It has since been upgraded to 0.1.58. All symptoms above were observed on 0.1.20. Confirm the defect still reproduces on a build of current main before writing a fix.
• If it does not reproduce, say so and close the PR — that is a successful outcome, not a failure. Do not invent a change to justify the dispatch.
• Branch from origin/main; the local checkout sits on codex/222-routed-pr-babysitter-v2 with unrelated WIP (b2ad725).
• mergePolicy: never — stop at review.
• Related: [factory] Readiness-labeled issues are silently lost when Factory restarts (event watermark is the only dispatch path) #240 (claim durability / readiness reconcile) is the same crash-durability theme on the read path. Coordinate; do not duplicate.

[khaliqgant]

Factory dispatch for 242
Implementers: ar-242-impl-factory
Reviewer: ar-242-review-factory

[khaliqgant]

Re-scope required — the issue description was wrong and has been rewritten

I filed #242 claiming dispatch state was "dropped before enqueue", citing relayfile writeback status showing pending: 0, failed: 0. That diagnosis was wrong. Please re-read #242; the body and title are rewritten.

What actually happens

Dispatch is not atomic. It has at least three effects — spawn implementer, spawn reviewer, write the GitHub claim (factory:in-progress + dispatch comment). If Factory dies partway through, the dispatch is left half-committed.

The discriminator is which instance dispatched, not anything about the issue:

dispatched by	claim written
a healthy instance (#242, #243, #244)	13–60 seconds
an instance that later died (#240, #241, #419)	7h 45m — only after a restart

For 7¾ hours those three had a running implementer, no reviewer at all, and no label or comment on GitHub. The 0.1.58 startup reconcile repaired all three within ~2 minutes of restart, spawning the three missing reviewers.

The write never reached relayfile's queue because it died inside Factory. Do not go looking for evidence in relayfile's writeback subsystem — that was my error.

What this means for your PR

factory#248 was written against the old description. Two things changed:

1. The fix is not "add retry/dead-letter to the writeback path." It is making dispatch atomic or explicitly recoverable — a crash must not leave a running agent with no claim and no reviewer.
2. A repair already exists but fires only at startup. 0.1.58's reconcile is the right idea at the wrong time. The gap is that a half-committed dispatch is never repaired while the daemon keeps running. That is likely the smallest correct fix: run the reconcile on a periodic sweep, and make it cover the reviewer, not just the label.

Also note

The production daemon has been upgraded 0.1.20 → 0.1.58 since #242 was filed. Every symptom in the issue was observed on 0.1.20. Confirm the defect still reproduces on a build of current main before changing code — if it does not, say so and close this PR. That is a successful outcome, not a failure; do not invent a change to justify the dispatch.

Verification requirements are in the rewritten issue. Kill Factory mid-dispatch and prove recovery happens without a restart.