Skip to content

[factory] Readiness-labeled issues are silently lost when Factory restarts (event watermark is the only dispatch path) #240

Description

@khaliqgant

Summary

Factory silently loses every readiness-labeled issue that is labeled while it is down, restarting, or mid-drain. The work is not queued, not retried, and not logged as skipped — it is simply never dispatched. On 2026-08-14 this produced a 4h47m window in which Factory was alive, ingesting, and logging as healthy while dispatching nothing.

Evidence

Factory resumes its live subscription from an event high-watermark:

[factory] live subscription starting {"transport":"subscribe-and-poll","highWatermark":"evt_1391663","replaySkewMarginMs":60000,"highWatermarkRouteUnavailable":false}

Five issues were labeled factory between 2026-08-14T06:11:27Z and 06:44:40Z. The Factory process alive at that time consumed those events and then died. Its successor resumed above that watermark, so the events were never re-delivered:

Issue Labeled Outcome
relay#1506 06:10:38Z Dispatch comment posted 06:11:37Z naming ar-1506-impl-relay; no branch, no PR, no live agent. Stuck factory:in-progress for 6h.
cloud#3021 06:10:48Z Same shape, 06:12:36Z.
relaycast#327 06:11:27Z Never dispatched. Zero comments.
relay#1507 06:44:40Z Never dispatched. Zero comments.
relaycast-cloud#61 06:44:26Z Never dispatched.

Over the following 4h47m the process logged 266 relayfile listTree ingest cycles, 20 babysitter PR-snapshot reads, and zero dispatches. No skipped dispatch line was emitted for any of the five — they were invisible to the dispatch path, not rejected by it.

Reproduction

Confirmed by re-emitting the label event above the watermark:

gh issue edit 327 --repo AgentWorkforce/relaycast --remove-label factory
gh issue edit 327 --repo AgentWorkforce/relaycast --add-label factory

relaycast#327 dispatched 75 seconds later with a live ar-327-impl-relaycast process. The same trigger applied to the other four dispatched all of them within 220s. Nothing else changed about those issues — same labels, same titles, same config. The only variable was the event's position relative to the watermark.

Two distinct defects

1. The watermark is the only path to dispatch.
It is an optimization for liveness, but nothing reconciles against it. Any issue whose readiness event lands below the resumed watermark is unreachable forever.

Asked-for behaviour: on startup, and on a periodic sweep, reconcile open issues carrying the readiness label against dispatch state directly from the issue source, independent of the event watermark. An issue that is open, carries safety.requireLabel, matches safety.requireTitlePrefix, and has no active claim must be dispatchable regardless of when it was labeled.

2. An orphaned claim is never reaped.
factory:in-progress is the claim and it fails closed. When the process holding it dies, the label survives with no live agent, and the issue can never be re-offered — it is strictly worse than untouched. relay#1506 and cloud#3021 sat in this state for 6h and I had to clear them by hand.

There is already an orphan-reaper code path — the log shows persisted dispatch-failed agents for orphan reaper with agent names — but it did not fire for these. Determine why, and make claim reaping robust to the claiming process dying: a claim whose agents are not live must be released.

Definition of done

  • An issue labeled while Factory is stopped is dispatched after Factory restarts, without any manual label re-emission. Cover this with a test.
  • A claim held by a dead process is released and the issue re-dispatched.
  • If an issue carrying the readiness label is deliberately not dispatched, that decision is logged with a reason. Silence is the failure mode this issue is about — do not replace an invisible skip with a quieter one.
  • Do not regress the watermark's liveness benefit; the reconcile sweep is additive.

Constraints

  • Branch from origin/main. The local checkout sits on codex/222-routed-pr-babysitter-v2 with an unrelated WIP commit (b2ad725) — do not build on it and do not disturb it.
  • mergePolicy: never. Open the PR and stop at review; the merge gate is the principal's.
  • The running Factory daemon executes from the installed npm package (node_modules/@agent-relay/factory), not this checkout, so editing this repo is safe.

Metadata

Metadata

Assignees

No one assigned

    Labels

    factoryFactory automation triggerfactory:in-progressFactory agents are working on this issue.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions