Skip to content

[factory] One failed PR-snapshot read permanently orphans a PR from the babysitter (no retry, no reconcile) #243

Description

@khaliqgant

Summary

A single failed PR-snapshot read at PR-open time permanently orphans that PR from the babysitter. There is no retry, no dead-letter, and no recovery path — and every subsequent event for that PR is silently discarded. Both log lines involved are debug level, so at normal log level the babysitter fails completely silently.

The mechanism

1. PR opens. #handlePrChange (src/orchestrator/factory.ts:10526) is the only path that spawns a babysitter. It reads the PR meta through the relayfile mount:

snapshot = parsePullSnapshot((await this.#mount.readFile(path)).content, parts.number)
} catch (error) {
  this.#logger.debug?.('[factory] babysitter could not read PR snapshot', { path, error: ... })
  return          // ← gives up permanently. no retry, no dead-letter, no queue
}

That read is failing in production — "error":"fetch failed"364 occurrences in the current log, still recurring:

[factory] babysitter could not read PR snapshot {"path":".../relaycast/pulls/329.../meta.json","error":"fetch failed"}
[factory] babysitter could not read PR snapshot {"path":".../relaycast-cloud/pulls/62.../meta.json","error":"fetch failed"}

2. No spawn means no owner. #babysitterOwnerFor (:10074) resolves an owner by scanning this.#babysitterPr for an entry whose ref.agentName matches the PR. No spawn → no entry → undefined.

3. Every later event is dropped. #routeBabysitterEvent (:10050):

const owner = await this.#babysitterOwnerFor(`${event.owner}/${event.repo}`, target.prNumber)
if (!owner) {
  this.#increment('babysitterEventsIgnoredUnownedPr')
  this.#logger.debug?.('[factory] ignored unowned PR event for babysitter routing', {...})
  continue        // ← reviews, comments, CI results, conflict state — all discarded
}

111 occurrences in the current log. Note these are the downstream symptom of step 1, not an independent problem — "unowned" means no babysitter agent was ever registered for that PR, and has nothing to do with PR authorship.

Observed impact

AgentWorkforce/cloud#3024 was opened 2026-08-14T13:41:34Z by app/agent-relay-code (a Factory-created PR, from the cloud#3021 dispatch) and is CONFLICTING / mergeStateStatus: dirty. Factory's log contains zero events for PR 3024. It was never routed, never babysat, and its conflict has gone unattended.

Resolving conflicts is explicitly the babysitter's job — src/config/schema.ts:278 describes it as continuing to work a PR "addressing review comments, resolving conflicts, and fixing CI". The capability exists; the PR simply never reaches it.

Asked-for behaviour

  1. Retry the PR-open snapshot read. A transient mount/fetch failure must not permanently decide that a PR is unshepherded. Retry with backoff, and on exhaustion dead-letter it visibly.
  2. Make orphaned PRs recoverable. An open, Factory-created PR with no babysitter owner should be detectable and adoptable by a periodic reconcile sweep — do not depend solely on catching the PR-open event exactly once. (This is the same shape as the read-path gap in [factory] Readiness-labeled issues are silently lost when Factory restarts (event watermark is the only dispatch path) #240: a one-shot event with no reconciliation.)
  3. Raise the log level. could not read PR snapshot and ignored unowned PR event are both logger.debug?.. A PR silently losing its shepherd is not debug-level information; at minimum the first should be warn, and repeated occurrences should escalate.
  4. Surface the counters. babysitterEventsIgnoredUnownedPr and babysitterFlatEventsUnreadable are already incremented — expose them where an operator will see them.

Definition of done

  • A PR whose open-event snapshot read fails is retried and still gets a babysitter.
  • An already-open Factory PR with no babysitter owner is adopted by a reconcile sweep. Verify against cloud#3024, which is currently in exactly this state.
  • These failures are visible at default log level.
  • Regression test: fail the snapshot read once, assert the babysitter is still spawned; leave a PR ownerless, assert the sweep adopts it.

Constraints

Metadata

Metadata

Assignees

No one assigned

    Labels

    factoryFactory automation triggerfactory:in-progressFactory agents are working on this issue.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions