Summary
#spawnAgent (src/orchestrator/factory.ts) has a crash-recovery idempotency
gate: batch.recordPlanned(...) is called with a deterministic
invocationId (from batch.invocationIdFor(record.issue, spec)) before
the actual spawn side effect runs, so that if the owning process crashes
between planning and confirming the spawn, a takeover retries the same
invocation instead of creating a duplicate worker. Later, batch.shouldSpawn(record, invocationId)
short-circuits #spawnAgent to a synthetic success (no fleet.spawn()
call, no record.agents entry) whenever that invocationId already has a
recorded outcome.
This gate has no way to learn that a deliberate release invalidated its
"already dispatched" memory. Concretely, in #ensureBabysitter's
fresh-spawn branch: a babysitter is spawned, then
markRoutedPrBabysitterRunning legitimately fails (claim lost — see the
fix landing in this PR for cleanup of the local bookkeeping keys), the
agent is released via fleet.release(...). A later retry for the same PR
correctly re-enters #ensureBabysitter and re-attempts #spawnAgent, but
batch.shouldSpawn still says "already dispatched" for the same
invocationId, so no real process gets spawned this time — while the rest
of the retry (claim renegotiation, markRoutedPrBabysitterRunning)
proceeds as if a real babysitter is now running. The state store ends up
believing a PR is being babysat when no process actually exists for it.
Where this was found
Found while fixing two review comments on the routed-PR-babysitter PR
(local #babysitterSpawned/#babysitterPr bookkeeping not being cleared
on a lost claim). Verified empirically that:
- The realistic failure modes of
markRoutedPrBabysitterRunning (another
owner adopted the claim, or the lease genuinely expired) do NOT trigger
this — a retry's claimRoutedPrBabysitter call correctly detects the
conflict and suppresses before ever reaching markRoutedPrBabysitterRunning
again, or legitimately mints a fresh claim.
- The scenario that does trigger it is: the original spawn genuinely
succeeded and was later abandoned (claim released for any reason, e.g.
a non-completion exit), and a subsequent retry's #spawnAgent call
reuses the same deterministic invocationId from the first attempt.
Suggested direction (not investigated in depth)
batch's planned/dispatched record for an invocationId should probably be
invalidated (or the "shouldSpawn" check should also consider) when the
corresponding agent has since been explicitly released — not just left to
imply "crash, always retry as-is." This likely needs a look at
recordPlanned/recordSpawn/shouldSpawn and whatever persists them via
#saveDispatchLifecycle, and isn't scoped to the babysitter code path —
any agent role that gets spawned, released, and retried for the same
deterministic invocation could hit the same gap.
Scope note
This is deliberately filed as its own issue rather than folded into the
in-flight PR fixing the two narrower review comments above — it's a
different, deeper subsystem (crash-recovery dispatch idempotency) that
deserves its own investigation rather than a quick patch alongside
unrelated fixes.
Summary
#spawnAgent(src/orchestrator/factory.ts) has a crash-recovery idempotencygate:
batch.recordPlanned(...)is called with a deterministicinvocationId(frombatch.invocationIdFor(record.issue, spec)) beforethe actual spawn side effect runs, so that if the owning process crashes
between planning and confirming the spawn, a takeover retries the same
invocation instead of creating a duplicate worker. Later,
batch.shouldSpawn(record, invocationId)short-circuits
#spawnAgentto a synthetic success (nofleet.spawn()call, no
record.agentsentry) whenever that invocationId already has arecorded outcome.
This gate has no way to learn that a deliberate release invalidated its
"already dispatched" memory. Concretely, in
#ensureBabysitter'sfresh-spawn branch: a babysitter is spawned, then
markRoutedPrBabysitterRunninglegitimately fails (claim lost — see thefix landing in this PR for cleanup of the local bookkeeping keys), the
agent is released via
fleet.release(...). A later retry for the same PRcorrectly re-enters
#ensureBabysitterand re-attempts#spawnAgent, butbatch.shouldSpawnstill says "already dispatched" for the sameinvocationId, so no real process gets spawned this time — while the rest
of the retry (claim renegotiation,
markRoutedPrBabysitterRunning)proceeds as if a real babysitter is now running. The state store ends up
believing a PR is being babysat when no process actually exists for it.
Where this was found
Found while fixing two review comments on the routed-PR-babysitter PR
(local
#babysitterSpawned/#babysitterPrbookkeeping not being clearedon a lost claim). Verified empirically that:
markRoutedPrBabysitterRunning(anotherowner adopted the claim, or the lease genuinely expired) do NOT trigger
this — a retry's
claimRoutedPrBabysittercall correctly detects theconflict and suppresses before ever reaching
markRoutedPrBabysitterRunningagain, or legitimately mints a fresh claim.
succeeded and was later abandoned (claim released for any reason, e.g.
a non-completion exit), and a subsequent retry's
#spawnAgentcallreuses the same deterministic
invocationIdfrom the first attempt.Suggested direction (not investigated in depth)
batch's planned/dispatched record for an invocationId should probably beinvalidated (or the "shouldSpawn" check should also consider) when the
corresponding agent has since been explicitly released — not just left to
imply "crash, always retry as-is." This likely needs a look at
recordPlanned/recordSpawn/shouldSpawnand whatever persists them via#saveDispatchLifecycle, and isn't scoped to the babysitter code path —any agent role that gets spawned, released, and retried for the same
deterministic invocation could hit the same gap.
Scope note
This is deliberately filed as its own issue rather than folded into the
in-flight PR fixing the two narrower review comments above — it's a
different, deeper subsystem (crash-recovery dispatch idempotency) that
deserves its own investigation rather than a quick patch alongside
unrelated fixes.