Skip to content

Spawn-idempotency gate can no-op a retry while claim bookkeeping proceeds as if it succeeded #229

Description

@kjgbot

Summary

#spawnAgent (src/orchestrator/factory.ts) has a crash-recovery idempotency
gate: batch.recordPlanned(...) is called with a deterministic
invocationId (from batch.invocationIdFor(record.issue, spec)) before
the actual spawn side effect runs, so that if the owning process crashes
between planning and confirming the spawn, a takeover retries the same
invocation instead of creating a duplicate worker. Later, batch.shouldSpawn(record, invocationId)
short-circuits #spawnAgent to a synthetic success (no fleet.spawn()
call, no record.agents entry) whenever that invocationId already has a
recorded outcome.

This gate has no way to learn that a deliberate release invalidated its
"already dispatched" memory. Concretely, in #ensureBabysitter's
fresh-spawn branch: a babysitter is spawned, then
markRoutedPrBabysitterRunning legitimately fails (claim lost — see the
fix landing in this PR for cleanup of the local bookkeeping keys), the
agent is released via fleet.release(...). A later retry for the same PR
correctly re-enters #ensureBabysitter and re-attempts #spawnAgent, but
batch.shouldSpawn still says "already dispatched" for the same
invocationId, so no real process gets spawned this time — while the rest
of the retry (claim renegotiation, markRoutedPrBabysitterRunning)
proceeds as if a real babysitter is now running. The state store ends up
believing a PR is being babysat when no process actually exists for it.

Where this was found

Found while fixing two review comments on the routed-PR-babysitter PR
(local #babysitterSpawned/#babysitterPr bookkeeping not being cleared
on a lost claim). Verified empirically that:

  • The realistic failure modes of markRoutedPrBabysitterRunning (another
    owner adopted the claim, or the lease genuinely expired) do NOT trigger
    this — a retry's claimRoutedPrBabysitter call correctly detects the
    conflict and suppresses before ever reaching markRoutedPrBabysitterRunning
    again, or legitimately mints a fresh claim.
  • The scenario that does trigger it is: the original spawn genuinely
    succeeded and was later abandoned (claim released for any reason, e.g.
    a non-completion exit), and a subsequent retry's #spawnAgent call
    reuses the same deterministic invocationId from the first attempt.

Suggested direction (not investigated in depth)

batch's planned/dispatched record for an invocationId should probably be
invalidated (or the "shouldSpawn" check should also consider) when the
corresponding agent has since been explicitly released — not just left to
imply "crash, always retry as-is." This likely needs a look at
recordPlanned/recordSpawn/shouldSpawn and whatever persists them via
#saveDispatchLifecycle, and isn't scoped to the babysitter code path —
any agent role that gets spawned, released, and retried for the same
deterministic invocation could hit the same gap.

Scope note

This is deliberately filed as its own issue rather than folded into the
in-flight PR fixing the two narrower review comments above — it's a
different, deeper subsystem (crash-recovery dispatch idempotency) that
deserves its own investigation rather than a quick patch alongside
unrelated fixes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions