fix(payouts): durable transfer intent — stop the direct-Connect double-pay (#10) - #73
fix(payouts): durable transfer intent — stop the direct-Connect double-pay (#10)#73keithfawcett wants to merge 2 commits into
Conversation
…e-pay (#10) runPayouts called stripe.transfers.create INSIDE the caller's tenant transaction, keyed on a payout id minted in that same transaction. Two double-pay paths followed: 1. the transfer succeeds and the COMMIT then fails — the Payout row and the paid commissions roll back, the money is gone, and the next run regroups under a NEW payout id, hence a NEW idempotency key, hence a second transfer; 2. an ambiguous Stripe error (timeout — the transfer may or may not exist) marked the payout failed, left the commissions approved, and the next run retried under a new key. This is a live path: self-host Connect payouts and the deliberate OPENPARTNER_ALLOW_UNFUNDED_CONNECT_PAYOUTS=1 override. It is not behind the funding flag. A deterministic key over the commission set does NOT fix it — if any commission is approved between attempts the set changes, the key changes, and the second transfer pays the overlap again. So mirror funding/executor.ts, which already had the right shape: - payouts.ts is now a PLANNER. It writes the Payout row as an intent (metadata.transferState='intent', frozen amountMinor + destination) and freezes its commission set by claiming the rows — Commission.payoutId stamped while status stays 'approved'. Every planner lookup filters payoutId is null, so claimed commissions can never be regrouped into a second, larger transfer. No Stripe call happens inside the transaction. - payout-transfers.ts is the EXECUTOR. Outside any transaction, on the privileged pool: preflight (set unchanged? partner still ready?), CAS intent→posted, POST with the durable key payout_<payoutId>, finalize in a short transaction, webhooks strictly after the commit. - Ambiguous outcomes stay 'posted': inside Stripe's ~24h window a retry replays the frozen key; past it the intent goes reconcile_required and is resolved by paging transfers.list({transfer_group}) for our metadata stamp — never a blind re-POST. Proven absent → re-armed as 'intent'. - Definite 4xx fails the payout and releases the claims so the next run regroups. A transfer that comes back reversed is never recorded paid. Also: the amount now comes from the rows actually claimed rather than a separately-snapshotted SUM, and withTenantTransaction (tenancy.ts) is the one way to open a tenant-scoped transaction — the scheduler and the admin POST /payouts/run both use it, the latter committing its intents before any money moves instead of borrowing the request transaction. New scheduler job payout-transfers (*/15) retries and reconciles intents left open by a crash, a timeout, or an unready partner. 16 DB-backed tests with a Stripe mock cover every path above, including the commit-fails-after-transfer and set-change scenarios. Operator doc: docs/direct-connect-payouts.md (state machine, recovery SQL, and the staging checklist to run in Stripe test mode before this pays anyone). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Turns the handoff brief into a status record: what each item was, what actually shipped, and — the part that is still open — the staging exercises that have to pass before either money path is trusted. Item A (#10, PR #73): planner/executor split with a durable payout intent and a frozen commission set. Item B (#12, PR #75): the three funding races plus a live-Stripe backstop for missed refund/reversal webhooks. Item C (#8, PR #74): the three missing tables, per-table primary keys, a portable SQL dump, and two array round-trip bugs the test found. No code left on any of the three; the remaining work is the staging checklists in docs/direct-connect-payouts.md and section H of the funding staging runbook, plus the two post-merge prod actions for #62 and #63. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…, missing interlock Codex review of #73 refuted both claims the PR made. Three real defects, all of them ways the "cannot double-pay" property failed. 1. A 409 IDEMPOTENCY CONFLICT WAS TREATED AS PROOF OF ABSENCE. Every 4xx classified as definite → the intent was failed and its commissions released. But 409 `idempotency_key_in_use` means another request is using this key RIGHT NOW and may well succeed; releasing lets the planner regroup those commissions under a NEW key while the first transfer lands. That is exactly the double-pay this PR exists to stop. 429 had the same problem (no transfer, but the intent was destroyed). Both are now ambiguous: the intent stays posted and the retry replays the frozen key or reconciles by listing. The funding executor never released on these; the divergence was mine. 2. TWO WORKERS COULD RE-POST ONE INTENT CONCURRENTLY. A `posted` intent past the 60s cooldown fell through to transfers.create with no claim, and the admin route, the weekly payouts job and the 15-minute executor job hold DIFFERENT locks, so they can overlap. The retry now takes a lease by swapping the exact `postedAt` it read — `posted → posted` matches for everyone, but only one worker can swap a given timestamp. 3. A RETRIED KEY REPLAYS A STALE OBJECT. Stripe answers a repeated idempotency key with the response it stored at creation, so `reversed` in that body is false even if the transfer has since been clawed back — and finalizing on it overwrote a reversal webhook's `failed` with `paid`, marking commissions paid on money that came back. Any attempt past the first now re-reads the transfer from Stripe before believing it, and a re-read that fails records nothing. Also: the frozen commission set had no protection from the OTHER side. `interlockCommissionReversal` only knew about funding allocations, so the admin reverse endpoint and the refund clawback would flip a commission claimed by a posted intent — Stripe still gets the frozen amount while fewer commissions are marked paid. It now holds commissions claimed by an open Payout intent too, which gives every reversal path the guard for free. (Preflight only helps before the first POST.) The PR also claimed money can never be stranded, which was overstated: a reversed transfer deliberately leaves its commissions claimed with no automatic path back. That case now has a documented operator disposition rather than an implied one. 7 new tests: 409 holds the claim, 429 holds the claim, two workers retry → one POST, stale replay is re-read and never resurrects a reversal, interlock holds a claimed commission, and a canceled intent frees it again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adversarial review pass (Codex, xhigh) — three real double-pay paths fixed in a38732eBoth claims in the PR body were refuted. Fixed:
Plus a gap on the other side: the frozen set had no protection from reversal. Claim correction: the PR said money can never be stranded. Not true as written — a reversed transfer deliberately leaves its commissions claimed with no automatic path back. 7 new tests: 409 holds the claim · 429 holds the claim · two workers retrying → exactly one POST · stale replay re-read and never resurrecting a reversal · interlock holds a claimed commission · a canceled intent frees it again. Suite: 269 passing. Known, pre-existing, not fixed here
|
Closes the last money-path item from the Aug 2026 payment audit (#10 in
docs/audit-remaining-work.md).The bug
runPayoutscalledstripe.transfers.createinside the caller's tenant transaction, with an idempotency key derived from a payout id minted in that same transaction. Two double-pay paths:status='paid'commissions roll back; the money already left Stripe. The next run regroups the same commissions under a new payout id → new idempotency key → duplicate transfer.failedand left the commissionsapproved; the next run retried with a new key → duplicate.This is a live path — self-host Connect payouts and the deliberate
OPENPARTNER_ALLOW_UNFUNDED_CONNECT_PAYOUTS=1override. It is not behindHOSTED_FUNDING_ENABLED.The tempting shortcut — a deterministic key over the commission set — was rejected: if a commission is approved between the failed attempt and the retry, the set changes, so does the key, and the second transfer pays the overlap again.
The fix
Mirrors
funding/executor.ts, which already had the right shape.payouts.tsis now a planner. Inside the caller's transaction it writes thePayoutrow as an intent (metadata.transferState='intent', frozenamountMinor+ destination) and freezes its commission set by claiming the rows —Commission.payoutIdstamped whilestatusstaysapproved. Every planner lookup filterspayoutId is null, so claimed commissions are invisible to the next run and can never be regrouped into a second, larger transfer. Commissions approved later simply form their own intent. No Stripe call happens inside a transaction.payout-transfers.tsis the executor. Outside any transaction, on the privileged pool:payout_<payoutId>— Stripe replays the original outcome. A 60s cooldown keeps a scheduler tick from racing an admin run into two concurrent POSTs.transfers.list({ transfer_group: payoutId })for ouropenpartner_payout_idstamp — never a blind re-POST. Proven absent → re-armed asintent.transferState, so two workers racing one intent can't both act.Wiring: new
payout-transfersscheduler job (*/15) retries + reconciles open intents; the weeklypayoutsjob executes right after every planning transaction commits;POST /payouts/runplans in a transaction of its own, commits, then executes scoped to that tenant so the admin still sees the outcome (it deliberately no longer borrows the request transaction).withTenantTransactionintenancy.tsis now the single way to open a tenant-scoped transaction.Also: the transfer amount is derived from the rows actually claimed rather than a separately-snapshotted
SUM(they can disagree under READ COMMITTED).Tests
16 new DB-backed tests with a hand-rolled Stripe mock (
payout-transfer-intent.test.ts), covering: intent freeze with no Stripe call · a second planning run can't regroup frozen commissions · the set-change scenario paying 80 + 25 and never the overlap · happy path · idempotent re-run · commit-failed-after-transfer replays the same key · ambiguous error holds the claim · past-window reconcile by listing (no re-POST) · reconcile-proves-absence re-arms · definite 4xx releases the claim · reversed commission cancels before Stripe · unready partner cancels · two executors race to exactly one transfer · reversed transfer never paid · tenant scoping · manual rail unchanged.Full suite: 262 passing.
pnpm typecheck+pnpm lintclean.Before this pays anyone real money
docs/direct-connect-payouts.mdhas the operator view: the state machine, how to inspect open intents, the recovery SQL, and a 6-step Stripe test-mode staging checklist (injected commit failure, injected timeout, past-window reconcile, set change, definite failure). No migration — the intent lives in the existingPayout.metadatajsonb, so nothing to run against prod.🤖 Generated with Claude Code