Skip to content

fix(payouts): durable transfer intent — stop the direct-Connect double-pay (#10) - #73

Open
keithfawcett wants to merge 2 commits into
mainfrom
fix/payout-transfer-intent
Open

fix(payouts): durable transfer intent — stop the direct-Connect double-pay (#10)#73
keithfawcett wants to merge 2 commits into
mainfrom
fix/payout-transfer-intent

Conversation

@keithfawcett

Copy link
Copy Markdown
Contributor

Closes the last money-path item from the Aug 2026 payment audit (#10 in docs/audit-remaining-work.md).

The bug

runPayouts called stripe.transfers.create inside the caller's tenant transaction, with an idempotency key derived from a payout id minted in that same transaction. Two double-pay paths:

  1. The transfer succeeds, the COMMIT then fails. Payout row + status='paid' commissions roll back; the money already left Stripe. The next run regroups the same commissions under a new payout id → new idempotency key → duplicate transfer.
  2. Ambiguous Stripe error (network/timeout — the transfer may or may not exist). The catch marked the payout failed and left the commissions approved; the next run retried with a new key → duplicate.

This is a live path — self-host Connect payouts and the deliberate OPENPARTNER_ALLOW_UNFUNDED_CONNECT_PAYOUTS=1 override. It is not behind HOSTED_FUNDING_ENABLED.

The tempting shortcut — a deterministic key over the commission set — was rejected: if a commission is approved between the failed attempt and the retry, the set changes, so does the key, and the second transfer pays the overlap again.

The fix

Mirrors funding/executor.ts, which already had the right shape.

payouts.ts is now a planner. Inside the caller's transaction it writes the Payout row as an intent (metadata.transferState='intent', frozen amountMinor + destination) and freezes its commission set by claiming the rows — Commission.payoutId stamped while status stays approved. Every planner lookup filters payoutId is null, so claimed commissions are invisible to the next run and can never be regrouped into a second, larger transfer. Commissions approved later simply form their own intent. No Stripe call happens inside a transaction.

payout-transfers.ts is the executor. Outside any transaction, on the privileged pool:

intent ──preflight ok──▶ posted ──success──▶ confirmed      (paid; commissions paid)
   ▲                       ├──definite 4xx──▶ canceled      (failed; claims released)
   │                       ├──ambiguous─────▶ stays posted  (retry replays the frozen key)
   │                       └──>24h──────────▶ reconcile_required
   └──────────listing proves no transfer─────────┘
  • Inside Stripe's ~24h window a retry re-POSTs the frozen key payout_<payoutId> — Stripe replays the original outcome. A 60s cooldown keeps a scheduler tick from racing an admin run into two concurrent POSTs.
  • Past the window the key may be pruned, so ambiguity is resolved by paging transfers.list({ transfer_group: payoutId }) for our openpartner_payout_id stamp — never a blind re-POST. Proven absent → re-armed as intent.
  • Preflight (before the first POST, while abandoning is still free) re-checks the claimed set is still approved, still sums to the frozen amount, and the partner is still Connect-ready; drift cancels the intent and releases the claims.
  • A transfer that comes back reversed is never recorded as paid.
  • Every transition is a compare-and-set on transferState, so two workers racing one intent can't both act.

Wiring: new payout-transfers scheduler job (*/15) retries + reconciles open intents; the weekly payouts job executes right after every planning transaction commits; POST /payouts/run plans in a transaction of its own, commits, then executes scoped to that tenant so the admin still sees the outcome (it deliberately no longer borrows the request transaction). withTenantTransaction in tenancy.ts is now the single way to open a tenant-scoped transaction.

Also: the transfer amount is derived from the rows actually claimed rather than a separately-snapshotted SUM (they can disagree under READ COMMITTED).

Tests

16 new DB-backed tests with a hand-rolled Stripe mock (payout-transfer-intent.test.ts), covering: intent freeze with no Stripe call · a second planning run can't regroup frozen commissions · the set-change scenario paying 80 + 25 and never the overlap · happy path · idempotent re-run · commit-failed-after-transfer replays the same key · ambiguous error holds the claim · past-window reconcile by listing (no re-POST) · reconcile-proves-absence re-arms · definite 4xx releases the claim · reversed commission cancels before Stripe · unready partner cancels · two executors race to exactly one transfer · reversed transfer never paid · tenant scoping · manual rail unchanged.

Full suite: 262 passing. pnpm typecheck + pnpm lint clean.

Before this pays anyone real money

docs/direct-connect-payouts.md has the operator view: the state machine, how to inspect open intents, the recovery SQL, and a 6-step Stripe test-mode staging checklist (injected commit failure, injected timeout, past-window reconcile, set change, definite failure). No migration — the intent lives in the existing Payout.metadata jsonb, so nothing to run against prod.

🤖 Generated with Claude Code

…e-pay (#10)

runPayouts called stripe.transfers.create INSIDE the caller's tenant
transaction, keyed on a payout id minted in that same transaction. Two
double-pay paths followed:

  1. the transfer succeeds and the COMMIT then fails — the Payout row and
     the paid commissions roll back, the money is gone, and the next run
     regroups under a NEW payout id, hence a NEW idempotency key, hence a
     second transfer;
  2. an ambiguous Stripe error (timeout — the transfer may or may not
     exist) marked the payout failed, left the commissions approved, and
     the next run retried under a new key.

This is a live path: self-host Connect payouts and the deliberate
OPENPARTNER_ALLOW_UNFUNDED_CONNECT_PAYOUTS=1 override. It is not behind
the funding flag.

A deterministic key over the commission set does NOT fix it — if any
commission is approved between attempts the set changes, the key changes,
and the second transfer pays the overlap again.

So mirror funding/executor.ts, which already had the right shape:

- payouts.ts is now a PLANNER. It writes the Payout row as an intent
  (metadata.transferState='intent', frozen amountMinor + destination) and
  freezes its commission set by claiming the rows — Commission.payoutId
  stamped while status stays 'approved'. Every planner lookup filters
  payoutId is null, so claimed commissions can never be regrouped into a
  second, larger transfer. No Stripe call happens inside the transaction.
- payout-transfers.ts is the EXECUTOR. Outside any transaction, on the
  privileged pool: preflight (set unchanged? partner still ready?), CAS
  intent→posted, POST with the durable key payout_<payoutId>, finalize in
  a short transaction, webhooks strictly after the commit.
- Ambiguous outcomes stay 'posted': inside Stripe's ~24h window a retry
  replays the frozen key; past it the intent goes reconcile_required and
  is resolved by paging transfers.list({transfer_group}) for our metadata
  stamp — never a blind re-POST. Proven absent → re-armed as 'intent'.
- Definite 4xx fails the payout and releases the claims so the next run
  regroups. A transfer that comes back reversed is never recorded paid.

Also: the amount now comes from the rows actually claimed rather than a
separately-snapshotted SUM, and withTenantTransaction (tenancy.ts) is the
one way to open a tenant-scoped transaction — the scheduler and the admin
POST /payouts/run both use it, the latter committing its intents before
any money moves instead of borrowing the request transaction.

New scheduler job payout-transfers (*/15) retries and reconciles intents
left open by a crash, a timeout, or an unready partner.

16 DB-backed tests with a Stripe mock cover every path above, including
the commit-fails-after-transfer and set-change scenarios. Operator doc:
docs/direct-connect-payouts.md (state machine, recovery SQL, and the
staging checklist to run in Stripe test mode before this pays anyone).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
keithfawcett added a commit that referenced this pull request Aug 9, 2026
Turns the handoff brief into a status record: what each item was, what
actually shipped, and — the part that is still open — the staging
exercises that have to pass before either money path is trusted.

Item A (#10, PR #73): planner/executor split with a durable payout intent
and a frozen commission set. Item B (#12, PR #75): the three funding
races plus a live-Stripe backstop for missed refund/reversal webhooks.
Item C (#8, PR #74): the three missing tables, per-table primary keys, a
portable SQL dump, and two array round-trip bugs the test found.

No code left on any of the three; the remaining work is the staging
checklists in docs/direct-connect-payouts.md and section H of the funding
staging runbook, plus the two post-merge prod actions for #62 and #63.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…, missing interlock

Codex review of #73 refuted both claims the PR made. Three real defects,
all of them ways the "cannot double-pay" property failed.

1. A 409 IDEMPOTENCY CONFLICT WAS TREATED AS PROOF OF ABSENCE. Every 4xx
   classified as definite → the intent was failed and its commissions
   released. But 409 `idempotency_key_in_use` means another request is
   using this key RIGHT NOW and may well succeed; releasing lets the
   planner regroup those commissions under a NEW key while the first
   transfer lands. That is exactly the double-pay this PR exists to stop.
   429 had the same problem (no transfer, but the intent was destroyed).
   Both are now ambiguous: the intent stays posted and the retry replays
   the frozen key or reconciles by listing.
   The funding executor never released on these; the divergence was mine.

2. TWO WORKERS COULD RE-POST ONE INTENT CONCURRENTLY. A `posted` intent
   past the 60s cooldown fell through to transfers.create with no claim,
   and the admin route, the weekly payouts job and the 15-minute executor
   job hold DIFFERENT locks, so they can overlap. The retry now takes a
   lease by swapping the exact `postedAt` it read — `posted → posted`
   matches for everyone, but only one worker can swap a given timestamp.

3. A RETRIED KEY REPLAYS A STALE OBJECT. Stripe answers a repeated
   idempotency key with the response it stored at creation, so `reversed`
   in that body is false even if the transfer has since been clawed back
   — and finalizing on it overwrote a reversal webhook's `failed` with
   `paid`, marking commissions paid on money that came back. Any attempt
   past the first now re-reads the transfer from Stripe before believing
   it, and a re-read that fails records nothing.

Also: the frozen commission set had no protection from the OTHER side.
`interlockCommissionReversal` only knew about funding allocations, so the
admin reverse endpoint and the refund clawback would flip a commission
claimed by a posted intent — Stripe still gets the frozen amount while
fewer commissions are marked paid. It now holds commissions claimed by an
open Payout intent too, which gives every reversal path the guard for
free. (Preflight only helps before the first POST.)

The PR also claimed money can never be stranded, which was overstated: a
reversed transfer deliberately leaves its commissions claimed with no
automatic path back. That case now has a documented operator disposition
rather than an implied one.

7 new tests: 409 holds the claim, 429 holds the claim, two workers retry
→ one POST, stale replay is re-read and never resurrects a reversal,
interlock holds a claimed commission, and a canceled intent frees it again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@keithfawcett

Copy link
Copy Markdown
Contributor Author

Adversarial review pass (Codex, xhigh) — three real double-pay paths fixed in a38732e

Both claims in the PR body were refuted. Fixed:

  1. A 409 idempotency conflict was treated as proof of absence (critical). Every 4xx was "definite" → fail the intent, release the claims. But 409 idempotency_key_in_use means another request holds the key right now and may still succeed; releasing lets the planner regroup those commissions under a new key while the first transfer lands — exactly the double-pay this PR exists to prevent. 429 had the same problem. Both are ambiguous now. (The funding executor never released on these; the divergence was mine.)

  2. Two workers could re-POST one intent concurrently (critical). A posted intent past the 60s cooldown fell through to transfers.create with no claim — and the admin route, the weekly payouts job and the 15-minute payout-transfers job hold different advisory locks, so they genuinely overlap. The retry now takes a lease by swapping the exact postedAt it read: posted → posted matches for every worker, but only one can swap a given timestamp.

  3. A retried key replays a stale object (critical). Stripe answers a repeated idempotency key with the response stored at creation, so reversed is false there even if the transfer has since been clawed back — and finalizing on it overwrote a reversal webhook's failed with paid, marking commissions paid on money that came back. Attempts past the first now re-read the transfer from Stripe; a re-read that fails records nothing.

Plus a gap on the other side: the frozen set had no protection from reversal. interlockCommissionReversal only knew about funding allocations, so /commissions/:id/reverse and the refund clawback would flip a commission claimed by a posted intent — Stripe still gets the frozen amount while fewer commissions are marked paid. It now holds commissions claimed by an open Payout intent too, so every reversal path gets the guard. Preflight only ever helped before the first POST.

Claim correction: the PR said money can never be stranded. Not true as written — a reversed transfer deliberately leaves its commissions claimed with no automatic path back. docs/direct-connect-payouts.md now documents the two operator dispositions instead of implying one exists.

7 new tests: 409 holds the claim · 429 holds the claim · two workers retrying → exactly one POST · stale replay re-read and never resurrecting a reversal · interlock holds a claimed commission · a canceled intent frees it again. Suite: 269 passing.

Known, pre-existing, not fixed here

  • Zero-decimal currencies. Math.round(amount * 100) is wrong for JPY — a ¥50 commission sends ¥5,000. Identical on main and in funding/state.ts:toMinor, so it's a platform-wide assumption rather than a regression, but it is real and worth its own PR.
  • transfer.updated ignores amount_reversed (routes/stripe-webhook.ts:386-395), so a partial reversal can flip a payout back to paid. Pre-existing.
  • Pool pressure: /payouts/run holds the request transaction while opening a second one from a pool of 10. Admin-only, low, but it would show up under concurrent clicks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant