fix(billing): catch tenant cancellations — ops alerts, end-customer guard, nightly Stripe reconcile - #60
Open
keithfawcett wants to merge 3 commits into
Open
fix(billing): catch tenant cancellations — ops alerts, end-customer guard, nightly Stripe reconcile#60keithfawcett wants to merge 3 commits into
keithfawcett wants to merge 3 commits into
Conversation
…uard, nightly Stripe reconcile A hosted tenant cancelled via the Stripe Customer Portal on Jul 17 and we never noticed: the prod webhook endpoint wasn't subscribed to customer.subscription.updated/deleted (docs listed neither), so the local billing mirror, white-label entitlement, and custom-domain routing all stayed live, and nobody was emailed. Three fixes: - Ops notifications (PLATFORM_OPS_EMAIL) on tenant billing lifecycle: cancellation scheduled/resumed (detected via previous_attributes on subscription.updated), subscription ended, and dunning failures on the tenant's own invoices. Shared sendOpsEmail extracted from brand-review. - Guard the webhook's tenant-billing paths on the event's customer being the tenant's OWN billing customer. Previously a merchant end-customer's subscription.updated/deleted (resolved via the Identity chain) would clobber Tenant.stripeSubscriptionId and could disable white-label. - Nightly billing-subscription-reconcile job (04:25 UTC): polls Stripe for every tenant that locally claims a subscription and heals missed-webhook drift — ended subs clear the mirror + revoke white-label/custom-domain exactly like the deleted webhook, live subs refresh HostedBillingState, newly discovered cancel_at_period_end alerts once (Config-keyed dedupe). Docs: deploy-production.md now lists subscription.updated/deleted as required events for Destination A and explains why they're load-bearing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codex review of PR #60 surfaced five real gaps; all fixed: - resource_missing NEVER heals: a canceled sub stays retrievable forever, so "missing" means the key can't see the id (wrong account / test-mode key). Healing on it would let one config mistake mass-revoke every live tenant's white-label + custom domain. Now reported as an error instead. - Stale-subscription safety: the deleted webhook and the nightly reconcile clear the pointer with ONE conditional UPDATE keyed on the exact sub id the event/poll referred to — a late retry for a since-replaced sub, or a resubscribe racing the poll, matches 0 rows and is skipped. The updated webhook likewise ignores subs that aren't the tenant's current one and never adopts a terminal sub into an empty pointer. Own-billing detection gains a sub-id fallback for tenants with no persisted stripeCustomerId. - Webhook-retry email dedupe: the cancel-notice Config marker now guards the webhook path too (not just the reconcile), deleted retries are silenced by the conditional clear, and dunning alerts dedupe per (invoice, attempt) so redeliveries stay quiet while every new collection attempt still notifies. - cancel_at coverage: cancellations scheduled via a bare cancel_at (dashboard date / Subscription Schedules) now notify, not just the Customer Portal's cancel_at_period_end. - Notify-then-mark: sendOpsEmail reports transport success and the cancel-notice marker is only written after a successful send, so an SMTP outage retries next pass instead of losing the notice forever. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ated guard Open review found a hole in the PR #60 stale-sub guard: it only rejected TERMINAL snapshots when the pointer was null, so a late/resent/out-of-order ACTIVE subscription.updated arriving after subscription.deleted (pointer already cleared) passed both guards and re-persisted the old sub id, resurrecting the canceled subscription and re-applying white-label. Stripe guarantees no event ordering, so this reordering is real. An updated may now touch billing state only when it refers to the tenant's CURRENT subscription (pointer non-null AND equal to sub.id). A null pointer is re-established solely by checkout.session.completed on a real (re)subscribe, so an updated never legitimately adopts one from null. Added a test for the active-after-delete ordering. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Incident
A white-label tenant cancelled their subscription via the Stripe Customer Portal on Jul 17 with cancel-at-period-end. Nobody was notified; the subscription ended Aug 3; the final invoice ($41.06 white-label proration) went into dunning against a detached payment method; and our local state (billing mirror,
Tenant.whiteLabel, custom-domain routing) stayed live because the event never reached us.Root cause: the prod webhook endpoint is not subscribed to
customer.subscription.updated/customer.subscription.deleted— the handlers exist, butdocs/deploy-production.md§6 never listed those events, so the endpoint config (built from that doc) doesn't send them. There was also no notification anywhere in the handlers, and no fallback when a webhook is missed: the nightly entitlement sweep trusts the local mirror the missing webhook was supposed to maintain.Fixes
Ops notifications (
PLATFORM_OPS_EMAIL, same channel as brand review —sendOpsEmailextracted toplatform-ops-mail.ts):subscription.updatedviaprevious_attributes.cancel_at_period_end, so you hear about churn weeks before the sub actually ends, with the customer's cancellation feedback includedEnd-customer guard (latent bug found during the fix): the webhook's
subscription.updated/deletedhandlers ran for any subscription event that resolved to a tenant — including a merchant end-customer's subscription resolved via the Identity chain. A churning end-customer could clobberTenant.stripeSubscriptionIdand disable the tenant's white-label. Both paths (plus the newinvoice.payment_failedpath) now require the event's customer to be the tenant's own billing customer.Nightly
billing-subscription-reconcilejob (04:25 UTC, before the domain jobs): polls Stripe for every tenant that locally claims a subscription and heals missed-webhook drift — an ended sub clears the mirror and revokes white-label + custom-domain routing exactly like thedeletedwebhook (sharedhandleTenantSubscriptionEnded), live subs refreshHostedBillingState, and a newly discoveredcancel_at_period_endalerts once (Config-keyed dedupe). A missed cancellation now self-heals within a day instead of never.Deploy notes
customer.subscription.updated+customer.subscription.deletedto webhook Destination A. Doc updated with a warning that these are load-bearing.Tests
billing-reconcile.test.ts(6 tests): canceled/resource_missing heal, notify-once dedupe, marker reset on resume, mirror refresh, per-tenant error isolation.🤖 Generated with Claude Code