JITSU-157 follow-up: don't retry a schema patch the database rejects on its merits - #1442
JITSU-157 follow-up: don't retry a schema patch the database rejects on its merits#1442absorbb wants to merge 1 commit into
Conversation
Found in production after deploying this branch. The retry after a failed patch assumed the failure means our view of the table is stale, and re-read the schema and patched again. That holds for a lost ALTER race, but not for a patch the database refuses every time: destinations whose events table has duplicate rows reject ADD CONSTRAINT ... PRIMARY KEY on every batch, and each attempt builds a full unique index under an ACCESS EXCLUSIVE lock on the customer's table before failing. ensureTable now compares what it tried to patch with what the fresh schema says is still missing. Identical means our view was never stale, so the retry would run the same statement and fail the same way - report the original error instead. A lost race looks different (the column is there now) and still retries. Measured on the affected destinations: ~400 of these doomed index builds every two hours, on top of the ~260 the failing events caused anyway. JITSU-157
There was a problem hiding this comment.
Reviewed the SQL table patch retry changes in this PR (including retry gating and the new coverage).
I did not find actionable bugs, security issues, or user-visible regressions in the changed code paths. The logic now avoids repeating identical failing patches while still retrying after true schema drift.
Validation: ran under (passes).
There was a problem hiding this comment.
Reviewed the SQL table patch retry changes in this PR (ensureTable retry gating and new samePatch coverage).
I did not find actionable bugs, security issues, or user-visible regressions in the changed code paths. The logic now avoids repeating identical failing patches while still retrying after true schema drift.
Validation: ran go test ./bulkerlib/implementations/sql -run TestSamePatch -count=1 from bulker/ (passes).
Follow-up to #1440, found in production after deploying it.
What I saw
The
.*-1lwkdestination from JITSU-157 is fine: its last25P02was at 13:09:59 — a batch that had been running 7.7 hours (time: 27740.34 s) and predates the deploy. Nothing since. What's left there is the retry-topic backlog draining.But the new
failed to patch table … Getting fresh table schemawarn line lit up a different, pre-existing problem on other destinations:Those tables contain duplicate rows, so the primary key is refused every time — 764 occurrences between Aug 2 and this morning, long before this branch. It's not a race and re-reading the schema cannot help.
Why it matters
ensureTable's retry assumed a failed patch means a stale view of the table. On this class of failure it just runs the same statement again — andADD CONSTRAINT … PRIMARY KEYbuilds a full unique index under anACCESS EXCLUSIVElock on the customer's table before failing. Measured over 2h on the affected destinations: 394 of these extra doomed index builds, on top of the 260 the failing events caused anyway.The retry-on-any-patch-error predates #1440 for the cached path; #1440 extended it to the non-cached path and, by logging it, made it visible. Either way it shouldn't be doing this.
The fix
ensureTablenow compares what it tried to patch against what the fresh schema says is still missing:TestSamePatchcovers the decision table;TestConcurrentAddColumnstill passes, and forcingsamePatchto always return true makes it fail — so the JITSU-157 recovery is still exercised.Full
bulkerlib/implementations/sqlsuite green against Postgres.Not fixed here
The underlying condition — destinations whose
eventstable has duplicate values in the primary key column, so bulker retries the constraint forever — is still there. This change stops us from making it worse; it doesn't stop the per-event attempt. Worth its own ticket.🤖 Generated with Claude Code