Skip to content

JITSU-157 follow-up: don't retry a schema patch the database rejects on its merits - #1442

Open
absorbb wants to merge 1 commit into
newjitsufrom
fix/jitsu-157-patch-retry-scope
Open

JITSU-157 follow-up: don't retry a schema patch the database rejects on its merits#1442
absorbb wants to merge 1 commit into
newjitsufrom
fix/jitsu-157-patch-retry-scope

Conversation

@absorbb

@absorbb absorbb commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #1440, found in production after deploying it.

What I saw

The .*-1lwk destination from JITSU-157 is fine: its last 25P02 was at 13:09:59 — a batch that had been running 7.7 hours (time: 27740.34 s) and predates the deploy. Nothing since. What's left there is the retry-topic backlog draining.

But the new failed to patch table … Getting fresh table schema warn line lit up a different, pre-existing problem on other destinations:

failed to patch table events: report.sql.create_primary_keys: failed to set primary key
statement: ALTER TABLE "public"."events" ADD CONSTRAINT "jitsu_pk_<uuid>" PRIMARY KEY ("eventn_ctx_event_id")
cause: ERROR: could not create unique index "jitsu_pk_<uuid>" (SQLSTATE 23505)

Those tables contain duplicate rows, so the primary key is refused every time — 764 occurrences between Aug 2 and this morning, long before this branch. It's not a race and re-reading the schema cannot help.

Why it matters

ensureTable's retry assumed a failed patch means a stale view of the table. On this class of failure it just runs the same statement again — and ADD CONSTRAINT … PRIMARY KEY builds a full unique index under an ACCESS EXCLUSIVE lock on the customer's table before failing. Measured over 2h on the affected destinations: 394 of these extra doomed index builds, on top of the 260 the failing events caused anyway.

The retry-on-any-patch-error predates #1440 for the cached path; #1440 extended it to the non-cached path and, by logging it, made it visible. Either way it shouldn't be doing this.

The fix

ensureTable now compares what it tried to patch against what the fresh schema says is still missing:

  • identical → our view was never stale, the retry would fail identically → return the original error, no second statement
  • different (the column is there now) → a genuine lost race → retry as before

TestSamePatch covers the decision table; TestConcurrentAddColumn still passes, and forcing samePatch to always return true makes it fail — so the JITSU-157 recovery is still exercised.

Full bulkerlib/implementations/sql suite green against Postgres.

Not fixed here

The underlying condition — destinations whose events table has duplicate values in the primary key column, so bulker retries the constraint forever — is still there. This change stops us from making it worse; it doesn't stop the per-event attempt. Worth its own ticket.

🤖 Generated with Claude Code

Found in production after deploying this branch. The retry after a failed patch
assumed the failure means our view of the table is stale, and re-read the schema
and patched again. That holds for a lost ALTER race, but not for a patch the
database refuses every time: destinations whose events table has duplicate rows
reject ADD CONSTRAINT ... PRIMARY KEY on every batch, and each attempt builds a
full unique index under an ACCESS EXCLUSIVE lock on the customer's table before
failing.

ensureTable now compares what it tried to patch with what the fresh schema says
is still missing. Identical means our view was never stale, so the retry would
run the same statement and fail the same way - report the original error instead.
A lost race looks different (the column is there now) and still retries.

Measured on the affected destinations: ~400 of these doomed index builds every
two hours, on top of the ~260 the failing events caused anyway.

JITSU-157

@jitsu-code-review jitsu-code-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the SQL table patch retry changes in this PR (including retry gating and the new coverage).

I did not find actionable bugs, security issues, or user-visible regressions in the changed code paths. The logic now avoids repeating identical failing patches while still retrying after true schema drift.

Validation: ran under (passes).

@jitsu-code-review jitsu-code-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the SQL table patch retry changes in this PR (ensureTable retry gating and new samePatch coverage).

I did not find actionable bugs, security issues, or user-visible regressions in the changed code paths. The logic now avoids repeating identical failing patches while still retrying after true schema drift.

Validation: ran go test ./bulkerlib/implementations/sql -run TestSamePatch -count=1 from bulker/ (passes).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant