fix(0136): diagnose the frozen coarse OHLCV rollups + recovery runbook by karczuRF · Pull Request #159 · rumblefishdev/stellar-prices-api

karczuRF · 2026-07-30T13:06:21Z

What

Every coarse OHLCV table — price_ohlcv_15m, _1h, _4h, _1d, _1w, _1M — has been frozen
since 2026-07-21 02:44 UTC. Only price_ohlcv_1m is live. Found while verifying the task-0072
MV rollout: change_7d_pct came back 0 for all 3,023 assets, which traced not to the new MV but to
price_ohlcv_1h having no rows in the trailing 7 days.

Nothing alarmed because eight of the nine refreshable MVs report status = Scheduled with an empty
exception every cycle — rolling up stale input is not an error. Only mv_ohlcv_1m_to_15m carries the
failure (TOO_MANY_PARTS, 40,377 occurrences, still firing), and everything downstream starves
quietly.

Docs, a test harness and task records only. No schema, no code, nothing deployed.

Root cause

The leading hypothesis (merges administratively stopped via SYSTEM STOP MERGES) reproduced
perfectly on local ClickHouse 26.3.10.60 — and was then falsified in production. SYSTEM START MERGES prices.price_ohlcv_1M was run against the leaf table and had no effect: parts unchanged at
750, parts_to_do unchanged at 60, part_log empty after six minutes. No state changed (the command
is a no-op where merges were never stopped).

What trace logging then showed is the real finding. Over 30 minutes price_ohlcv_15m emits zero
merge-machinery log lines — only the MV's insert attempt and its TOO_MANY_PARTS failure, once a
minute. The control price_ohlcv_1m logs block allocation and reservations continuously; the pool
logs background merge memory usage dozens of times a minute for other tables; even
price_ohlcv_15m_bak has a live CleanupThread.

So the merge selector is not declining these tables — it is never asked about them. The storage's
background operations assignee is not being scheduled. That is a level below merge policy, which
explains why every knob checked came back innocent, and why START MERGES did nothing: it releases a
lock on a decision, it does not restart a task that is not running.

Recovery — `DETACH`/`ATTACH`, validated locally

The assignee is created in IStorage::startup(), which is why a server restart would reliably
re-establish it. That is ruled out: ch-prod-01 is shared with BE and we are authorised to touch
prices.* only.

DETACH TABLE + ATTACH TABLE re-runs startup() for one table — the per-table equivalent of a
restart, scoped to a single table in a single database. Validated on the prod CH pin
(scripts/test-0136-detach-attach-recovery.sh, CH 26.3.10.60):

Test	Result
T4 — prod-shaped. Table pinned at `parts_to_throw_insert`, inserts failing with the identical `Code: 252`	detach 106 ms / attach 105 ms → parts 30 → 1 within 10 s, INSERT accepted again
T1 — pending mutation. Wedged table carrying an unexecuted `ALTER DELETE`	mutation survives the detach and completes after attach (`is_done` 0 → 1); `MergeParts` + `MutatePart` resume
T3 — dependent MV. `DETACH` of a refreshable MV's `TO` target while the MV is live	allowed; MV errors `Code: 60` during the window only, then self-recovers after `ATTACH`, no recreate
Data integrity	`count() FINAL` unchanged (200 → 200) across detach/attach

Limits of that evidence, stated in the runbook so it is not over-read: the local wedge is built
with SYSTEM STOP MERGES, which production falsified — so this proves the command rebuilds a table's
scheduler, not that it clears prod's specific unknown cause. The mechanism argument is what carries:
ATTACH builds a new storage object from scratch, so no in-memory state survives it. OPTIMIZE as a
by-hand stopgap returned Code: 236 locally, but only because the simulation stops merges — it stays
unvalidated and unused. And the 105 ms attach was 30 parts; _15m has ~5,000 part descriptors to
load, so expect meaningfully slower.

One trap worth flagging: count() is useless as a health probe here. RMT dedup made a target
table read 800 → 400 while its MV was inserting every 5 seconds. Every runbook gate is written
against system.part_log instead.

Not yet run against production. Order when it is: _1M (leaf, off the critical path) with a hard
decision gate → _15m (the blocker) → the other four, one at a time.

Nine hypotheses tested and rejected

Recorded in the task so they are not re-run: the 15m table being dropped; disk (496 GiB free, 28%);
merge pool saturation (68k–285k merges/day cluster-wide); a global merge stop (_1m merges 5,220×/24h);
the mutation pool threshold (legal at 20 of an effective 32, pool sampled idle 18 of 20 times);
write amplification across partitions (all 5,000 parts sit in one); a structural difference between
tables (SHOW CREATE is byte-identical to the healthy _1m); a pending mutation alone (local control
completed in seconds); and finally the stopped-merges flag itself.

Provenance — the 07-17 window was ours

The six price_ohlcv_*_bak tables and the six pending Phoenix deletes look like evidence of an
unexplained third-party maintenance window. They are not: _bak is task 0095 (backup before the
APPEND MV recreate, already tracked by 0105), the deletes are task 0097's preroll-amm-reprice.sql
STAGE 0. SYSTEM STOP MERGES appears nowhere in the repository. The trigger stays unresolved and
does not gate recovery.

Also measured

Frozen merges mean ReplacingMergeTree has collapsed nothing since 07-17: sdex is 6.52× duplicated
in partition 202607 alone, the AMM sources 2–3× table-wide. Not a correctness problem — every
consumer reads with FINAL (prices-api/src/assets/queries_ch.rs:578 and the views.sql surfaces
BE uses), so no wrong number has been served. The cost is ~9.9 M redundant rows in one partition and
ten days of FINAL running over 5,000 unmerged parts on every /ohlcv request and BE query. A
working theory that phoenix specifically was double-counted by the un-applied delete was tested and
rejected — sdex, untouched by 0097, is the most duplicated of all.

0136 — the incident: impact, full evidence chain, nine rejected hypotheses, provenance.
scripts/repro-0136-merge-freeze.sh — local reproduction of the freeze signature (control + test).
scripts/test-0136-detach-attach-recovery.sh — new: local validation of the recovery, four
tests on the prod CH pin, with the limits of the simulation recorded in the header.
docs/runbooks/0136-coarse-rollup-merge-recovery.md — recovery plan, rewritten around
DETACH/ATTACH. Gates run off system.part_log rather than count(); Step 0 additionally
captures count() FINAL baselines and SHOW CREATE for all six tables; a 60 s hang gate with a
stop-and-escalate instruction; the falsified START MERGES attempt moved to an appendix. Flags
that physical row counts will drop while count() FINAL should not, and that the 07-21 gap does
not self-heal — closing it needs a bounded incremental pre-roll, never preroll.sql.
0137 — new: rollup freshness alarm. Health was measured on the MV, not the data.
0105 — blocked: the _bak tables are the only rollback for the recovery.
0072 — rollout status (steps 0–3 done, MV live and healthy), change_7d_pct AC marked blocked.
0135 — second measured failure mode on price_usd (21 assets publishing 0 while vwap_24h knows
the price). Pre-existing, not a 0072 regression.

Staying open

This PR stays open until the recovery has been run and verified in production — the open PR is the
verification-pending signal.

Found while verifying the 0072 MV rollout on ch-prod-01. change_7d_pct came back 0 for all 3,023 assets, which traced not to the new MV but to price_ohlcv_1h having no rows since 2026-07-21. All six coarse tables are frozen; only price_ohlcv_1m is live. Nothing alarmed because eight of the nine refreshable MVs report success every cycle — rolling up stale input is not an error. Only mv_ohlcv_1m_to_15m carries the failure (TOO_MANY_PARTS, 40,377 occurrences, still firing), and everything downstream starves quietly. Six ALTER DELETE mutations from the 0097 Phoenix rework were created 2026-07-17 11:06:52 on exactly the six frozen tables and have never been attempted in 13 days — no failures, empty fail_reason. Partition 202606 of price_ohlcv_15m stopped being touched 15 seconds later, and every active part is level 1: merged once, never again. Six hypotheses tested and rejected, recorded so they are not re-run: the 15m table was not dropped; disk is 28.2% free; merges run 68k-285k times a day cluster-wide; price_ohlcv_1m merged 5,220 times in 24h so nothing is stopped globally; the mutation pool threshold is legal (20 of an effective 32) and 18 of 20 occupancy samples showed an idle pool; and the 5,000 parts sit in one partition, so this is not the 0132 write- amplification shape. Leading hypothesis is that merges and mutations are administratively stopped on these six tables — it fits every observation, including the total silence in every diagnostic, since that state is exposed in no system table for non-replicated MergeTree. It is recorded as UNTESTED: the discriminating check is a state change on a shared production cluster and was deliberately not run. Impact reaches /ohlcv above 1m granularity, BE's 0199 LP-analytics contract, 0072's change_7d_pct, and 0127's M2 backfill-depth gate. The detection gap is the durable lesson and has its own acceptance criterion.

Runs both arms on local docker ClickHouse 26.3.10.60. Control (merges normal, then ALTER DELETE): the mutation completes in seconds — is_done=1, parts_to_do=0, MutatePart events in part_log, rows 800 -> 532. So a pending mutation on its own does not freeze a table, which eliminates the competing explanation. Test (SYSTEM STOP MERGES, then the same ALTER DELETE, then more inserts): is_done=0, parts_to_do=4, latest_fail_time 1970-01-01, empty fail_reason, parts climbing 4 -> 16 while inserts keep succeeding, and part_log showing NewPart only — no MergeParts, no MutatePart. Every observable matches ch-prod-01. Also corrects a wrong reading in the original write-up. A fresh INSERT into a ReplacingMergeTree produces a level-1 part; a plain MergeTree produces level 0. So prod's uniform level 1 across 5,000 parts does not mean "merged exactly once" — it means not one of them has ever been merged, which is stronger evidence for the freeze than what was first recorded. This confirms the mechanism, not prod's state: that the six tables are actually in the stopped state can only be shown by SYSTEM START MERGES against ch-prod-01, which remains deliberately not run.

Provenance correction. The six price_ohlcv_*_bak tables are OURS — task 0095 created them on 2026-07-17 as the restore path before the APPEND rollup-MV recreate, and 0105 already tracks dropping them. The six Phoenix ALTER DELETE mutations are 0097's preroll-amm-reprice.sql STAGE 0, run the same day. Read as an unexplained third-party maintenance window they are misleading evidence, so the task now says so directly. SYSTEM STOP MERGES appears nowhere in the repository — not in preroll-amm-reprice.sql, not in 0095's steps, not in any runbook — and the operator does not recall running it. The trigger stays unresolved and does not gate recovery, since the fix is identical either way. The transferable lesson is narrower: an ad-hoc STOP MERGES is invisible afterwards, so it must be paired with its START in the same runbook step. The runbook probes on price_ohlcv_1M rather than the blocked _15m: it is the leaf of the chain, smallest by parts, and off the critical path, so a wrong hypothesis costs one small table's merges. It carries a hard 5-minute decision gate that stops the operation if nothing happens, and treats the negative result as worth recording. Two things the earlier write-up missed. Resuming merges changes row counts by design — RMT collapses the duplicate-PK rows 0097's idempotent pre-roll wrote, and the pending delete removes phoenix rows in range — so step 0 captures a per-source baseline first. And the 07-21 gap does NOT self-heal: the rollup MVs read a bounded recent window, so closing it needs a bounded incremental pre-roll, never preroll.sql, which expects TRUNCATE-d tables and would re-run the 0090 history-loss incident. 0105 is blocked: the _bak tables are the only rollback for this recovery, and its watch period was never really satisfied since the rollup stalled four days after 0095. Detection is spawned as 0137. Nine days passed silently because a rollup MV that reads stale input still reports success — health was measured on the MV, not on the data.

…eory Frozen merges mean ReplacingMergeTree has collapsed nothing since 07-17, so the coarse tables carry heavy same-key duplication: sdex is 6.52x in partition 202607 alone, phoenix 3.06x, aquarius 2.70x, soroswap 2.07x table-wide. A working hypothesis that phoenix specifically was double-counted — because 0097's STAGE 0 delete never applied — is rejected by the control. sdex was never touched by 0097 and is the most duplicated of all, and the AMM figures are table-wide (diluted by older merged partitions) while sdex's is the frozen partition alone. So this is generic unmerged backlog, not a missing delete. It is also not a correctness problem. Every consumer reads with FINAL — the OHLCV API query at prices-api/src/assets/queries_ch.rs:578 and the views.sql surfaces BE reads in-cluster — so duplicates collapse at read time and no wrong number has been served. The cost is ~9.9M redundant rows in one sdex partition plus nine days of FINAL running over 5,000 unmerged parts on every /ohlcv request and BE query. Recovery is therefore compaction and read-latency recovery, not data correction. Recorded so the runbook's "row counts will change" note is not mistaken for a correctness claim. Backfill state also captured: sdex_archive completed 07-27, soroban_amm is the known-dead leg (no writes since 07-14), so nothing is contending for I/O.

… scheduled Ran the runbook's step-1 probe against ch-prod-01: SYSTEM START MERGES prices.price_ohlcv_1M, the leaf table, smallest blast radius. Six minutes later parts were still 750, parts_to_do still 60, and part_log completely empty. No state changed — the command is a no-op where merges were never stopped — so nothing needed undoing. The local reproduction still holds: SYSTEM STOP MERGES does produce this signature. It is not what is happening here. A matching signature was never proof of a matching cause, which is why the probe went on the leaf table behind an abort gate. text_log is enabled at trace level, and it names the real finding. Over 30 minutes price_ohlcv_15m emits zero merge-machinery lines — only the MV's insert attempt and the TOO_MANY_PARTS failure, once a minute. The control, price_ohlcv_1m, logs block allocation and reservations continuously; the pool logs background merge memory usage dozens of times a minute for other tables; even price_ohlcv_15m_bak has a live CleanupThread. So the selector is not declining these tables, it is never asked about them, which is a level below merge policy and explains why every knob checked came back innocent. It also explains the failed probe: START MERGES releases a lock on a decision, it does not restart a task that is not running. The assignee is initialized at table startup, so the action that reliably re-establishes it is a server restart. That is BE's call on a shared cluster and is not ours to take. An OPTIMIZE would discriminate further but is a state change and was not run. Runbook steps 1-3 marked superseded; steps 0 and 4-7 stand unchanged for once the tables can merge again.

A cluster restart is ruled out — ch-prod-01 is shared with BE and we may touch prices.* only. DETACH + ATTACH re-runs startup() for a single table, rebuilding the background operations assignee that stopped being scheduled. Four tests on local CH 26.3.10.60, all green: T4 table pinned at parts_to_throw_insert, inserts failing with the same Code: 252 as prod -> detach 106ms / attach 105ms -> parts 30 to 1 within 10s, INSERT accepted again T1 a pending ALTER DELETE survives the detach and completes after the attach (is_done 0 to 1); MergeParts + MutatePart resume T3 DETACH of a refreshable MV's TO target is allowed with the MV live; the MV errors Code: 60 only during the window, then self-recovers after ATTACH with no recreate -- count() FINAL unchanged across detach/attach Records two limits in the header so the evidence is not over-read: the local wedge is built with SYSTEM STOP MERGES, which prod falsified, so this proves the command rebuilds a table's scheduler rather than that it clears prod's specific cause; and T2 (OPTIMIZE as a by-hand stopgap) returned Code: 236 only because the simulation stops merges, leaving it unvalidated either way.

Steps 1-3 were built on SYSTEM START MERGES, which production falsified on 07-30. They are rewritten around DETACH + ATTACH, validated locally in the preceding commit, and the falsified attempt moves to an appendix — kept, because it is what forced the trace-logging pass that found the real cause and because it will keep looking like the obvious first move to anyone reading this cold. Gates are rewritten against system.part_log rather than count(). RMT dedup makes count() fall while writes land perfectly — a target table read 800 to 400 during the local test while its MV inserted every 5 seconds. NewPart proves inserts land, MergeParts proves merging resumed. Also: - corrects the mechanism section: the assignee is never scheduled, which is why every merge-policy knob looked innocent - states the cluster restart is out of scope and every command names a single prices table - Step 0 additionally captures count() FINAL baselines and SHOW CREATE for all six tables, saved off-cluster as the ATTACH safety net - adds a 60s hang gate with a stop-and-escalate instruction, a second SSH session as a precondition, DETACH+ATTACH as one invocation, and a warning to never use DETACH ... PERMANENTLY - sharpens the row-count warning: physical rows drop hard, count() FINAL should not move, and the phoenix delete is a partial reduction since ALTER DELETE only touches parts existing when it was created on 07-17 Steps 4-7 are unchanged.

… recovery The task file still recorded a server restart as the fix and BE as the owner of it. The operator ruled that out on 07-31 — the cluster is shared and we are scoped to prices.* — and DETACH/ATTACH was validated on the prod CH pin the same day, so the handover framing was the first thing a future session would have read and the last thing that is still true. - "Handover point" becomes "Recovery — DETACH/ATTACH, validated locally", with the four test results, the limits of the evidence (the local wedge is SYSTEM STOP MERGES, which prod falsified; OPTIMIZE unvalidated; do not quote the 105ms attach for a 5,000-part table), and why it has not been run yet - the falsified stopped-merges section keeps its reproduction but now opens with an explicit do-not-read-this-as-the-cause banner - implementation steps 1-3 rewritten around DETACH/ATTACH: one invocation, a 60s hang gate, plain DETACH never PERMANENTLY, and gates on system.part_log rather than count(), which RMT dedup makes fall while writes land perfectly - the row-count warning is sharpened: physical rows drop hard, count() FINAL should not move, and the phoenix delete is partial since ALTER DELETE only touches parts existing when it was created - root-cause AC restated as mechanism-known/trigger-unknown, plus a second AC for the chosen recovery; both [~], neither runs on prod yet - records the two holds to confirm first — the cleanup rule (still unverified, operator creds expired) and 0105 Task stays in backlog and unactivated; nothing was run against production.

karczuRF added 8 commits July 30, 2026 13:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(0136): diagnose the frozen coarse OHLCV rollups + recovery runbook - #159

fix(0136): diagnose the frozen coarse OHLCV rollups + recovery runbook#159
karczuRF wants to merge 8 commits into
developfrom
fix/0136_coarse-rollup-tables-frozen-since-2026-07-21

karczuRF commented Jul 30, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

karczuRF commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Root cause

Recovery — DETACH/ATTACH, validated locally

Nine hypotheses tested and rejected

Provenance — the 07-17 window was ours

Also measured

Contents

Staying open

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

karczuRF commented Jul 30, 2026 •

edited

Loading

Recovery — `DETACH`/`ATTACH`, validated locally