perf(0132): write only newly-discovered assets per reconcile - #155
Merged
karczuRF merged 2 commits intoJul 29, 2026
Merged
Conversation
The live ledger-processor re-emitted the entire ~200k-row asset registry to ClickHouse on every reconcile run — a 9,413x write amplification (1.9B rows/day, 98.6% of all DB writes) billed as ~$337/mo of AWS to Hetzner Lambda egress. Reported by the BE team, confirmed in prod system.part_log. The ReplacingMergeTree dedups server-side, so the bug is pure cost, not correctness — which is why every functional test passed. Capture the surrogate-id watermark (next_id) before a run and write only the assets interned at/after it via write_new_assets; a run that discovers no new assets writes nothing (empty guard the full write_assets lacked). Rows and their deterministic sac_address are identical to before. The full write_assets is kept for the one-shot backfill/discovery/oracle callers. BE's late-SAC concern does not apply here: sac_address_of is a deterministic derivation from the asset identity, complete when interned. Spawns 0133 (egress/write-volume alarm) so a future amplification surfaces on a dashboard, not a bill.
…ciler Code review found a data-loss regression in the first cut. The Reconciler is a warm Arc singleton reused across Lambda invocations, so recapturing the surrogate-id watermark from the in-memory registry each run was unsafe: a run that interned new assets and then failed a later write left next_id advanced but the assets unpersisted, so the redelivered doorbell's next run recaptured a watermark past them and skipped them forever — orphaning the candles that reference their ids. The old full re-emit self-healed this. Track a durable persisted_asset_watermark in ProcessingState that advances only after a successful asset write; a failed run leaves it unmoved so the retry re-writes those assets. Write assets before candles so the dimension row is referentially ahead of the facts. Add an O(1) steady-state guard in write_new_assets (since >= watermark) to avoid scanning the ~200k registry on every idle reconcile. Fix a stale write_asset_metadata doc comment. Regression test assets_from_a_failed_run_are_written_on_the_next_run fails against the pre-fix code and passes now.
karczuRF
deleted the
perf/0132_ledger-processor-full-asset-registry-reemit
branch
July 29, 2026 10:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
200k-row$337/mo** of AWS→Hetzner Lambda egress. Reported by the BE team, confirmed in prodprices.assetsregistry to ClickHouse on every reconcile run — a 9,413× write amplification (1.9B rows/day, 98.6% of all DB writes) billed as **system.part_log. ReplacingMergeTree dedups server-side, so this was pure cost, not correctness — which is why every functional test passed.next_id) before a run and write only the assets interned at/after it viawrite_new_assets; a run that discovers no new assets writes nothing (empty guard the fullwrite_assetslacked). Rows and their deterministicsac_addressare identical to before.write_assetsretained for the one-shot backfill/discovery/oracle callers;prices.assets/asset_metadatasingle-writer split (0067) preserved.sac_address_ofis a deterministic derivation from the asset identity, complete when interned — no side table needed.no new → 0 rows,one new → 1 row); workspace compiles, clippy clean.Post-deploy check:
system.part_logdailyassetswrite volume should drop from ~1.9B to ~thousands/day.