feat(0133): write-amplification probe + alarm (0132 guardrail) - #156
Merged
karczuRF merged 3 commits intoJul 29, 2026
Merged
Conversation
The guardrail for task 0132: nothing watched write volume, so a 9,413x asset re-emit bled ~$337/mo of egress undetected for weeks. This adds an hourly probe + alarm so the next amplification hits Slack, not a bill. New crate write-amplification-probe (mirrors the 0056 probe pattern): a scheduled Lambda reads rows-written-per-hour per prices table from system.part_log (as prices_reader) and republishes the max as the Prices/Ingest MaxRowsWrittenPerHour CloudWatch metric. A CloudWatch alarm fires when it exceeds config.opsAlarms.writeAmplificationRowsPerHour (default 10M — the busiest legit table is <1M/h, 0132 was ~130M/h) → the existing 0056 SNS topic → Slack. Infra: EventBridge rate(1h) rule + createWorkerLambda on the api/reader mTLS bundle + PutMetricData scoped to Prices/Ingest; observability alarm with OkAction; schedule + threshold in types.ts and production.json. Absolute rows/hour rather than a written/real ratio: prices_reader is granted only system.part_log (+ prices.*), not system.parts, and an absolute guardrail with wide margin is sufficient; a true ratio is a documented future enhancement. Deploy prerequisite (one-time, admin): GRANT SELECT ON system.part_log TO prices_reader. Verified: 4 unit tests, clippy clean, arm64 bootstrap builds, infra tsc build + full cdk synth pass; probe/alarm/IAM confirmed in synthesized CFN.
…on alarm Finding 1 (false-positive on legit bulk loads): system.part_log counts ALL writes to prices.* — including the 0088 backfill, coarse pre-rolls, and enrichment bursts — which an absolute row count cannot distinguish from a re-emit amplification. Mitigate: require a sustained 3-hour breach (a 0132 runaway persists for weeks; legit bulk is bursty) and document the threshold as operator-tunable / ack-able during known heavy loads. The threshold still needs empirical validation against real backfill/pre-roll peaks (noted); a written-vs-real ratio is the robust future fix. Finding 2: alarm evaluationPeriods/datapointsToAlarm 1 -> 3 (sustained, not a lone anomalous hour). Finding 3: the 1h window was coupled across the SQL query, the schedule, and the alarm period with no link — add WINDOW_HOURS + explicit coupling notes. Finding 4: remove the unused thiserror dependency (copy-paste). Verified: 4 tests pass, clippy clean (all features), infra tsc build + full cdk synth pass; sustained-breach (eval/datapoints=3) confirmed in the CFN.
…log measurement Finding 1 follow-up. A 14-day system.part_log measurement proved the absolute threshold could not be a guess: a legit one-hour price_ohlcv_15m_bak copy (07-17 rollup rework) wrote 154M rows/hour — HIGHER than the 0132 bug's ~130M — so no absolute value separates legit from amplification. The discriminator is duration: the _bak spike was one hour; the highest sustained legit load is price_ohlcv_1m at ~16M/hour (07-26 backfill day, ~18h); 0132 ran ~130M/hour for days. Set the threshold to 50M (was a guessed 10M): ~3x above the ~16M sustained legit peak, ~2.6x below the ~130M bug. Combined with the existing 3-hour sustained window (which clears the one-hour spikes), no legit event in the measured window breaches it for 3 sustained hours, while a 0132-class runaway still does. Verified: infra tsc build + full cdk synth pass; threshold 50M / eval 3 / datapoints 3 confirmed in the synthesized CFN.
karczuRF
added a commit
that referenced
this pull request
Jul 29, 2026
#157) * Revert "feat(0133): write-amplification probe + alarm (0132 guardrail) (#156)" This reverts commit 5add848. * docs(lore-0133): complete + archive — BE owns the guardrail (shared-infra alarm) After the BE response the approach changed: instead of a prices-owned write-amplification probe, BE will cover this at the shared-infra layer with a transfer-cost alarm they own. That meets the task goal (the next amplification hits an alarm, not a bill) without a prices-owned probe or the system.part_log grant that the prices CH users (XML-managed in BE's users_xml, readonly to SQL) could not receive without a BE services.xml change. PR #156's code is reverted in this branch (unused). Task marked completed and moved to archive with a Resolution note; the design + measured threshold are retained for the record.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#stellar-prices-api-bot), not a bill.write-amplification-probe(mirrors the 0056 probe pattern): a scheduled Lambda reads rows-written-per-hour perprices.*table fromsystem.part_logand republishes the max as thePrices/IngestMaxRowsWrittenPerHourCloudWatch metric.config.opsAlarms.writeAmplificationRowsPerHour→ existing 0056 SNS topic → Slack.OkActionset; missing data non-breaching (probe-down covered by its own-errorsalarm).system.part_logmeasurement showed the highest sustained legit load is ~16M rows/hour (a backfill day), while one-hour bulk spikes reach ~154M but clear within the 3-hour window; 0132 ran ~130M/hour for days. 50M sits ~3× above sustained legit, ~2.6× below the runaway.rate(1h)rule +createWorkerLambda+PutMetricDatascoped toPrices/Ingest; schedule + threshold intypes.ts/envs/production.json.Verification
tscbuild + fullcdk synthpass; probe function, EventBridge rule, and the alarm (threshold 50M, 3-hour sustained, GreaterThanThreshold, notBreaching) all confirmed in the synthesized CloudFormation.