perf(api): remove the Workers KV edge-cache backend - #388
Merged
Conversation
#387 put the org warehouse config on a durable KV tier, on the hypothesis that a KV `get` is a cancellable subrequest and so cheaper to abandon than an uncancellable `cache.match()`. `makeKvBackend` said to treat it as an experiment until `cache.read_status` per `cache.backend` said otherwise. It does. Measured over 24h on the live deploy: - KV reads that COMPLETE take 92ms (vs 6ms on the Cache API), and 79% of them hit their deadline anyway. - It was bound where the cost isn't. 94% of the Postgres fallback it targets is apps/alerting (4,646 resolutions/day at p50 573ms, ~44min of blocked wall time), which has no KV binding and so could never use the tier. apps/api, which had it, falls back at p50 24ms — cheaper than a KV read. - It regressed three unrelated buckets: CacheBackendLive returned KV whenever EDGE_CACHE was bound, so qe-direct, qe-execute and autumn-customer flipped to KV at the 40ms service default — below KV's ~92ms floor, so they time out 100% of the time at span p50 9.0s. The reason this and the earlier Cache API attempt both failed is that the cost is not a cold-isolate miss. Grouping alerting's Postgres resolutions by trace: 1,033 traces do zero, while 106 traces do 22 each — half of all of them. It is an in-request fan-out where every branch misses the memo because none has finished writing it yet. No shared cache fixes concurrent siblings; it just turns N Postgres reads into N cache reads contending for the same six connection slots. The fix is to resolve the config once before the fan-out, the way `warehouse.warmRoute` already does in query-engine.http.ts — follow-up, not in this change. With KV gone the durable tier is dead in prod (durableTierIsInvalidated would admit only "memory"), so it is removed entirely rather than left alive only under test. The memo + Postgres path is unchanged, and it is strictly more correct: no entry can outlive an invalidation.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
🍁 Maple PR previewNote Preview resources were removed when this pull request closed. Final commit |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reverts the KV edge-cache tier from #387. Its own comment said to treat it as an experiment until
cache.read_statuspercache.backendsaid otherwise — it does.Measured (prod traces, 24h to 2026-08-11 12:00 UTC)
KV is slower and mostly times out. On the live deploy, reads that complete take 92ms (vs 6ms on the Cache API), and 31/39 (79%) hit their deadline.
It was bound where the cost isn't. The Postgres fallback it targets, split by service:
94% is in
alerting, which has noEDGE_CACHEbinding and is refused bydurableTierIsInvalidated, so it could never use the tier.maple-api, which had it, falls back at p50 24ms — cheaper than a KV read.It regressed three unrelated buckets.
CacheBackendLivereturned KV wheneverEDGE_CACHEwas bound, so it flipped every bucket, but onlyorg-clickhouse-configgot a KV-appropriate deadline.qe-direct/qe-execute/autumn-customerinherited the 40ms service default — below KV's ~92ms floor — and time out 100% of the time at span p50 9.0s (19 reads, 19 timeouts).Neither cache attempt could have worked. The cost is not a cold-isolate miss. Grouping alerting's Postgres resolutions by trace:
74% of traces do zero — the memo works. Half of all reads come from 106 traces doing 22 each, and those traces resolve config only 2,861 times total. It's an in-request fan-out where every branch misses the memo because none has finished writing it yet — same shape as the Aug 5
BucketCacheServiceherd. A shared cache can't help concurrent siblings; it converts N Postgres reads into N cache reads contending for the same six connection slots, which is exactly the 79% timeout.What changed
CacheBackendLive.ts—makeKvBackend,KvLike,isKvLike,KV_MIN_TTL_SECONDS, theenv.EDGE_CACHEbranch, and the now-unusedWorkerEnvironmentlayer.cache-backend.ts/edge-cache.ts—"workers-kv"off the name union;backendNameoffEdgeCacheServiceShape(its only consumer was the durable-tier guard). Thecache.backendspan attribute is unaffected.OrgClickHouseSettingsService.ts— the durable tier entirely. With KV gonedurableTierIsInvalidatedwould admit only"memory", leaving a path that is dead in prod and live only under test.invalidateOrgRuntimeConfigcollapses into the syncinvalidateOrgRuntimeConfigMemo.EDGE_CACHEKV namespace + worker binding inalchemy.run.ts, and thewrangler.jsoncentry. Alchemy will destroy the namespace on the next deploy — it holds only cache entries.CacheBackendLive.test.ts(tested onlymakeKvBackend); replaced the three durable-tier cases inOrgClickHouseSettingsService.test.tswith two asserting memo-only behaviour.The comment above
resolveCachedSettingsnow records both failed attempts and why, so this isn't tried a third time.Reviewer notes
EDGE_CACHE_READ_TIMEOUT_MSstays — that's the Cache API deadline, unrelated.warmRoute-style prewarm before the per-rule fan-outs inAnomalyDetectionService/AlertsService; those files have unrelated in-flight work right now.Verification
bun typecheck— 37/37.OrgClickHouseSettingsService25/25,lib/cache17/17, oxlint clean on touched files.git grepfinds no remainingEDGE_CACHE/workers-kv/makeKvBackend/backendName.cache.backendonEdgeCacheService.getOrComputeshould show onlyworkers-cache, with no bucket at 100% timeout and read p50 ~6ms.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.