Skip to content

perf(api): remove the Workers KV edge-cache backend #388

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Makisuo merged 1 commit into from
Aug 10, 2026
Merged

perf(api): remove the Workers KV edge-cache backend #388

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 0 additions & 8 deletions apps/api/alchemy.run.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,13 +99,6 @@ export const createMapleApi = ({ stage, domains }: CreateMapleApiOptions) =>
title: resolveWorkerName("mcp-sessions", stage),
})

// EdgeCacheService backend. Binding present => KV; drop it to fall back to
// `caches.default` (see CacheBackendLive.ts for the measured reason KV is
// being trialled).
const edgeCache = yield* Cloudflare.KV.Namespace("EDGE_CACHE", {
title: resolveWorkerName("edge-cache", stage),
})

// Long-running schema-apply: chunks heavy backfill migrations across durable
// steps so they never hit the Worker request budget. Class is exported from
// src/worker.ts. The first Workflow arg IS the physical workflow name; the
Expand Down Expand Up @@ -203,7 +196,6 @@ export const createMapleApi = ({ stage, domains }: CreateMapleApiOptions) =>
AI: Cloudflare.AI.Gateway("maple-api-ai"),
CHAT_SESSION: chatSession,
MCP_SESSIONS: mcpSessions,
EDGE_CACHE: edgeCache,
// Read side of the replay payload store; absent bindings degrade to
// inline-only hydration (see platform/ReplayBlobStore.ts).
REPLAY_BLOBS: replayBlobs,
Expand Down
101 changes: 0 additions & 101 deletions apps/api/src/platform/CacheBackendLive.test.ts
View file
Open in desktop

This file was deleted.

99 changes: 7 additions & 92 deletions apps/api/src/platform/CacheBackendLive.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
import { Effect, Layer, Metric } from "effect"
import { WorkersCache } from "@maple/effect-cloudflare/workers-cache"
import { WorkerEnvironment } from "@maple/effect-cloudflare/worker-environment"
import { CacheBackend, type EdgeCacheBackend, makeMemoryBackend } from "@maple/cache"
import * as QueryEngineMetrics from "@/observability/QueryEngineMetrics"

Expand Down Expand Up @@ -46,100 +45,16 @@ const makeWorkersBackend = (cache: Cache): EdgeCacheBackend => ({
})

/**
* Minimal structural view of a Workers KV binding — enough for the three
* `EdgeCacheBackend` operations. Duck-typed rather than importing
* `@cloudflare/workers-types`, matching how `worker.ts` reads `MCP_SESSIONS`.
* Workers KV was trialled here as a second backend (#387, 2026-08-10) on the
* theory that a KV `get` is a cancellable subrequest and so cheaper to abandon
* than an uncancellable `cache.match()`. Prod measurement refuted it and it was
* removed — see the note above `resolveCachedSettings` in
* `OrgClickHouseSettingsService.ts` for the numbers. Don't reach for it again
* without re-reading them.
*/
interface KvLike {
get(key: string, type: "json"): Promise<unknown | null>
put(key: string, value: string, options?: { expirationTtl?: number }): Promise<void>
delete(key: string): Promise<void>
}

const isKvLike = (candidate: unknown): candidate is KvLike =>
typeof candidate === "object" &&
candidate !== null &&
"get" in candidate &&
"put" in candidate &&
"delete" in candidate

// KV rejects an `expirationTtl` below 60s.
const KV_MIN_TTL_SECONDS = 60

/**
* Workers KV as the edge-cache backend, trialled against `caches.default`.
*
* The Cache API is not slow — measured on 2026-08-10, a completed `cache.match()`
* costs a p50 of 7ms. The problem is that 15% of reads never complete: a
* `cache.match()` holds one of the Worker's six simultaneous-connection slots
* while it waits for response headers, and it is **not cancellable**, so the
* 40ms deadline abandons the wait but not the slot — `compute` then opens a
* seventh connection and queues behind the read it just gave up on (p95 16.4s,
* 1003s of blocked wall time in one day). See `DEFAULT_EDGE_CACHE_READ_TIMEOUT_MS`.
*
* KV is worth trying because a `get` is an ordinary subrequest rather than an
* uncancellable cache read, so abandoning one is closer to actually free. That
* is the hypothesis, NOT a measured fact — if KV turns out to contend for the
* same six slots it will show the same bimodal split. `cache.backend` is on
* every `getOrCompute` span precisely so the two can be compared; check the
* `cache.read_status` distribution per backend before trusting this.
*
* `cacheTtl` is left at the KV default (60s). Raising it would buy more
* colo-local hits, but a colo that has the key cached would keep serving it for
* that long after an explicit `delete` — and invalidation-on-mutation is the
* property the org config cache is relying on.
*
* Values are stored wrapped as `{ v, exp }`, which the other two backends do not
* need:
*
* - `v` because `EdgeCacheBackend` uses `undefined` to mean "not cached" while a
* cached `null` is a legitimate value — for the org config bucket it is the
* COMMON one ("this org is managed, use Tinybird"), and caching it is the
* single biggest win there. KV's `get` returns `null` for a missing key, so
* storing a bare `null` would make every managed org a permanent miss.
* - `exp` because KV refuses an `expirationTtl` under 60s, and real buckets ask
* for less (`qe-evaluate` and the integrations routes use 30s). The other
* backends carry their own deadline (`Cache-Control: max-age`, `expiresAt`),
* so without this a 30s entry would be served for 60s. `expirationTtl` still
* goes out at the 60s floor to bound storage; `exp` is what expires the read.
*/
export const makeKvBackend = (kv: KvLike): EdgeCacheBackend => {
const composite = (bucket: string, hash: string) => `${bucket}:${hash}`

return {
name: "workers-kv",
get: async (bucket, hash, nowMs) => {
const envelope = await kv.get(composite(bucket, hash), "json")
if (envelope === null || typeof envelope !== "object" || !("v" in envelope)) return undefined
const { v, exp } = envelope as { v: unknown; exp?: number }
// KV cannot expire an entry sooner than 60s, so short-TTL buckets carry
// their real deadline in the envelope and expire on read.
if (typeof exp === "number" && nowMs >= exp) return undefined
return v
},
put: async (bucket, hash, value, ttlSeconds, nowMs) => {
const ttl = Math.floor(ttlSeconds)
await kv.put(
composite(bucket, hash),
JSON.stringify({ v: value, exp: nowMs + ttl * 1_000 }),
{ expirationTtl: Math.max(KV_MIN_TTL_SECONDS, ttl) },
)
},
delete: async (bucket, hash) => {
await kv.delete(composite(bucket, hash))
},
}
}

export const CacheBackendLive = Layer.effect(
CacheBackend,
Effect.gen(function* () {
const env = yield* WorkerEnvironment
const kv = env.EDGE_CACHE
if (isKvLike(kv)) {
return CacheBackend.of(makeKvBackend(kv))
}

const cache = yield* WorkersCache
if (!cache) {
// The fallback is per-isolate, so nothing is shared across requests that
Expand All @@ -155,4 +70,4 @@ export const CacheBackendLive = Layer.effect(
}
return CacheBackend.of(makeWorkersBackend(cache))
}),
).pipe(Layer.provide(Layer.mergeAll(WorkersCache.layer, WorkerEnvironment.layer)))
).pipe(Layer.provide(WorkersCache.layer))
Loading
Loading