Skip to content

feat(kvstore): decay eviction freshness and split hit half-life by role#550

Open
unsaltedbutter-ai wants to merge 1 commit into
antirez:mainfrom
unsaltedbutter-ai:kv-eviction-freshness
Open

feat(kvstore): decay eviction freshness and split hit half-life by role#550
unsaltedbutter-ai wants to merge 1 commit into
antirez:mainfrom
unsaltedbutter-ai:kv-eviction-freshness

Conversation

@unsaltedbutter-ai

@unsaltedbutter-ai unsaltedbutter-ai commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

KV disk cache evicts small, frequently-reused prefixes before large single-use checkpoints

Under budget-pressure eviction, ds4_kvstore_entry_eviction_score drops small cold prefixes reused across many sessions and keeps large multi-turn checkpoints that were hit 0-1 times and will not be resumed.

Seen after lowering DS4_KV_DISK_SPACE_MB from 256 GiB to 32 GiB. The boot eviction removed every reused prefix and kept ~25 one-shot conversation snapshots:
evicted disk-cache-full tokens=533 hits=16 size=29.86 MiB (reused prefix)
evicted disk-cache-full tokens=564 hits=16 size=30.27 MiB
kept ... tokens=91920 hits=1 size=1229.8 MiB (one-shot chat)
Two effects combine:

  1. The hit count decays with a 6h half-life on last_used. A prefix reused on a daily or weekly cadence has its hit evidence erased between uses, so at eviction time a 16-hit prefix scores like a 0-hit file.
  2. score = (effective_hits + 1) * tokens/file_size. The constant +1 gives every entry a permanent floor proportional to token density. Large checkpoints are denser (~14 KiB/token vs ~50 KiB/token for a short prefix), so once a prefix's hits decay it ranks below big idle checkpoints no matter how often it was reused.

Net: at a tight budget the cache keeps low-value large one-shot conversations even if quite old and evicts often-but-infrequently reused small prefixes.

Two changes to ds4_kvstore_entry_eviction_score:

  • Replace the constant +1 score floor with a freshness term that decays with idle time (8h half-life). A checkpoint that stops being reused loses its density-driven score within about a day instead of riding its raw size forever.
  • Decay the hit count on a per-role clock: cold session prefixes, reused across sessions, use a 21-day half-life; conversation tips keep the existing 6h. Only cold is treated as an entry anchor now, so evict/shutdown tips no longer get the 2x anchor prior.

Effect: a large one-shot conversation stays protected inside its resume window, then falls below small, frequently-reused prefixes once idle. The manual pin (future last_used) and the superseded-prefix penalty are unchanged.

Replaying an actual purge with the new scoring keeps the reused prefixes that were evicted at every budget and idle-age tested. Half-lives are #defines; the values are logged in the KV disk cache banner.

Related to #444
Orthogonal to #448

Replace the constant +1 score floor with a freshness term that decays with
idle time, so a never-reused checkpoint loses its density-driven score within
about a day instead of persisting on raw size forever. Decay the hit count on a
per-role clock: cold session prefixes, reused across independent sessions, use a
21-day half-life while conversation tips keep the 6h half-life. Classify only
cold as an entry anchor, so evict and shutdown tips no longer receive the
durable-anchor bonus.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant