Share Pebble block cache across stores

[bootjp]

Author: bootjp

Summary:

• Reuse one process-wide Pebble block cache across all stores opened in the node process.
• Keep explicit per-open cache references so NewPebbleStore, restore reopen paths, and temp restore DBs release their borrowed refs on close/error.
• Update the scaling roadmap to mark shared block-cache M1 as shipped while leaving memtable/compaction budgets and fairness as follow-ups.

Validation:

• GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./store
• GOCACHE=$(pwd)/.cache GOLANGCI_LINT_CACHE=$(pwd)/.golangci-cache golangci-lint --config=.golangci.yaml run ./store --timeout=5m
• git diff --check

[bootjp]

@codex review

[coderabbitai]

Warning

Review limit reached

@bootjp, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 41 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ffcd916d-e085-4d2d-81fd-81f0daee8244

📥 Commits

Reviewing files that changed from the base of the PR and between e1890c5 and d8ae776.

📒 Files selected for processing (6)

• docs/design/2026_06_12_proposed_scaling_roadmap.md
• docs/design/2026_06_23_proposed_scaling_roadmap.md
• monitoring/pebble.go
• monitoring/pebble_test.go
• store/lsm_store.go
• store/lsm_store_env_test.go

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[gemini-code-assist]

Code Review

This pull request implements a process-wide shared Pebble block cache across all stores in a node, replacing the previous behavior where each store allocated its own private 256 MiB cache. This change addresses the high memory overhead when running multiple Raft groups on a single node. The design roadmaps have been updated to reflect that this milestone (M1) has landed, and new tests have been added to verify that multiple stores correctly share the same cache instance. I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9eec7e7f77

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Emit shared cache metrics only once

Once all stores use this one process-wide cache, Pebble’s Metrics().BlockCache is the same global cache snapshot for every DB. The existing collector still polls each raft-group store and exports size/capacity/hit/miss values with a group label (monitoring/pebble.go 327-346), so on any node hosting multiple groups the same cache usage and counter deltas are emitted once per group; per-group series show traffic from other groups and node-level sums/dashboards overcount by roughly the group count. Please either publish shared block-cache metrics once per process/node or keep these metrics scoped to per-store caches.

Useful? React with 👍 / 👎.

[bootjp]

Addressed the shared block-cache metrics finding.

Changes:

• Emit elastickv_pebble_block_cache_{size,capacity}_bytes and {hits,misses}_total once per process/node instead of once per raft group.
• Keep L0, compaction, and memtable metrics group-scoped.
• Split block-cache counter state from per-group compaction counter state so shared cache counters are not overcounted.
• Added a regression test proving two group sources produce a single block-cache series.

Validation:

• GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./monitoring -run 'TestPebbleCollector|TestSetFSMApplySyncMode' -count=1 -timeout=120s
• GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./store -run 'Test(DefaultPebbleOptions|NewPebbleStoreSharesProcessCache|PebbleCacheEnvOverride|SetSmallPebbleCacheForTestRestores)' -count=1 -timeout=180s
• GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./monitoring ./store -count=1 -timeout=300s
• GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./... -run '^$' -count=1 -timeout=300s
• GOCACHE=$(pwd)/.cache GOLANGCI_LINT_CACHE=$(pwd)/.golangci-cache golangci-lint run ./monitoring ./store --timeout=5m
• git diff --check

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: d8ae776d61

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".