Skip to content

refactor: remove the TABLASSERT_FULLMAP_SHARDS build tunable - #63

Merged
SkyeAv merged 2 commits into
mainfrom
delete-shards-env-var
Jul 31, 2026
Merged

refactor: remove the TABLASSERT_FULLMAP_SHARDS build tunable#63
SkyeAv merged 2 commits into
mainfrom
delete-shards-env-var

Conversation

@SkyeAv

@SkyeAv SkyeAv commented Jul 31, 2026

Copy link
Copy Markdown
Owner

Removes the TABLASSERT_FULLMAP_SHARDS environment variable so the fullmap RECORDS shard count is fixed at the compile-time cap (16) rather than runtime-configurable. Default builds are byte-for-byte unchanged — 16 was already the default — so this removes only the override, not the sharding.

Build phase

  • Removed resolve_shard_count(): the sole reader of TABLASSERT_FULLMAP_SHARDS in rust/src/fullmap.rs; the build call site is now let shard_count = SHARD_COUNT_SHARDS;.
  • Kept the shared helpers: round_down_pow2 (still used by the read path) and env_usize (still used by the seven surviving tunables) are untouched — no dead code, clippy clean under -D warnings.

Read-path compatibility

  • shard_count_of is preserved: the read path still reads the per-database meta table (shards) and rounds it down to a power of two, so databases built earlier with fewer shards (the former 4-shard layout, or a 2-shard build) still open exactly the shard files that exist and route with the matching mask.
  • Tests retained: runtime_shard_count_builds_and_reads_fewer_shards (drives build_fullmap_inner with 2 shards directly) and legacy_four_shard_db_reads_only_four_shards still pass and now document this as the parameterized/legacy path rather than an env-var feature.

Docs

  • docs/fullmap.md: dropped the TABLASSERT_FULLMAP_SHARDS row from the Build Tunables table and reworded the Output Artifact shard-file line (count fixed at 16; read path still honors a database's recorded count).
  • CONTRIBUTING.md: removed the variable from the example tunables list (the docs/fullmap.md#build-tunables-environment link stays the source of truth).
  • CHANGELOG.md: added an ## Unreleased### Breaking Changes entry; the two historical 8.0.1 mentions are intentionally retained as the release record.

Design

  • Why remove: the shard count is a layout invariant the read path must agree on; exposing it as a build-time knob added surface area (round-down/clamp logic, meta recording, docs) for a value that was effectively always 16. Pinning it simplifies the build without changing any default output.
  • Accepted caveat: anyone still setting TABLASSERT_FULLMAP_SHARDS gets it silently ignored (no error); their builds produce the same 16-shard layout as before. Flagged as a Breaking Change in the changelog for visibility.

Testing

  • cargo fmt --check --manifest-path rust/Cargo.toml → clean (also enforced by the pre-commit hook).
  • cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings → clean.
  • cargo test --manifest-path rust/Cargo.toml55 passed (unit) + 10 passed, 1 ignored (golden; ignored = opt-in regenerate_golden), incl. schema_and_shard_count_are_pinned.
  • uv run maturin develop --manifest-path rust/Cargo.toml → built + installed the extension.
  • uv run ruff check . → all checks passed; uv run ruff format --check . → 63 files already formatted.
  • uv run pytest tests/test_fullmap.py tests/test_fullmap_golden.py tests/test_cover_fullmap.py tests/test_rs.py -q --no-cov57 passed (incl. the 16-shard layout assertion).
  • uv run pytest -q --ignore=tests/test_qc.py616 passed, 27 skipped, 1 failed.

Caveats / not introduced here

  • The single pytest failure and 8 pyright errors are pre-existing environment gaps, not regressions: both are ModuleNotFoundError for sklearn/numpy in qc.py/test_qc.py, because the [qc] extra (torch/sentence-transformers/scikit-learn) isn't synced in this worktree. No .py files are touched by this PR; CI installs --extra qc (.github/workflows/ci.yml) and runs these green.

Summary by CodeRabbit

  • Breaking Changes

    • Fullmap RECORDS files now use a fixed 16-shard layout for newly created databases.
    • The configurable shard-count setting has been removed and is ignored during creation.
  • Compatibility

    • Existing databases continue to be read using their recorded shard count, preserving compatibility with older layouts.
  • Documentation

    • Updated fullmap configuration and development documentation to reflect the fixed shard layout and removed setting.

The fullmap RECORDS shard count is no longer environment-configurable; the
build always writes SHARD_COUNT_SHARDS (16) shard files, which was already the
default, so default builds are byte-for-byte unchanged.

- Drop resolve_shard_count() (the sole reader of the env var) and pin the build
  call site to SHARD_COUNT_SHARDS.
- Keep round_down_pow2 and the read path's shard_count_of, which still honor the
  per-database META `shards` value so databases built with fewer shards under the
  old variable keep opening and resolving correctly.
- Scrub stale env-var / resolve_shard_count references from comments and tests.
- Remove the tunable from docs/fullmap.md and CONTRIBUTING.md; record the removal
  under CHANGELOG "Unreleased" (historical 8.0.1 entries retained).

Verification: cargo fmt / clippy(-D warnings) / test (55 unit + 10 golden) green;
ruff check + format green; fullmap Python tests (57) green.
@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 9a30c7b6-fb72-480e-be37-7fa17e1f6161

📥 Commits

Reviewing files that changed from the base of the PR and between ee46534 and c68b8ff.

📒 Files selected for processing (1)
  • rust/src/fullmap.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • rust/src/fullmap.rs

📝 Walkthrough

Walkthrough

Fullmap builds now use a fixed 16-shard RECORDS layout. Reads continue to use and normalize the shard count recorded in database metadata. Documentation and test descriptions no longer reference the removed environment-controlled setting.

Changes

Fullmap shard layout

Layer / File(s) Summary
Fixed public shard layout
rust/src/fullmap.rs, docs/fullmap.md, CHANGELOG.md, CONTRIBUTING.md
The public build path writes 16 shards and removes TABLASSERT_FULLMAP_SHARDS resolution. Documentation and regression tests describe and verify the fixed layout.
Metadata-based read compatibility
rust/src/fullmap.rs
The read path uses META.shards and normalizes non-power-of-two, missing, invalid, and zero values.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely identifies removal of the TABLASSERT_FULLMAP_SHARDS build tunable, which is the main change.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch delete-shards-env-var

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
rust/src/fullmap.rs (1)

2240-2242: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Cover the removed environment variable through the public API.

The changed assignment is in build_fullmap_db, but the shown layout tests call build_test, which invokes build_fullmap_inner directly. They cannot catch a regression where TABLASSERT_FULLMAP_SHARDS=2 still affects the public build. Add a test that sets the variable, calls build_fullmap_db, and asserts META.shards == "16" plus all 16 sibling shard files. Restore the process environment in a guard or serialize this test.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/src/fullmap.rs` around lines 2240 - 2242, Extend the fullmap tests to
cover the public build_fullmap_db API: set TABLASSERT_FULLMAP_SHARDS to 2,
invoke build_fullmap_db, and assert META.shards is "16" and all 16 sibling shard
files are created. Restore the environment variable with a guard or serialize
the test to avoid process-wide environment races.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust/src/fullmap.rs`:
- Around line 2344-2347: Update shard_count_of and its callers to reject present
META.shards values that are zero, non-power-of-two, or outside the supported
range instead of normalizing them; preserve the missing-metadata default
separately. Remove the test expectation that accepts normalization, and add an
end-to-end lookup test using multiple shard files to verify malformed metadata
cannot silently omit records.

---

Nitpick comments:
In `@rust/src/fullmap.rs`:
- Around line 2240-2242: Extend the fullmap tests to cover the public
build_fullmap_db API: set TABLASSERT_FULLMAP_SHARDS to 2, invoke
build_fullmap_db, and assert META.shards is "16" and all 16 sibling shard files
are created. Restore the environment variable with a guard or serialize the test
to avoid process-wide environment races.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: cab3a110-2b35-4b1d-b18e-1e5d60a3334c

📥 Commits

Reviewing files that changed from the base of the PR and between ee49b62 and ee46534.

📒 Files selected for processing (4)
  • CHANGELOG.md
  • CONTRIBUTING.md
  • docs/fullmap.md
  • rust/src/fullmap.rs

Comment thread rust/src/fullmap.rs
Adds public_build_ignores_shards_env_var: sets TABLASSERT_FULLMAP_SHARDS=2,
builds via the public build_fullmap_db, and asserts META.shards="16" with all
16 sibling shard files present (and no s16) plus a 50-term lookup. The existing
layout tests call build_fullmap_inner / build_test directly, so they could not
catch a public path that re-reads the removed variable. Addresses CodeRabbit's
public-API coverage note on #63.
@SkyeAv

SkyeAv commented Jul 31, 2026

Copy link
Copy Markdown
Owner Author

Addressed the review:

  • Public-API coverage (nitpick) — done. Added public_build_ignores_shards_env_var in c68b8ff: sets TABLASSERT_FULLMAP_SHARDS=2, builds via the public build_fullmap_db, and asserts META.shards="16" with all 16 sibling shard files present (and no s16) plus a 50-term lookup. The layout tests call build_fullmap_inner / build_test directly, so this closes the gap where a re-wired public path could go unnoticed. Env var is set then removed (house pattern, edition 2021), and nothing else reads it. Rust gate green: 56 unit + 10 golden, clippy -D warnings clean.
  • shard_count_of normalization (major) — reasoned decline in the inline thread. Pre-existing, deliberate, documented read-path behavior unchanged by this PR (only comments were edited here). Rejecting malformed META.shards is a separate read-path design change; offered as a follow-up issue.

@SkyeAv
SkyeAv merged commit 0554c1d into main Jul 31, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant