refactor: remove the TABLASSERT_FULLMAP_SHARDS build tunable - #63
Conversation
The fullmap RECORDS shard count is no longer environment-configurable; the build always writes SHARD_COUNT_SHARDS (16) shard files, which was already the default, so default builds are byte-for-byte unchanged. - Drop resolve_shard_count() (the sole reader of the env var) and pin the build call site to SHARD_COUNT_SHARDS. - Keep round_down_pow2 and the read path's shard_count_of, which still honor the per-database META `shards` value so databases built with fewer shards under the old variable keep opening and resolving correctly. - Scrub stale env-var / resolve_shard_count references from comments and tests. - Remove the tunable from docs/fullmap.md and CONTRIBUTING.md; record the removal under CHANGELOG "Unreleased" (historical 8.0.1 entries retained). Verification: cargo fmt / clippy(-D warnings) / test (55 unit + 10 golden) green; ruff check + format green; fullmap Python tests (57) green.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughFullmap builds now use a fixed 16-shard RECORDS layout. Reads continue to use and normalize the shard count recorded in database metadata. Documentation and test descriptions no longer reference the removed environment-controlled setting. ChangesFullmap shard layout
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
rust/src/fullmap.rs (1)
2240-2242: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winCover the removed environment variable through the public API.
The changed assignment is in
build_fullmap_db, but the shown layout tests callbuild_test, which invokesbuild_fullmap_innerdirectly. They cannot catch a regression whereTABLASSERT_FULLMAP_SHARDS=2still affects the public build. Add a test that sets the variable, callsbuild_fullmap_db, and assertsMETA.shards == "16"plus all 16 sibling shard files. Restore the process environment in a guard or serialize this test.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/src/fullmap.rs` around lines 2240 - 2242, Extend the fullmap tests to cover the public build_fullmap_db API: set TABLASSERT_FULLMAP_SHARDS to 2, invoke build_fullmap_db, and assert META.shards is "16" and all 16 sibling shard files are created. Restore the environment variable with a guard or serialize the test to avoid process-wide environment races.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@rust/src/fullmap.rs`:
- Around line 2344-2347: Update shard_count_of and its callers to reject present
META.shards values that are zero, non-power-of-two, or outside the supported
range instead of normalizing them; preserve the missing-metadata default
separately. Remove the test expectation that accepts normalization, and add an
end-to-end lookup test using multiple shard files to verify malformed metadata
cannot silently omit records.
---
Nitpick comments:
In `@rust/src/fullmap.rs`:
- Around line 2240-2242: Extend the fullmap tests to cover the public
build_fullmap_db API: set TABLASSERT_FULLMAP_SHARDS to 2, invoke
build_fullmap_db, and assert META.shards is "16" and all 16 sibling shard files
are created. Restore the environment variable with a guard or serialize the test
to avoid process-wide environment races.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: cab3a110-2b35-4b1d-b18e-1e5d60a3334c
📒 Files selected for processing (4)
CHANGELOG.mdCONTRIBUTING.mddocs/fullmap.mdrust/src/fullmap.rs
Adds public_build_ignores_shards_env_var: sets TABLASSERT_FULLMAP_SHARDS=2, builds via the public build_fullmap_db, and asserts META.shards="16" with all 16 sibling shard files present (and no s16) plus a 50-term lookup. The existing layout tests call build_fullmap_inner / build_test directly, so they could not catch a public path that re-reads the removed variable. Addresses CodeRabbit's public-API coverage note on #63.
|
Addressed the review:
|
Removes the
TABLASSERT_FULLMAP_SHARDSenvironment variable so the fullmap RECORDS shard count is fixed at the compile-time cap (16) rather than runtime-configurable. Default builds are byte-for-byte unchanged —16was already the default — so this removes only the override, not the sharding.Build phase
resolve_shard_count(): the sole reader ofTABLASSERT_FULLMAP_SHARDSinrust/src/fullmap.rs; the build call site is nowlet shard_count = SHARD_COUNT_SHARDS;.round_down_pow2(still used by the read path) andenv_usize(still used by the seven surviving tunables) are untouched — no dead code, clippy clean under-D warnings.Read-path compatibility
shard_count_ofis preserved: the read path still reads the per-databasemetatable (shards) and rounds it down to a power of two, so databases built earlier with fewer shards (the former 4-shard layout, or a 2-shard build) still open exactly the shard files that exist and route with the matching mask.runtime_shard_count_builds_and_reads_fewer_shards(drivesbuild_fullmap_innerwith 2 shards directly) andlegacy_four_shard_db_reads_only_four_shardsstill pass and now document this as the parameterized/legacy path rather than an env-var feature.Docs
docs/fullmap.md: dropped theTABLASSERT_FULLMAP_SHARDSrow from the Build Tunables table and reworded the Output Artifact shard-file line (count fixed at 16; read path still honors a database's recorded count).CONTRIBUTING.md: removed the variable from the example tunables list (thedocs/fullmap.md#build-tunables-environmentlink stays the source of truth).CHANGELOG.md: added an## Unreleased→### Breaking Changesentry; the two historical 8.0.1 mentions are intentionally retained as the release record.Design
16. Pinning it simplifies the build without changing any default output.TABLASSERT_FULLMAP_SHARDSgets it silently ignored (no error); their builds produce the same 16-shard layout as before. Flagged as a Breaking Change in the changelog for visibility.Testing
cargo fmt --check --manifest-path rust/Cargo.toml→ clean (also enforced by the pre-commit hook).cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings→ clean.cargo test --manifest-path rust/Cargo.toml→55 passed(unit) +10 passed, 1 ignored(golden; ignored = opt-inregenerate_golden), incl.schema_and_shard_count_are_pinned.uv run maturin develop --manifest-path rust/Cargo.toml→ built + installed the extension.uv run ruff check .→ all checks passed;uv run ruff format --check .→ 63 files already formatted.uv run pytest tests/test_fullmap.py tests/test_fullmap_golden.py tests/test_cover_fullmap.py tests/test_rs.py -q --no-cov→57 passed(incl. the 16-shard layout assertion).uv run pytest -q --ignore=tests/test_qc.py→616 passed, 27 skipped, 1 failed.Caveats / not introduced here
ModuleNotFoundErrorforsklearn/numpyinqc.py/test_qc.py, because the[qc]extra (torch/sentence-transformers/scikit-learn) isn't synced in this worktree. No.pyfiles are touched by this PR; CI installs--extra qc(.github/workflows/ci.yml) and runs these green.Summary by CodeRabbit
Breaking Changes
Compatibility
Documentation