fix(reranker): score passages with tokenizer text pairs

[aolin480]

Note

This PR should be merged before the NestJS analyzer work. The analyzer PR is stacked on top of this branch so the framework changes can be reviewed separately: aolin480/codebase-context-react-nextjs#3.

Summary

• Score reranker query/passage inputs through the tokenizer's supported text_pair option instead of the positional call shape.
• Keep @huggingface/transformers on the released ^3.8.1 dependency while preserving the cache-corruption recovery path.
• Align release truth tests and ContextBench cwd assertions with the current 2.3.0 release surfaces and macOS path canonicalization.

Verification

• pnpm run type-check
• pnpm run build
• pnpm run format:check
• pnpm test
• Fresh-cache Hugging Face smoke on Node v24.15.0 with @huggingface/transformers@3.8.1

Notes

• This PR does not upgrade @huggingface/transformers.
• The smoke test confirmed 3.8.1 loads the reranker and embedding models; the issue was the tokenizer call shape, not the package version.

[gemini-code-assist]

Code Review

This pull request updates the CrossEncoderTokenizer interface and its usage in the reranker to align with Hugging Face Transformers tokenizer options, refactors release tests to dynamically use the package version instead of hardcoded strings, and introduces path normalization in contextbench tests to ensure consistent path comparisons. Feedback suggests correcting return_tensor to return_tensors in the tokenizer interface and safely handling an optional options parameter in the test mock to avoid potential runtime errors.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[greptile-apps]

Greptile Summary

This PR fixes the tokenizer call shape in the cross-encoder reranker to use the text_pair named option rather than a positional second argument, aligning with the @huggingface/transformers v3 API. It also makes version-tracking tests forward-compatible (no more hardcoded 2.2.0) and applies realpathSync to resolve macOS /tmp → /private/tmp symlink mismatches in the ContextBench path assertions.

• src/core/reranker.ts: scorePair now calls cachedTokenizer(query, { text_pair: passage, padding, truncation, max_length }) instead of the three-positional-argument form; the CrossEncoderTokenizer interface is updated accordingly.
• tests/reranker-recovery.test.ts: Mock tokenizer and the new toHaveBeenCalledWith assertion both reflect the corrected call shape.
• tests/release-truth-surfaces.test.ts / .github/workflows/publish-npm-on-release.yml: Version references are now derived from package.json rather than hardcoded to a specific release, so the tests stay green across future bumps.

Confidence Score: 4/5

The change is safe to merge — the core tokenizer call shape fix is correct and the mock/assertion in the test correctly reflects the new API surface.

The reranker call shape change is straightforward and well-tested. The only notable item is a typo in the newly added CrossEncoderTokenizer interface (return_tensor instead of return_tensors), which has no runtime impact today since that field is never passed, but could silently mislead a future caller.

The CrossEncoderTokenizer interface in src/core/reranker.ts has the minor field-name discrepancy worth fixing before it propagates.

Important Files Changed

Filename	Overview
src/core/reranker.ts	Core reranker fix: tokenizer call shape updated from positional to text_pair named option; interface expanded to match transformers v3 API, though return_tensor (singular) in the new interface appears to be a typo for return_tensors.
tests/reranker-recovery.test.ts	Mock tokenizer updated to match new single-text + options call shape; a new toHaveBeenCalledWith assertion validates the text_pair field but only for one of the three scored results.
tests/release-truth-surfaces.test.ts	Version checks now derive expectedVersion from package.json rather than hardcoding 2.2.0; test descriptions updated to match; safe and correct change.
tests/contextbench-baseline-schema-gate.test.ts	Adds normalizeFilesystemPath using realpathSync to resolve macOS symlinks before comparing cwd capture paths; applied consistently at three assertion sites.
src/embeddings/transformers.ts	Comment-only change: updates inline TSC workaround note for clarity; no runtime behavior affected.
.github/workflows/publish-npm-on-release.yml	Manual dispatch default tag updated from v2.2.0 to v2.3.0 to match current release; aligns with the now-dynamic release truth test.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Caller
    participant rerank
    participant ensureModelLoaded
    participant AutoTokenizer
    participant AutoModel
    participant scorePair

    Caller->>rerank: rerank(query, results)
    rerank->>rerank: isAmbiguous(results)?
    alt scores not ambiguous
        rerank-->>Caller: return results (unchanged)
    else scores ambiguous
        rerank->>ensureModelLoaded: ensureModelLoaded()
        ensureModelLoaded->>AutoTokenizer: from_pretrained(model)
        AutoTokenizer-->>ensureModelLoaded: cachedTokenizer
        ensureModelLoaded->>AutoModel: "from_pretrained(model, {dtype:'q8'})"
        alt model load fails (corrupt cache)
            AutoModel-->>ensureModelLoaded: Error (Protobuf/parse)
            ensureModelLoaded->>ensureModelLoaded: resetLoadedState()
            ensureModelLoaded->>ensureModelLoaded: rmSync(modelCacheDir)
            ensureModelLoaded-->>rerank: throw Error
            rerank-->>Caller: throw Error
            Note over Caller: Next call retries from scratch
        else model load succeeds
            AutoModel-->>ensureModelLoaded: cachedModel
            ensureModelLoaded-->>rerank: resolved
            loop for each result in top-K
                rerank->>scorePair: scorePair(query, passage)
                scorePair->>scorePair: "cachedTokenizer(query, { text_pair: passage, padding, truncation, max_length: 512 })"
                scorePair->>cachedModel: cachedModel(inputs)
                cachedModel-->>scorePair: "{ logits: { data } }"
                scorePair-->>rerank: logits.data[0]
            end
            rerank->>rerank: sort by crossScore desc
            rerank->>rerank: apply sigmoid to scores
            rerank-->>Caller: [...reranked, ...rest]
        end
    end

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Caller
    participant rerank
    participant ensureModelLoaded
    participant AutoTokenizer
    participant AutoModel
    participant scorePair

    Caller->>rerank: rerank(query, results)
    rerank->>rerank: isAmbiguous(results)?
    alt scores not ambiguous
        rerank-->>Caller: return results (unchanged)
    else scores ambiguous
        rerank->>ensureModelLoaded: ensureModelLoaded()
        ensureModelLoaded->>AutoTokenizer: from_pretrained(model)
        AutoTokenizer-->>ensureModelLoaded: cachedTokenizer
        ensureModelLoaded->>AutoModel: "from_pretrained(model, {dtype:'q8'})"
        alt model load fails (corrupt cache)
            AutoModel-->>ensureModelLoaded: Error (Protobuf/parse)
            ensureModelLoaded->>ensureModelLoaded: resetLoadedState()
            ensureModelLoaded->>ensureModelLoaded: rmSync(modelCacheDir)
            ensureModelLoaded-->>rerank: throw Error
            rerank-->>Caller: throw Error
            Note over Caller: Next call retries from scratch
        else model load succeeds
            AutoModel-->>ensureModelLoaded: cachedModel
            ensureModelLoaded-->>rerank: resolved
            loop for each result in top-K
                rerank->>scorePair: scorePair(query, passage)
                scorePair->>scorePair: "cachedTokenizer(query, { text_pair: passage, padding, truncation, max_length: 512 })"
                scorePair->>cachedModel: cachedModel(inputs)
                cachedModel-->>scorePair: "{ logits: { data } }"
                scorePair-->>rerank: logits.data[0]
            end
            rerank->>rerank: sort by crossScore desc
            rerank->>rerank: apply sigmoid to scores
            rerank-->>Caller: [...reranked, ...rest]
        end
    end

Loading

_{Reviews (1): Last reviewed commit: "test(release): Keep current version chec..." | Re-trigger Greptile}