Skip to content

Memory: HNSW post-filter can under-fill similar-incident recall #67

Description

@JumpTechCode

Summary

similar_resolved_incidents applies a status IN ('resolved','closed') filter after the ANN traversal with a hard LIMIT k. pgvector's HNSW filters post-traversal, so with the default ef_search=40 and a corpus dominated by open incidents, the query can silently return fewer than k results or miss closer resolved matches.

Evidence

  • sentinel/persistence/repositories.py:882-931ORDER BY i.embedding <=> CAST(:qvec AS vector) LIMIT :k (:908-909) with WHERE ... i.status IN ('resolved','closed') (:903-907).
  • No over-fetch multiplier (LIMIT :k is bound directly), no ef_search tuning, no partial index on resolved rows (the only HNSW index idx_incidents_embedding is unfiltered), no hnsw.iterative_scan.
  • The EXPLAIN test only validates the unfiltered ORDER BY.

Fix (options)

  • Over-fetch then filter: LIMIT k * N in a subquery, filter to resolved/closed, take top-k.
  • Or raise ef_search for this query (SET LOCAL hnsw.ef_search).
  • Or add a partial HNSW index on resolved/closed rows.
  • Or enable hnsw.iterative_scan (pgvector ≥ 0.8).
  • Add a recall test that seeds a resolved-sparse corpus and asserts k results.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions