Kaapi: Adherence to knowledge base metric

[AkhileshNegi]

Is your feature request related to a problem?
Kaapi's native LLM-as-judge needs a hallucination-detection metric: is the answer grounded in the org's knowledge base or invented? Cosine similarity can't answer this, and we have no standardized way to check groundedness today.

Describe the solution you'd like

• Spend time in OpenAI logs checking the chunks returned for an answer — can they serve as grounding context for the judge?
• Explore an additional OpenAI call with the vector store as parameter, asking if the answer is present in the knowledge base.
• Look at Langfuse evaluator templates for RAGAS (faithfulness/groundedness) and how they can be mapped to Kaapi

[AkhileshNegi]

Findings on chunk-based groundedness

The chunks we need are already retrieved on the eval path — we just don't ask OpenAI to hand them back. file_search retrieves passages and feeds them to the model regardless; include=["file_search_call.results"] only controls whether they're echoed back to us. Our production response path already sets it, so we have a working example; the eval path omits it, so chunks get retrieved, billed, used, then discarded. Adding it costs no extra tokens or latency. That gives us question + chunks + answer per eval item, which we send to an LLM-as-judge for a 0–1 groundedness score — the RAGAS faithfulness idea, written back to Langfuse through the same path cosine already uses. It also makes the second vector-store call redundant, since the chunks are the grounding context.

Open: chunk selection (at score_threshold=0.0, max_num_results=20 we get 0.9-relevant passages next to 0.09 junk — feeding all twenty invites false passes), ~40KB of chunk text per item needing a home, and whether the Batch API honors include (confirmed on the fast path only).