Is your feature request related to a problem?
Another metrci we want to add is whether an assistant's answer actually follows its prompt — right language, right tone, answering what it should and refusing what it shouldn't. Need a metric for this.
Describe the solution you'd like
Research and prototype a judge that checks:
- Language and tone match the prompt.
- Answers what it should, refuses what it shouldn't.
- Uses the fallback response when it doesn't know, instead of inventing facts.
- Holds up against prompt injection.
Try it on a real org's assistant, check cost per question. Output: a scored Adherence to prompt metric that feeds the weighted average.
Is your feature request related to a problem?
Another metrci we want to add is whether an assistant's answer actually follows its prompt — right language, right tone, answering what it should and refusing what it shouldn't. Need a metric for this.
Describe the solution you'd like
Research and prototype a judge that checks:
Try it on a real org's assistant, check cost per question. Output: a scored Adherence to prompt metric that feeds the weighted average.