You are asked to prove an AI assistant is good enough to launch. How do you construct the evidence?

What the interviewer is testing

Whether you can build a defensible evaluation rather than relying on impressions from demonstrations.

The evidence

A representative question set drawn from real user questions where possible — from support tickets, search logs or a pilot — not questions the team invented, which are systematically easier and better-formed than what users actually ask. A few hundred is usually sufficient.

Ground truth per question: the correct answer, and the source passages that support it. This is manual work and it is the highest-return investment in the whole project.

Separate retrieval and generation metrics. Recall at k for retrieval; faithfulness, correctness and completeness for generation. Measuring only end-to-end quality means you cannot tell which half to fix.

Adversarial coverage: ambiguous questions, questions the corpus cannot answer (does it abstain, or invent?), questions with outdated information, prompt injection attempts, and out-of-scope requests. Abstention behaviour is frequently the most important property and the least tested.

Segment analysis across question types, document types and user groups, because an aggregate score hides systematically poor performance on a subset.

The judging

LLM-as-judge for scale, with the known controls: pairwise comparison rather than absolute scoring, randomised order, a specific rubric, reasoning before verdict — and validated against human labels on a sample, or you do not know what it is measuring.

Human review of a sample regardless, particularly for the high-consequence categories.

What "good enough" means

Define it before measuring: the acceptable rate for each failure category, agreed with the business. A 5% rate of unhelpful answers may be fine; a 0.5% rate of confidently wrong answers on a consequential topic may not be. Weight by consequence, not by frequency.

What a strong answer adds

The launch is not the end of the evidence. Commit to the same evaluation running on every prompt, model or corpus change, plus production monitoring of proxy signals — regeneration rate, thumbs-down, escalation to human support — so degradation is detected rather than reported.

Common weak answers

A demonstration with curated questions. An aggregate quality score with no failure taxonomy.