Search the practice set
126 questions, 454 terms and 400 topics in 20 areas.
18 results for “LLM Evaluation”
LLM Evaluation
A repeatable measurement of whether an AI system's outputs are good enough, on cases that reflect the actual task.
Prompt Registry
A versioned store of production prompts with their model bindings, parameters and evaluation results, so a prompt change is a reviewable, traceable, reversible deployment.
Prompt Versioning
Treating prompts as versioned, reviewed, tested artefacts rather than as strings edited in place.
AI Gateway
A shared proxy in front of model providers that centralises routing, keys, quotas, caching, logging and safety policy.
Architecture Trade-off Analysis Method
A structured evaluation that scores an architecture against prioritised quality-attribute scenarios and identifies the points where those attributes conflict.
Context Window
The maximum number of tokens a model can attend to in one request, holding the system prompt, history, retrieved context, tools and the answer.
Guardrail
A deterministic check applied to a model's input or output, enforcing rules that cannot be left to the model itself.
Model Router
Directing each request to a model chosen by the task's difficulty, cost and latency budget, rather than sending everything to the largest model available.
Prompt Injection
An attack in which text from an untrusted source is interpreted by the model as instructions rather than as data.
Retrieval-Augmented Generation
Retrieving relevant documents at query time and putting them in the model's context, so answers are grounded in your data rather than in training data.
Semantic Cache
Caching model responses keyed by the meaning of the request rather than by its exact text, so near-duplicate questions are served without a model call.
Tool Calling
Giving a model a set of typed function definitions it can request to invoke, with the application executing the call and returning the result.
An LLM feature that worked last week now gives worse answers. Nothing was deployed. How do you find out what changed, and what should have been in place?
What the interviewer is testing Whether you treat an AI feature as a system with configuration and dependencies, or as a black box that mysteriously drifts. Wha
A client wants an assistant that answers questions from 50,000 internal documents which change weekly. RAG or fine-tuning? What actually determines the quality?
What the interviewer is testing Whether you understand what each technique actually does, and whether you know that RAG quality is a retrieval problem. Why RAG
LLM Evaluation
Held-out sets, rubric judging, CI gates and production sampling.
LLM Application Architecture
The shape of a production system with a model in the request path.
Research & Evaluation
Assessing a technology quickly without adopting it by accident.
Model Selection
Capability, latency, cost and the evaluation that decides between them.