Answer quality degraded overnight. No code was deployed. What are the candidate causes?
Show the full answer Hide the answer
What the interviewer is testing
Whether you know the uncontrolled change surfaces specific to LLM applications.
The candidates
A prompt was edited. Prompts are production logic and are frequently the least controlled artifact in the system — editable by anyone with console access, with no history, no review and no rollback.
The provider updated the model behind the endpoint. If the version is not pinned, the model can change without notice. This is an uncontrolled change to your system by a third party, and it is the cause teams least expect.
The retrieval corpus changed. Documents added, removed or re-indexed; an embedding model updated; a chunking parameter altered. Any of these changes what the model is shown without changing the model or the prompt.
Retrieval quality degraded for an infrastructural reason — the index is stale, a filter is excluding documents, the reranker is failing open.
Input distribution shifted. Users started asking a different kind of question, perhaps because of a marketing campaign or a product change.
A guardrail is firing more, truncating or blocking responses.
The controls that make this diagnosable
Version everything and log it with each request: prompt version, model version, retrieval index version, guardrail configuration. Then "what changed" is a query rather than an investigation.
Pin model versions where the provider allows, and treat moving to a new version as a release with an evaluation run.
Prompts in version control, changed through review, evaluated against a regression set before deployment, and deployable and rollback-able as artifacts.
A scheduled evaluation run against a fixed set, alerting on regression. This detects provider-side change, which nothing else will.
What a strong answer adds
The organisational question this raises usefully: who owns prompt changes? They are often authored by product or domain specialists rather than engineers, which is fine — provided the change still goes through evaluation and review rather than around it.
Common weak answers
Assuming the model provider degraded quality without checking your own change surfaces. Adjusting the prompt without an evaluation set to measure against.