An LLM feature that worked last week now gives worse answers. Nothing was deployed. How do you find out what changed, and what should have been in place?

What the interviewer is testing

Whether you treat an AI feature as a system with configuration and dependencies, or as a black box that mysteriously drifts.

What can change when nothing was deployed

Five candidates, and a system that cannot distinguish between them is not operable:

1. The model changed underneath you. Providers update models behind stable aliases. Behaviour shifts with no change on your side and no notification you necessarily saw. This is the most common cause and the most preventable.

2. A prompt was edited outside the pipeline. If prompts live in a console, a database row or an admin UI, somebody changed one and there is no commit to point at.

3. The retrieval corpus changed. For RAG, new or re-indexed documents shift what is retrieved. The generation is identical; the context is not.

4. Input distribution shifted. A new customer segment, a new locale, a change in how users phrase things. The system did not get worse; the questions got harder.

5. A parameter drifted — temperature, max tokens, top-p, or a truncation limit that started biting as documents grew.

How to find it

The investigation is only possible if the logging is right, which is the real answer to the question. With adequate logging: pull a set of recent poor responses and a set of good ones from last week, and compare the prompt version, model version, retrieved context, parameters and input characteristics across the two. The differing field is the cause.

Without that logging, you are reduced to guessing and re-testing, which is why this fails so often.

What should have been in place

Pin the model version explicitly. Never call a floating alias in production. Adopting a new version becomes a deliberate change, evaluated first.

Prompts in version control, reviewed like code, deployed like code, with a version identifier.

Every response logged with its prompt version, model version, parameters and — for RAG — the retrieved chunk IDs. Without this a regression cannot be attributed, and this is the single most valuable thing on the list.

An evaluation suite with a threshold, in CI. A curated set of real inputs with expected properties, scored, run against every prompt change and every model upgrade. Keep a held-out set that is never used for tuning, or the suite becomes an overfit.

Continuous production sampling. A percentage of live responses scored automatically, plus human review of a sample, trended over time. This is what detects drift before a user reports it — the scenario in the question is one where nobody noticed for a week.

Failed cases fed back into the eval set, so the suite gets stronger at exactly the things that actually break.

What a strong answer adds

Framing prompts and model versions as production configuration with worse controls than code — the same observation that applies to WAF rules, feature flags and IAM policies. The fix is not special to AI: version it, review it, stage it, log which version produced what, and be able to roll back.