Incident Response for AI Systems
Why an AI incident often has no error to page on, what detection has to rely on instead, and the response steps that differ from a conventional outage.
A conventional incident announces itself: something returns errors, latency spikes, a dashboard turns red. An AI incident frequently does none of that. The system is available, fast, and confidently wrong, and the first signal is a user complaint, a journalist's email, or a pattern someone noticed in a sample.
Detection without an error signal
The detection sources that actually work are ordered roughly by how early they fire.
Automated monitoring on output distribution, refusal rates, guardrail triggers and confidence, which catches gross behavioural change within hours. It does not catch a model that is subtly wrong in a specific category.
Human review sampling, a small continuous stream of outputs assessed against a rubric, which catches quality problems monitoring cannot express. Its latency is the review cadence.
User reports, which are the highest-precision signal and the lowest-recall one, since most affected users do not report. A reporting path that is easy to find and routed to someone who acts is worth more than most monitoring.
External discovery, which is the worst case and is common. Researchers, journalists and users on social platforms find problems, and an organisation with no channel for those reports finds out at the same time as everyone else.
What differs in the response
Scoping is harder. For an outage, the affected set is a time window. For a model behaving badly on a category of input, determining who was affected requires querying prediction logs by content, which is only possible if those logs exist with enough fidelity. This is the moment prediction logging pays for itself.
Mitigation options are broader and each is imperfect. Roll back the model, disable the feature, add a filter, route affected inputs to a human, or adjust a threshold. Each trades availability against risk differently, and unlike an outage there is usually no option that is simply correct.
Recurrence prevention needs a regression case. The fix for a bad model output is not only a model change; it is a test case in the behavioural suite so the specific failure cannot return silently. An incident that produces a fix and no test will produce the same incident after the next retrain.
Affected parties may need telling. Where a system made consequential decisions about people, the response includes deciding whether and how to notify and remediate, which is a legal and communications question that engineering cannot answer alone and should not be discovering during the incident.
When it breaks
No severity scale for behavioural incidents. Existing scales are built around availability, so a model producing harmful output for a small subgroup does not map onto them and gets triaged as low. A separate rubric for behavioural severity, based on harm and affected population rather than on request volume, is what stops that.
Nobody owns model behaviour at 3am. On-call rotations cover infrastructure. A behavioural incident needs someone who can evaluate model output and authorise a rollback, and if that person is not in the rotation the response waits for business hours.
Rollback is not always available. If the previous model is unloaded, or the incident is caused by a provider-side model update, the option does not exist. Knowing in advance which mitigations are actually available for each system is part of preparation.
Reporting obligations have clocks. Regulatory regimes impose notification deadlines for serious incidents, and those clocks start at discovery rather than at resolution. Determining who assesses reportability, and how fast, belongs in the runbook rather than in a conversation held while the incident is running.
14 flashcards for this concept
Click a card to reveal the answer.