Your infrastructure drift detection fires 200 alerts a day. The team has muted it. How do you make it useful again?

What the interviewer is testing

Whether you understand that alert noise is a design defect, and whether you can distinguish legitimate drift from real drift.

The core distinction

Not all drift is unwanted, and a detector that does not know the difference produces exactly this outcome.

Legitimate and expected: replica counts under an autoscaler, tags applied by cloud provider services, fields populated by controllers, timestamps and generated identifiers, certificate rotation. These will always differ from source and must be explicitly ignored in the tooling.

Real drift that matters: security group rules, IAM policies, encryption settings, network configuration, retention and backup settings, resources created outside code entirely.

The remediation

Classify and suppress. Go through the current 200 and categorise them. In most estates the large majority are the legitimate kind, and suppressing them takes the volume to something a human will read.

Tier by consequence. Security and access drift pages someone. Configuration drift raises a ticket. Cosmetic drift is reported weekly in aggregate. A single undifferentiated alert stream is what caused the muting.

Fix the backlog before enforcing. Drift that has existed for months is not news; it is a known state that needs a remediation plan. Alerting on it daily is what trains people to ignore the channel.

Move towards reconciliation. Detection tells you about drift; a GitOps-style reconciliation loop removes it automatically, so the only alerts are for things that could not be reconciled — a far smaller and more actionable set.

The underlying cause worth naming

Drift is created by the emergency at 3 AM. Someone opens a security group or raises a limit to stop an outage, and the change never returns to code. That is not a discipline failure; it is a missing workflow. The fix is a lightweight path to backfill an emergency change into the repository within a defined window, tracked as part of incident follow-up.

What a strong answer adds

Noting the dangerous consequence of muting: the next apply silently reverts the emergency fix and re-opens the incident, and disaster recovery from code rebuilds an environment that was never the one you were running. Drift is not a hygiene issue — it is a recovery correctness issue.

Common weak answers

Reducing the check frequency, which hides the problem. Adding a dashboard, which nobody looks at either.