You introduce service maturity scorecards. Within a month every service scores green and nothing has improved. Why, and what now?
Show the full answer Hide the answer
What the interviewer is testing
Whether you anticipate Goodhart's law and can design measurement that resists it.
Why it happened
The checks measured what is easy to verify rather than what predicts failure. A runbook exists — containing one line. An alert is configured — routing to a channel nobody watches. An SLO is defined — at 90%, which nothing can breach. Each check passes and the underlying property is absent.
Publishing comparative scores made it a target. Once teams are ranked, effort goes to the score. This is entirely rational behaviour and should be expected.
What now
Re-derive the checks from incidents. Every check should be defensible by naming an incident class it would have prevented. Checks that cannot pass that test should be removed, which usually cuts the list substantially.
Make checks behavioural rather than declarative. Not "a runbook exists" but "the runbook was opened during the last incident". Not "an alert is configured" but "an alert fired and was acknowledged within the target". Not "backups are enabled" but "a restore was tested in the last quarter, with the measured time". These are much harder to satisfy without the real property.
Stop publishing league tables. Use the scorecard as a conversation between the platform and a team about specific gaps, and report only aggregate trends upward.
Weight by consequence. A uniform scorecard is simultaneously too weak for the payment service and too strict for the internal admin tool.
What a strong answer adds
Treating a scorecard as a detection instrument rather than a control: its purpose is to give the architecture function current, evidence-based visibility of operational risk across the estate without asking anyone to fill in a survey. Judged that way, gaming is a signal that the instrument is measuring the wrong thing.
Common weak answers
Adding more checks. Attaching consequences to the score, which intensifies the gaming.