Control Heartbeat

A nightly job checks that no production storage bucket allows public access and raises an alert when one does. For four months it has raised nothing. The team reports the control as effective. The job has not run since a dependency upgrade in month one: its schedule fires, the process exits non-zero within seconds, and nothing watches the exit code.

A control that reports only exceptions cannot tell you it is alive, because its silence means both "clean" and "dead". Every exception-reporting control has this property, and it is the single most common way a well-designed control stops operating without anyone noticing.

The pattern is small: emit a positive record on every run, carrying the run time, the population size evaluated and the findings count - zero included - and alert on the absence of the record rather than on the presence of a finding.

Why it matters

Design effectiveness and operating effectiveness are assessed separately for exactly this reason, and operating effectiveness is the harder of the two to demonstrate. An assessor sampling a period wants evidence that the control ran on each occasion it should have. A heartbeat series answers that directly: 92 expected runs in the quarter, 92 records, each with a population count.

It also changes the alert that catches the failure. Alerting on findings catches violations. Alerting on missing heartbeats catches the loss of the control itself, which is the bigger event and the one nothing else is watching.

Implementation patterns

• One record per run, with run id, start and end time, population evaluated, findings count, rule set version. The population count is what detects the quiet narrowing of scope - a run that evaluated 400 resources where last week's evaluated 9,000 is a failure wearing a success badge.
• Alert on two consecutive missing intervals, not one, so a single retry does not page anyone. For a nightly control that is 48 hours of exposure, which is the cost of the choice and should be stated.
• Monitor the population count as a time series with a band, typically plus or minus 20% week on week, because scope collapse is more common than schedule failure.
• The same record is the audit evidence. Keep the series for the assurance period plus report lag, and the question "did this control operate throughout" becomes a query rather than a project.
• Fail-closed where the control is preventive. A gate that cannot evaluate should block; a detective control cannot block, so a heartbeat is the whole of its liveness story.

Industry example

The pattern is standard practice in two places architects already know: a dead man's switch on an alerting pipeline, where a synthetic event must arrive every interval or the monitoring itself is declared down, and the backup world's restore-test record, where the only acceptable evidence that backups work is a dated record of a restore rather than a dated record of a backup job. Published postmortem accounts of monitoring blind spots repeat the same shape - a check that had been failing to run for weeks in production, in a system whose dashboards were green because nothing was reporting problems.

Failure scenarios

• Silent stop. The scheduler fires, the process dies on a missing credential, and four months of silence reads as four months of clean operation.
• Scope collapse. The control runs, an inventory query returns one account instead of twelve, and findings drop to zero for the right-looking reason.
• Heartbeat without content. A record proving the job ran, with no population count, so a run that evaluated nothing is indistinguishable from a run that evaluated everything.
• Nobody watching the heartbeat. The records are written to a table with no alert attached, which moves the blind spot one layer out.

Trade-offs

When not to use it

A preventive control in the request path announces its own failure: if the admission check cannot evaluate, deploys stop and thirty engineers say so within minutes. Adding a heartbeat there is instrumentation without assurance value. Skip it too for controls whose findings are continuous - something that reports dozens of items daily already proves it is alive, and the useful signal is a sudden drop in volume rather than a separate record. The pattern earns its place on controls that are expected to find nothing, which is most compliance monitoring and every backup verification.

Interview question

Q: A detective control has reported no findings for four months. What do you ask for before accepting that as evidence of effectiveness, and what would you add to the design?

What a strong answer covers: zero findings and zero runs are indistinguishable from the outside · a per-run record carrying population and findings counts · alerting on missing intervals rather than on findings, with the interval count sized against acceptable exposure · population count as a time series to catch scope collapse · the heartbeat series doubling as the operating-effectiveness evidence · and when it is unnecessary, namely preventive controls that block work when they break.

Quick check

Quiz: Why is "no alerts for four months" weak evidence that a detective control is working? Because an exception-only control is silent when it is clean and silent when it is dead, and nothing in the design distinguishes the two.

Flashcard: Which two fields must a control's run record carry, and what does each catch? - A findings count, which distinguishes a clean run from no run, and the population evaluated, which catches a control whose scope silently collapsed to a fraction of the estate.