Audit events are emitted by a sidecar to a queue and written to immutable storage by a consumer. During a security incident the queue backs up by 40 minutes. What happens, and what would you change in advance?

Second by second, what happens

T+0: The incident begins. Queue depth starts growing because the consumer is sharing the pressure that caused the incident, or because an operator has scaled something down to restore the user-facing path.

T+5 min: Everything still appears correct. Audit emission is asynchronous by design, so the application is unaffected and no alert is defined on queue age.

T+20 min: Responders begin querying the audit store to establish who accessed which account. The store returns results - from 20 minutes ago. The records that matter are the ones not yet written, and nothing in the response indicates that the answer is incomplete.

T+25 min: A responder restarts a pod to contain the incident, or the platform reschedules one. Any events buffered in that sidecar's memory are gone permanently. There is no error, no gap marker and no count.

T+40 min onwards: The backlog drains and the store becomes complete again - except for whatever was lost in memory, which nobody can enumerate.

Where it amplifies

The audit path shares infrastructure with the thing being audited, so the audit record is least available exactly when it is most needed. And because the evidence arrives late rather than wrong, responders make decisions on a partial picture while believing it is whole, which is worse than having no data: the absence of a record is read as the absence of an event.

What the user and the regulator see

Users see nothing. The consequence lands weeks later, when a regulator or an auditor asks who accessed account X at 14:05 and the honest answer is that the record may not exist and the gap cannot be bounded.

What stops it - mechanisms, not monitoring

1. Emit privileged and administrative actions synchronously, in the same transaction as the change. If the audit write fails, the action fails. This is the only construction that makes "every privileged action is recorded" a true statement rather than an aspiration, and the cost is one write on a low-volume path.
2. Sequence numbers per source, so gaps are detectable. A monotonic counter per emitter turns a silent loss into a provable one: "events 40,112 to 41,890 from pod X were never received." Being able to bound the gap is often the difference between a finding and a breach notification.
3. Durable local buffering before the network. Append to local disk with fsync, then ship. A restart loses nothing, at the cost of disk and a slower emit path.
4. Alert on age-of-oldest-unwritten-event, not on queue length. Length depends on throughput; age is the quantity that maps to evidence loss. Alert at a few minutes.
5. Give the audit path its own capacity and its own failure domain - separate queue, separate consumer, separate storage account, not autoscaled down with the application. It should be able to keep running while the application is degraded, which is the one scenario it exists for.
6. Decide and write down the fail-closed boundary. Privileged actions fail closed; high-volume read access fails open with a detectable gap. Pretending both can fail closed produces an availability incident the first time the audit store has a bad day.

What would have to be true for it to self-heal

Only that nothing restarts before the backlog drains - which is precisely what responders do during an incident. A design whose correctness depends on nobody restarting anything during an incident is not a design.

When this is the wrong thing to harden

An internal tool with no regulatory obligation and no privileged actions, where losing some access records has no consequence anyone can name, does not need any of this - and saying so explicitly is better than building a control nobody maintains. The over-correction is just as real: making audit emission synchronous on a read path of hundreds of thousands of events a second turns an audit-store slowdown into a customer-facing outage. Fail closed on privileged actions, fail open with a detectable gap on high-volume reads, and write the boundary down rather than discovering it.