Audit Path Availability

Audit events are emitted asynchronously to a queue and written to immutable storage by a consumer. It is a good design, and it has one property nobody states: during an incident the queue backs up, because the consumer shares the pressure that caused the incident or because an operator scaled something down to protect the user-facing path.

Twenty minutes in, responders query the audit store to establish who accessed which account. It returns results from twenty minutes ago, with nothing in the response indicating that the answer is incomplete. The records that matter are the ones not yet written, and the absence of a record reads as the absence of an event.

Then somebody restarts a pod to contain the incident. Whatever was buffered in that sidecar's memory is gone, with no error, no gap marker and no count.

Why it matters

Audit exists for two moments - an incident and an audit - and both are moments of stress for the systems the audit path shares. A design that is correct in steady state and lossy under pressure is wrong in exactly the cases it was built for.

The consequence is worse than missing data. Responders making containment decisions on a partial record while believing it complete will conclude an account was not accessed when the evidence simply has not arrived. Later, when a regulator asks who accessed account X at 14:05, the honest answer is that the record may not exist and the gap cannot be bounded - and the inability to bound the gap is often what turns a finding into a notification obligation.

Implementation patterns

• Emit privileged and administrative actions synchronously, in the same transaction as the change. If the audit write fails, the action fails. This is the only construction under which "every privileged action is recorded" is a true statement, and the cost is one write on a low-volume path.
• Sequence numbers per emitter. A monotonic counter turns silent loss into provable loss: "events 40,112 to 41,890 from pod X never arrived." Bounding a gap is a different conversation from discovering one.
• Durable local buffering before the network - append to local disk with fsync, then ship - so a restart loses nothing.
• Alert on age-of-oldest-unwritten-event, not on queue length. Length depends on throughput; age maps to evidence loss. A few minutes is the right threshold.
• Separate failure domain and separate capacity: its own queue, consumer and storage account, not autoscaled down with the application, so it keeps running while the application is degraded.
• Write down the fail-closed boundary. Privileged actions fail closed; high-volume read access fails open with a detectable gap. Claiming both fail closed produces an availability incident the first time the audit store has a bad day.
• Make completeness visible in query results - a query over a window with a known gap should say so rather than returning a confident subset.

Industry example

The pattern recurs in published incident write-ups across the industry: diagnostic and audit tooling that shares infrastructure with the system it observes becomes unavailable during large failures, and time-to-diagnose rather than the original defect dominates the outage. The 2021 Roblox outage, where the telemetry stack depended on the same cluster that was failing and engineers progressively lost visibility over roughly 73 hours, is the best-documented version of the general shape. Regulated sectors encode the countermeasure rather than the aspiration: financial and healthcare audit requirements specify write-once storage with integrity evidence, because an audit trail that can be lost or altered under pressure is not evidence.

Failure scenarios

• Backlog during the incident, so responders see a stale picture presented as current.
• In-memory buffer lost to a restart, which is the action responders take first.
• Shared autoscaling. The audit consumer is scaled down to free capacity for the user path, by an automation that does not know what it is deciding.
• The audit store behind the same identity provider that is failing, so nobody can read it.
• Unbounded gaps. No sequence numbers, so the organisation cannot state what is missing and must assume the worst.
• Fail-closed on a read path. Audit emission made synchronous everywhere, so an audit-store slowdown becomes a customer-facing outage - the over-correction, and the reason the boundary has to be written down.

Trade-offs

When not to use it

Do not apply the full treatment to everything. High-volume read-access logging at hundreds of thousands of events a second cannot be synchronous without becoming the system's bottleneck, and the correct design there is asynchronous with sequence numbers and an age alert.

For an internal tool with no regulatory obligation and no privileged actions, where losing some access records has no consequence anyone can name, a single queue and best-effort delivery is proportionate - and saying so explicitly is better than building a control nobody will maintain.

Interview question

Q: Your audit events go through a queue to immutable storage. During a security incident the queue is 40 minutes behind and a responder restarts the affected pods. What have you lost, what can you prove, and what would you change before the next incident?

What a strong answer covers: that the missing records are exactly the relevant ones, and that the store's confident response hides it · the in-memory loss on restart being unbounded without sequence numbers, and why bounding it changes the regulatory conversation · synchronous in-transaction emission for privileged actions, with the fail-closed boundary written down and reads deliberately excluded · local durable buffering, per-emitter sequence numbers, and an alert on age-of-oldest-unwritten-event rather than queue depth · a separate failure domain and capacity for the audit path, including not autoscaling it down with the application · and the honest trade: the audit store's availability becomes the action's availability on the synchronous path.

Quick check

Quiz: Why is queue length the wrong alert for an audit pipeline? Because length depends on throughput while age-of-oldest-unwritten-event is the quantity that maps directly to evidence you cannot produce during an incident.

Flashcard: What single addition turns silent audit loss into provable loss? - A monotonic sequence number per emitter, so a gap can be stated exactly ("events 40,112 to 41,890 from pod X never arrived") instead of merely suspected.