Log Delivery Backpressure
What reaches the application when the log path downstream stops draining, and the drop-or-stall choice that every logging stack makes somewhere whether or not anyone chose it.
At 11:04 a log store starts rejecting writes. At 11:06 p99 on three services sharing the same nodes has gone from 80 ms to 9 seconds, CPU is 18%, the application error rate is zero and nothing was deployed. The services are not slow because of load. They are slow because a log line is a write, and a write to something that is not draining blocks the thread holding it.
Every logging stack resolves this somewhere: in the application's appender, in the container runtime's log driver, in the node agent, or at the backend. The question is whether the resolution was chosen or inherited.
Why it matters
A blocked write produces latency and no errors, which is the hardest signature to diagnose: the error rate stays flat, CPU stays low because threads are parked rather than busy, and the dashboards of the component that actually failed look like a storage problem with someone else's name on it. The failure also crosses service boundaries, because the shared resource is the node, so services that log almost nothing are taken down by a service that logs a great deal.
Implementation patterns
- Bound every buffer and name the drop policy. An unbounded in-process queue converts log backpressure into an out-of-memory kill, which is worse than dropping records.
- Know the container default. Container log delivery offers direct blocking delivery and a non-blocking mode that buffers per container, with a default
max-buffer-sizeof 1 MB and messages dropped once it is full. The blocking mode is the default, and Docker's documentation warns plainly that applications are likely to fail in unexpected ways when stdout or stderr blocks. - Export the drop counters as product metrics, with an alert. A loss you can count is manageable; a loss you infer from missing lines during a postmortem is not.
- Cap record size in the library so one verbose stack trace cannot consume the buffer that everything else shares.
- Keep records that must not be lost off the telemetry path. An audit or financial record belongs on an acknowledged write with retries and an error path, not in a pipe designed to be lossy.
Industry example
The two documented container delivery modes are the clearest published case, because the default is the dangerous one and the warning sits in the vendor's own manual. Node-level agents add the second absorber, usually a memory buffer with optional disk spill, and a platform team of any size eventually discovers that the spill directory shares a disk with the container runtime.
Failure scenarios
- Request threads parked in a write, with latency up and errors at zero, spreading to every workload on the node.
- Loss during the only interesting minute. A burst fills the buffer exactly when the records explain an incident, so the pipeline drops the evidence and keeps the routine traffic that preceded it.
- Disk spill filling the node. The protection against dropping records becomes a full disk, which takes down the container runtime and everything on it.
- Agent restarts under memory pressure, losing whatever it had buffered without a counter anywhere that says how much.
Trade-offs
| Choose | Gains | Pays |
|---|---|---|
| Block the writer | No record is lost | Telemetry becomes a latency dependency of the request path |
| Drop with a counter | Requests stay fast and the loss is measurable | The dropped records are the ones you wanted |
| Spill to local disk | Survives a long backend outage | A new resource to exhaust and a slow replay that competes with live data |
When not to use it
Dropping is the wrong default for a record a regulator will ask about. For those, blocking is correct, and then it must be an explicit transactional write with a visible error, so the request fails loudly rather than hanging for nine seconds. The mistake in the opening timeline is not that a write blocked; it is that a blocking write sat on the request path because nobody chose a mode.
Interview question
Q: Logging is making your services slow during a telemetry incident. Walk me through what you change, and what you refuse to change.
What a strong answer covers: the blocking default at the container boundary and why CPU is low; bounding and drop policies at each absorber; exporting drop counters with an alert; capping record size; moving audit records onto an acknowledged path; and the refusal, which is scaling the log store as the fix, since that removes this trigger and leaves the amplifier in place.
Quick check
Quiz: Latency is up 100x, CPU is 18% and the error rate is zero during a log-store outage. What is the mechanism? — Request threads are blocked in synchronous writes to a log path that is not draining.
Flashcard: Which container log delivery mode is the default and what does the other one do? — Blocking is the default; non-blocking buffers per container, 1 MB by default, and drops new messages when full.