Pager Boundary

At 20:40 a checkout service starts dropping requests. The container that died was not the application: it was the platform's log-shipping sidecar, whose in-memory buffer grew faster than it could flush and pushed the pod past its 128 MiB limit. The product team is paged, spends eleven minutes establishing that the code involved is not theirs, and pages the platform team, who discover the same default is in every pod in the estate.

The moment a platform ships code that runs inside someone else's process or pod — a sidecar, an agent, an injected library, a base image — it joins that team's failure domain and that team joins its blast radius. Both facts are permanent. What is optional, and usually missing, is a written answer to who gets woken up, and by what signal.

A pager boundary is that answer. It is not an escalation policy; escalation happens after someone has already been woken and has already lost ten minutes. It is a pre-agreed mapping from symptom to owner, with a signal that distinguishes them mechanically.

Why it matters

The cost of not having one is paid twice. During the incident it is minutes of misdirected response, which is the most expensive time in the whole lifecycle. Afterwards it is an unowned defect: the product team mitigates locally by raising a limit, the platform never hears about it, and the same default stays armed in every other pod — so the estate experiences the same failure repeatedly in log-volume order, once per team, with no aggregate visible to anyone.

The blast radius was set by the default, not by the incident. One limit and one buffering strategy multiplied across 2000 pods is also roughly 250 GB of reserved cluster memory, which makes the same decision a cost decision. Naming the owner is what gets both numbers onto one team's roadmap.

Implementation patterns

• Write the mapping into the platform contract, symptom by symptom: the platform's component is killed, cannot flush, or adds latency above a threshold → platform pages; the application exhausts memory or errors on its own account → consumer pages.
• Make one signal decide it. Container-level termination reasons and per-container resource metrics let an alert route on which container in the pod died. Pod-level alerting cannot make this distinction, which is why the routing is a telemetry design choice.
• Publish fleet-level counters the consumer cannot be expected to gather: component version spread per namespace, component-caused restarts per 1000 pods per week, and its p99 added latency. These turn a local mitigation into an estate-wide defect the platform can see.
• Give the component its own error budget, separate from the services that host it, so a bad week is visible as the platform's problem rather than dispersed across forty teams' budgets.
• Ship changes to it as a rollout with a stop button. Because pods pick up a new version when they restart, the rollout is gradual and invisible by default; pin the version per namespace so it can be halted and reversed.
• Review the boundary when the component gains a feature. A log shipper that starts making sampling decisions has moved into the request path and the mapping needs revisiting.

Industry example

The mesh model is the widely documented case. Envoy came out of Lyft, open-sourced in 2016, and its premise is that networking behaviour moves out of applications into a proxy the platform owns — retries, timeouts, mutual TLS, metrics. The gain is uniform behaviour across languages; the consequence is that a retry default, a drain timer or a memory limit chosen by the platform now expresses itself as an application incident, and the proxy's documented behaviour on restart (connections that do not finish draining are cut) becomes a product team's error. Mature mesh operators respond as this practice suggests: the proxy has its own dashboards, error budget and on-call, plus an explicit list of symptoms that page the mesh team rather than the service team.

Failure scenarios

• Silent local mitigation. A team raises the sidecar's limit in their own manifest, the symptom disappears for them, and the fleet-wide defect loses its only reporter.
• Reverse-escalation loops. Neither team owns a latency regression that appears only when both are involved, and the incident stalls in a channel while p99 stays broken.
• Ownership by paging order. Whoever is paged first inherits the fix permanently, which makes the platform's defects follow the alert routing table rather than the code.
• A fleet-wide rollout nobody scheduled. The injected version is unpinned, and a routine restart wave picks up a bad component version across several namespaces at once.

Trade-offs

Drawing the boundary means the platform takes on-call for code running in production it does not deploy and cannot always reproduce, which is a real staffing commitment — it is the point at which a platform team needs a rotation of its own. Leaving it undrawn is cheaper for the platform and moves the cost onto consumers as debugging time in unfamiliar code, which is how platforms acquire a reputation for slowing teams down. A defensible middle position: the platform owns the component's behaviour and the consumer owns the decision to run it, with a documented way to opt out.

When not to use it

A platform that ships only build-time artefacts — a template, a generator, a Terraform module — does not need this. The generated code is the team's, and pretending otherwise creates an on-call commitment for code the platform no longer controls. If you can change a consumer's runtime behaviour without them deploying, you need a pager boundary; if you cannot, you do not.

Interview question

Q: Your platform injects an agent into every pod. A product team's service is killed because the agent exceeded its memory limit. Who is on call, what do you change in the contract, and what do you change in the telemetry?

What a strong answer covers: the product team's reliability budget is spent while the platform owns the fix, because the platform chose the code and the default; the defect is fleet-wide by construction; the contract gains a symptom-to-owner mapping and an opt-out; the telemetry gains container-level termination signals so alerts can route, plus fleet counters for version spread and component-caused restarts; the mechanism-level fix is bounded buffering with a documented drop policy and a limit derived from measured peak throughput; and raising the limit is rejected as a fix that merely moves the threshold.

Quick check

Quiz: A platform-injected sidecar is OOM-killed and takes a product service with it. Whose budget is spent and who owns the fix? — The product team's budget, because their users saw it; the platform owns the fix, because it chose the code and the default that run in that pod.

Flashcard: What signal lets an alert decide whether the platform or the consumer should be paged for a pod failure? — Container-level termination reason and per-container resource metrics, since pod-level alerting cannot tell which container died.