A platform team must give 180 services latency and error telemetry inside one quarter. Sixty are Java and Go services with no instrumentation at all, 40 are Python, and the rest are vendor binaries and legacy jobs nobody can rebuild. Which approach fits the constraint?
Show the full answer Hide the answer
The deciding property
Two facts in the stem settle it. A third of the estate cannot be rebuilt, so any plan requiring a code change has a permanent hole. And the questions that justify the spend — which tenant, which plan, which feature flag, which cache decision — exist only inside the process, so any plan that never touches application code answers none of them. The workload forces a split, and the only real decision is which half comes first.
What the kernel-level agent gives you
No code change, no library in the process, no restart. The published budget for the OpenTelemetry eBPF profiler is an upper limit of 1% CPU and 250 MB of memory per host, and it unwinds native stacks from .eh_frame data without needing DWARF debug information or frame pointers, with separate unwinders for the JVM, Python, Ruby, PHP, V8 and .NET. That is the shape of the cost: one privileged agent per node, sensitive to kernel version, owned by the platform rather than by 180 teams. It is also why it ships in weeks: the unit of rollout is a node pool.
Where it stops
- No business attributes. Tenant, plan and flag are application concepts that never appear in a syscall.
- Asynchronous hops break. Propagating context across a queue or a thread pool is an act the application performs. Without it the agent sees two unrelated operations, so a request that crosses a broker is two traces.
- In-process work is not decomposed. A cache lookup or a lock wait is not an RPC and produces no span, so "where did the time go" stops at the process boundary.
- The OpenTelemetry profiling signal is still Alpha, so backend portability is a live risk rather than a settled one.
Why the other options fail
Mandating SDK instrumentation first is the textbook answer with the constraint removed. It needs 180 teams to prioritise the same quarter, and it still cannot cover the binaries nobody can rebuild. Telemetry arrives last for the oldest systems, which are the ones nobody understands.
Treating the agent's coverage as complete is the common mistake once the rollout succeeds. The graphs look full, so nobody notices that no incident can be attributed to a customer and that every queue boundary is a dead end. This is the option that feels finished and quietly caps the platform's value.
Mesh sidecar metrics alone give rate, errors and duration per service pair, which is genuinely useful and is not instrumentation. The mesh sees RPCs it proxies: not database calls, not in-process time, not jobs that never make a network request. For the legacy batch jobs it reports nothing at all.
What would flip the decision
| If this changes | Choose | Because |
|---|---|---|
| Every service is rebuildable and in one runtime | SDK in a shared base image first | Richer data for the same rollout effort |
| Nodes are managed and privileged agents are refused | SDK or mesh | The agent needs node-level access you cannot get |
| The only requirement is per-service RED | Mesh metrics | Already present and free of per-service work |
| Per-customer attribution becomes the main question | SDK everywhere | No kernel signal carries tenancy |
When not to build this
Under about 20 services on one runtime, a shared instrumentation library beats a privileged per-node agent: better data, fewer moving parts, no kernel dependency. The agent earns its keep when the estate is polyglot, partly unowned, and too large to coordinate — not because it is more sophisticated.