Zero-Code Instrumentation
also called Automatic Instrumentation, Agent-Based Instrumentation
Producing telemetry by attaching an agent to a process or a node instead of editing application code, which buys fleet-wide coverage in weeks and cannot see anything that only exists as application meaning.
A platform team is asked to give 180 services latency and error telemetry in a quarter. Sixty have no instrumentation, 40 are in another language, and the rest are vendor binaries and legacy jobs nobody can rebuild. Any plan requiring a code change has a permanent hole in it, and any plan that never touches code answers none of the questions that pay for the programme.
Agent-based instrumentation resolves the coverage half of that problem and only that half. Its unit of rollout is a node pool or a base image rather than a team's backlog, which is why it ships in weeks.
Why it matters
Coverage is the precondition for everything else. A service with no telemetry is not a gap in a dashboard; it is the service that an incident stops at. Agent-based collection also standardises the shape of the data, because one implementation emits it, rather than 180 teams each interpreting a convention slightly differently.
The published cost envelope is modest for the kernel-level form: the OpenTelemetry eBPF profiler states upper limits of 1% CPU and 250 MB of memory per host, unwinds native stacks from .eh_frame data without requiring DWARF debug information or frame pointers, and ships separate unwinders for the JVM, Python, Ruby, PHP, V8 and .NET.
Implementation patterns
- Bytecode or runtime hooks inside the process: a JVM agent, or monkey-patched client libraries in Python and Node. Rich data, per-runtime operational surface, and it needs a restart.
- Kernel-level eBPF agents outside the process: no library, no restart, whole-host visibility, privileged and sensitive to kernel version.
- Mesh sidecars, which give rate, errors and duration per service pair for the traffic they proxy, and nothing for work that makes no network call.
- The split that works: agents fleet-wide for coverage, hand-written spans and attributes on the few paths where business attribution pays. Decide the split by asking who can rebuild the binary and which questions need tenancy.
- Guard against double instrumentation, where an agent and an SDK both wrap the same call and every request reports two spans with inconsistent parents.
Industry example
The OpenTelemetry project ships both forms deliberately and both run in production estates today: language agents that instrument supported libraries without code changes, and the eBPF profiler for whole-system coverage with no in-process component at all. The profiling signal is still Alpha, which is the honest statement of maturity for the kernel-level path and the main reason to keep the SDK option open.
Failure scenarios
- Asynchronous hops break silently. Propagating context across a queue or a thread pool is an application act, so without it the agent reports two unrelated operations and a request that crosses a broker becomes two traces.
- Runtime upgrades break the agent. A JVM or kernel upgrade can disable instrumentation for a whole fleet, and the symptom is missing data rather than an error.
- Attribute explosion. Default attributes derived from request headers are a cardinality risk, which is why the stable HTTP conventions demoted the host-name attribute to opt-in.
- False completeness. The graphs look full, so nobody notices that no incident can be attributed to a customer and that in-process time is one opaque block.
Trade-offs
| Choose | Gains | Pays |
|---|---|---|
| Kernel-level agent | Coverage including unowned binaries in weeks | No business attributes and no async context |
| In-process SDK | Tenancy, flags and custom spans | 180 teams and a release each |
| Mesh metrics only | Already running, zero per-service work | Only proxied RPCs; nothing in-process |
When not to use it
Under roughly 20 services on a single runtime, a shared instrumentation library in the base image is simpler, gives better data, and avoids a privileged per-node component. Skip the agent too when the question driving the work is per-customer attribution, since no kernel-level signal carries tenancy and the agent would be a detour. Agents earn their place when the estate is polyglot, partly unowned and too large to coordinate, not because they are the more advanced option.
Interview question
Q: You can either mandate SDK instrumentation across 180 services or roll out an eBPF agent fleet-wide. Which do you start with, and what does your choice fail to see?
What a strong answer covers: the two deciding facts, which are rebuildability and whether business attribution is needed; the agent's published overhead and unwinding mechanism; the three blind spots of tenancy, async context and in-process time; the hybrid split with SDK spans on revenue paths; and the conditions that flip it, including a single-runtime estate or a ban on privileged agents.
Quick check
Quiz: What does an eBPF agent see when a request crosses a message broker? — Two unrelated operations, because context propagation across the broker is an application-level act it cannot perform.
Flashcard: What does zero-code instrumentation buy and what can it never provide? — Fleet-wide coverage with no code change and about 1% CPU per host; never tenancy, feature flags or in-process attribution.