An SDK upgrade moves 300 services from the v1.20 HTTP conventions to the stable v1.23 set. `http.status_code` becomes `http.response.status_code` and `http.server.duration` in milliseconds becomes `http.server.request.duration` in seconds. 140 dashboards and 60 alert rules reference the old names. Which rollout avoids both a telemetry gap and a silent unit error?
Show the full answer Hide the answer
What breaks and in what order
Alert rules break first and silently. A rule selecting http.status_code evaluates to an empty result, and an empty result does not fire. The rule does not error, does not page, and does not appear anywhere as broken; the service simply stops having an alert. Dashboards break next and loudly, which is the harmless half.
The unit change is the one that outlives the migration. A threshold written for milliseconds against a metric now reported in seconds is wrong by a factor of 1000, so a panel comparing http.server.request.duration to 500 reads healthy forever, and the histogram bucket boundaries moved with the unit, so bucket-level queries need rewriting rather than rescaling.
The sequence
- Turn on dual emission with the documented
OTEL_SEMCONV_STABILITY_OPT_IN=http/dup, which emits both the old and the stable conventions. Reversible by removing the variable. It costs duplicated attributes and metrics for the window, so check the cardinality headroom first. - Measure the migration from the telemetry. Count series carrying the new attribute per service, so "how far through are we" has an answer and the cutover is not a guess.
- Rewrite queries against the new names, converting units explicitly, and review every rule that compares against an absolute latency number.
- Flip services to stable-only in waves, watching the old series fall to zero per service. This is the point of no return for any query you missed.
- Remove the flag once no old series remain.
The window must cover the longest dashboard lookback plus one on-call rotation; 30 days is typical when panels compare month over month. The migration guide expects instrumentations to keep the dual-emitting major version patched for at least six months, which bounds how long you may take.
Why the other options fail
Renaming attributes in the collector looks like a one-line fix and handles the attribute rename only. It cannot fix the metric name or the unit, it hides which services have actually migrated, and when a service begins emitting both sets the processor produces duplicates that double every count.
One release train with dashboard repairs afterwards accepts a silent alerting gap as the cost. The gap is invisible precisely because empty rules are quiet, so the team learns about it during the first incident that should have paged.
Pinning to the experimental conventions freezes 300 services on a major version that is maintained for a limited period, while every newly added instrumentation defaults to stable. The estate then carries both conventions permanently with no plan, which is the state the migration was supposed to end.
Where data can diverge quietly
Attribute demotions change what you can query, not just how you spell it. net.host.name became opt-in as server.address because it is derived from request headers and is a cardinality risk, so a dashboard grouping by host may legitimately return nothing after the cutover. That is a design decision in the convention, not a bug in the rollout, and it belongs in the migration note rather than in an incident.
When this is over-engineering
Six services and ten dashboards: upgrade, rewrite in an afternoon, accept five minutes of blank panels. The dual-emit window is justified by the number of queries you cannot audit in one sitting, not by the number of services.