Log Schema Drift

Structured logging turns log lines into records with fields, and dashboards and alerts then query those fields. That makes the field set an interface — one consumed by teams who did not write the service and are not in the code review that changes it.

Interfaces break. What makes this one dangerous is the shape of the breakage: a query filtering on a field that no longer exists, or that now holds a different type, returns zero rows rather than an error. A panel counting failed trips by city goes to zero for 3 of 40 services, which is visually identical to the good news that there were no failed trips.

The variants are mundane. A field promoted from a string to an object when someone adds detail. An integer id emitted as a string by a new client library. A field that moves under a context. prefix after a logging framework upgrade. None of them look like breaking changes to the person making them.

Why it matters

It defeats the monitoring of the monitoring. Threshold alerts fire when a number is too high; almost nobody alerts on a number that stopped existing. A team can run for weeks on a dashboard that is structurally incapable of showing a problem, and discover it during an incident, which is the worst possible time to learn that a panel has been lying.

It also costs data permanently. In stores that fix a field's type from the first document seen, later documents that disagree are coerced or rejected at ingestion. The record is not merely unqueryable; in many configurations it was never stored.

Implementation patterns

• Keep the shared field set small — service, build, trace id, tenant, region, outcome. Every mandated field is another interface to maintain, and a long list gets satisfied mechanically rather than meaningfully.
• Validate the schema in CI. A test that asserts the emitted field names and types against a checked-in schema catches this at the pull request, where it is free.
• Treat a rename as a deprecation. Emit both fields for a window, migrate the queries, then remove the old one — the same discipline applied to any other public interface.
• Alert on absence. Every aggregation that drives a decision needs a companion rule: no data for N intervals is a problem, not a pass.
• Put the store's rejected-document and mapping-conflict counters on the platform dashboard and page on a sustained non-zero rate. This is a direct measure of telemetry emitted and lost.
• Namespace the free-form fields so that only the shared set is a contract and everything else can change without ceremony.

Industry example

The failure is generic to schema-on-read stores in production. Elasticsearch-style dynamic mapping fixes a field's type on first sight and raises a mapping conflict for later disagreement; the conflict is recorded in an ingestion counter that is usually not on any dashboard. Teams that have been bitten typically end up with a log schema registry validated in continuous integration, which is the same answer the event-streaming world reached for message schemas, arrived at independently because the failure mode is the same: a producer change that breaks a consumer nobody told.

Failure scenarios

• The zero that is not a zero: a panel reads zero failures because the filter no longer matches, and on-call reads it as good news.
• The partially-broken dashboard: three of twelve services drift, so the number is plausible but wrong, which is worse than obviously broken.
• Rejected documents: records fail ingestion entirely and are gone, so the incident cannot be reconstructed after the fact.
• A silently narrowed alert: a rule filtering on severity: "error" after the field became level never fires again.
• Cross-team invisibility: the change is reviewed and approved by people who do not know the field is joined on elsewhere.

Trade-offs

Enforcing a schema costs flexibility and adds a gate. Services must coordinate a field change with the platform, a CI check can block a release for a logging detail, and someone has to own the registry. In exchange you get breakage at the pull request instead of during an incident, and a set of cross-service queries that are actually reliable. The cost is paid continuously by producers; the benefit accrues to consumers, which is why enforcement needs a platform owner rather than goodwill.

When not to use it

Inside one team's service, with no cross-service dashboards, this is overhead. Convention and code review are cheaper and work, because the people changing the field are the people querying it. The discipline earns its cost at the point where a dashboard or an alert spans services owned by different teams — and it is worth introducing the day that first happens rather than the day it first breaks.

Interview question

Q: After a release, a dashboard counting errors by region reads zero for three of forty services. Nothing is erroring and no alert fired. Walk me through your investigation and then tell me what you would build so it cannot recur.

What a strong answer covers: checking document counts by service and build before touching the dashboard; the store's rejection and mapping-conflict counters; a diff of emitted fields between builds; the diagnosis that a field's type or nesting changed and the query silently stopped matching; and the preventions — a small shared field set, schema validation in CI, renames handled as deprecations, and alerting on absence rather than only on thresholds.

Quick check

Quiz: Why is log schema drift more dangerous than a logging outage? Because an outage produces an obvious absence of data, while drift produces a confident, well-formed zero that looks like good news.

Flashcard: A structured-log field changes type. What do queries filtering on it return? Zero rows, not an error — and in a store with fixed mappings the disagreeing documents may have been rejected at ingestion and never stored at all.