intermediate 2 min answer

A ride-hailing platform of Grab's shape standardises on structured logs. After a release, the on-call dashboard that counts failed trips by city reads zero for three services and normal numbers for the rest. No errors are being reported anywhere. What broke, and what should have caught it?

grabstructured loggingschema driftlog managementsilent failure
Show the full answer Hide the answer

The first three things to look at, in order

  1. Document counts by service and by build, not the dashboard's aggregate. If the three services are still writing records but the aggregation returns nothing, the records exist and the query no longer matches them. If the counts themselves are zero, this is an ingestion problem and a different investigation.
  2. The store's rejected-document and mapping-conflict counters. Every schema-on-read log store keeps one. It is almost never on a dashboard, which is why this class of failure survives for weeks.
  3. A diff of the emitted field set between the previous build and the current one, taken from the raw records rather than from the code.

The diagnosis

A field changed type or shape, and the query silently stopped matching. The common variants: city was promoted from a string to a nested object {id, name}; trip_id began being emitted as an integer where it had been a string; a field moved under a context. prefix when a logging library was upgraded.

In a store that infers mappings from the first document it sees, the type is fixed for the index, and later documents that disagree are either coerced, rejected, or land in an index with a conflicting mapping. A filter on city: "Jakarta" then matches nothing from those services. The aggregation returns zero, not an error, because from the query's point of view there genuinely were no matching failed trips.

The misleading signal is the absence of errors. Logging pipelines are built to never fail the request that produced the line, so every failure in them is expressed as silence. A dashboard reading zero and a dashboard reading zero because the data stopped arriving are visually identical.

The fix, in order

  1. Alert on absence, not only on thresholds. Every aggregation that drives a decision needs a companion rule: this series has produced no data for N intervals. Most incidents of this class are found by a human noticing a suspiciously round number.
  2. Put the store's rejection and mapping-conflict counters on the platform dashboard and page on a sustained non-zero rate. They are a direct measure of telemetry that was emitted and lost.
  3. Version the log schema and validate it in CI. The field names and types that dashboards and alerts depend on are an interface between services and the platform. Treat a rename as a breaking change: emit both fields for a deprecation window, then remove the old one.
  4. Keep the shared field set small. Five to ten fields — service, build, trace id, tenant, region, outcome — are what cross-service queries actually use. Everything else can be free-form, because nothing depends on it.

When this is the wrong answer

If only one service is affected and it is new, the likeliest cause is simply that nobody instrumented it, and a schema registry would have been solving a problem that did not exist. Schema enforcement earns its cost once dashboards and alerts span teams; inside one team's service, convention and code review are cheaper and work.