advanced 2 min answer

A food-delivery platform in the mould of Swiggy has 150 services, each logging in its own style. Fifty dashboards and 30 alerts depend on regexes over free text and on field names such as userId, user_id and uid. Platform wants one schema with consistent event names and field names. Describe how you migrate without breaking a single alert.

migrationstructured loggingschemadual emitalertsevent naming
Show the full answer Hide the answer

The sequence

The principle is expand, then contract: the new shape coexists with the old until nothing reads the old one.

  1. Define the schema narrowly. About 12 mandatory fields (ts, service, version, level, event, trace_id, order_id, tenant, error.code and so on), and an event naming rule such as domain.noun.verb (order.payment.failed). A long list is ignored; a short one is enforced.
  2. Normalise at the pipeline first. Add a mapping stage in the log agent that copies userId, user_id and uid into user.id and leaves the originals. This delivers the new field to every service with zero code change, and every old query still works.
  3. Migrate queries to the new fields while the aliases run. Dashboards and alerts are rewritten once, against user.id and event, and the old and new versions run side by side for a week to compare counts.
  4. Services adopt the library that emits the schema natively and dual-emits: new fields plus the legacy ones, for two release cycles. A CI lint rejects log calls without an event name.
  5. Contract. Remove the legacy emit, then the aliases.

Where data can diverge, and how you would know

The silent failure is a query that returns zero rather than an error. For every migrated alert, compare count-per-5-minutes of old and new for a week and alert on divergence above 2%. Add a canary alert: if event=order.payment.failed count is zero for 15 minutes in business hours, page.

The point of no return

Step 5. Before removing the legacy fields, wait for one full retention window (for example 30 days) so that no investigation, report or audit query on historical data still needs them, and grep the dashboard store for the old names.

The rollback at each stage

Steps 2 to 4 are reversible by turning off the new mapping or the new emit. Step 5 is not, which is why it waits.

How long it really takes

Three to four quarters for 150 services. The technical work is weeks; the time is spent in other teams' release cycles and in finding the dashboards nobody remembers owning.

What it costs

Dual emit raises log volume by roughly the size of the duplicated fields, often 10 to 20% for two cycles, so schedule the contract step or the bill stays.

When this is the wrong answer

For 8 services in one repository, skip aliases: change the library, the dashboards and the alerts in one pull request. The staged approach exists because ownership is spread across many teams who cannot be moved at the same moment.