A search index built by a projection has been serving wrong results for three weeks because of a bug in the projection logic. Walk through recovery.

What is being tested

Whether you understand the single most valuable property of a derived read model — that it can be thrown away — and whether you have thought about what makes that true in practice.

The recovery

1. Confirm the write model is unaffected. This is the first question and it determines everything. If the authoritative store is correct, this is an inconvenience. If the projection has been written back to, or if a workflow made decisions from the bad index and persisted them, it is a data-correction incident with a much larger scope.

2. Assess the blast radius. Which records are wrong, and did anything act on them? A wrong search ranking is cosmetic. A wrong permission or price shown in search that someone acted on is not. This determines whether you need customer communication.

3. Fix the projection logic, and version it. The version matters: it lets you build the corrected projection alongside the current one and compare, rather than replacing in place and hoping.

4. Rebuild. Build a new index from the authoritative source under a new name or alias, verify it against a sample, then atomically switch the alias. This gives an instant rollback — switch the alias back — which an in-place rebuild does not.

5. Backfill any downstream effects, if anything consumed the bad output.

6. Add the monitoring that would have caught it. Almost certainly a comparison job: sample records from the write model, check the projection agrees, alert on divergence rates. This catches the next bug, whatever it is.

The prerequisite that makes this a one-day job

The read model must be rebuildable from the authoritative source. Systems where the projection has accumulated state that exists nowhere else — a field set only by the indexing job, a counter incremented at index time — have quietly lost this property, and the recovery becomes a reconstruction exercise instead.

Two practices protect it:

• Test the rebuild regularly, not just when you need it. A rebuild that has never been run will take four days the first time, discovered under pressure.
• Never write to the read model from anywhere but the projection. The moment a second writer exists, the model is authoritative for something, and it can no longer be discarded.

What a strong answer adds

Projection lag as an SLI. A stalled or broken projection produces no errors — the write store is healthy, the search service is healthy, and results are silently wrong. The only signals are lag and a correctness comparison, and both must be monitored deliberately.

Different derived stores have different lag. In a large serving architecture — the shape LinkedIn operates, where an authoritative store feeds a search index, a graph store and several denormalised views by consuming a change stream — each derived system is behind by a different amount. A user action touching two of them can produce a visibly inconsistent experience, and the product must decide which inconsistencies are acceptable rather than pretending none exist.