A BI tool queries the operational read replica directly through a federation layer, avoiding a copy. At 09:03 replica lag starts climbing. By 09:12 checkout p99 has tripled. By 09:30 the replica is 40 minutes behind and support is reporting stale order status. What failed, and which design decision allowed it?

The trigger

An analyst's dashboard filtered on a computed expression — a date wrapped in a function, or a case-insensitive comparison on a string column. The federation layer could not push that predicate down, so it did what federation layers do when a predicate is not pushable: it issued a broad query and filtered the rows itself.

A query that was meant to touch a day touched the whole table. On the replica that is a long sequential read that evicts the working set from the buffer cache.

Why it propagated

Three couplings turn one bad query into a customer-visible incident:

1. Cache pollution. The replica's memory was sized for the operational working set, perhaps 64 GB against a 4 TB table. A single full scan replaces it, so every subsequent operational read goes to disk because the pages it needs were evicted by rows nobody will read again. This is why checkout slowed although checkout's own queries did not change.
2. Replication apply is single-threaded enough to be blockable. Long-running read transactions on the replica hold back apply or force conflict cancellations, depending on the engine. Either way lag grows, and lag on a replica that serves order status is a correctness problem, not a performance one.
3. No resource boundary. The federation layer connected as a normal client, with the same pool, priority and statement timeout as the application.

Why detection lagged

The dashboards watched the analytical layer and the application separately. Nobody had a signal that spanned them, so the analytics team saw a slow report and the platform team saw checkout latency, and the two were correlated by a human at 09:20 rather than by a monitor at 09:04.

The structural fix versus the tempting local fix

The tempting fix is to ask the analyst not to write that filter, or to add a statement timeout. A timeout is worth having and it does not stop the same query being re-run three times.

The structural fix is to decide what federation is for. Query federation is appropriate for low-volume, highly selective lookups against systems whose predicates push down; it is not a substitute for ingestion. Concretely: a separate replica that no customer path depends on, a dedicated database role with its own connection cap and a statement timeout in the tens of seconds, and a planner check that rejects a federated query whose plan has no pushed-down predicate rather than executing it.

When federation is right, and when not to use it

Choose federation when the source is small, the predicate is an indexed equality, and the alternative is a copy that would be stale by the time it is read: a currency table, a slowly-changing reference list, a lookup against a system of record in production. The price of the copy you avoid is a permanent coupling, so pay it only where the copy would genuinely be worse. Not copying data does not remove the coupling; it moves it from storage to the query path, where it is harder to see. Every virtualised source is a production dependency of whoever writes the SQL, and the people writing the SQL usually do not know that.