Ingestion-Query Isolation

Analytics, observability and event platforms have two workloads with opposite characteristics. Ingestion is continuous, high-volume, and must never fall behind. Querying is spiky, arbitrary, and can consume unbounded resources.

Sharing infrastructure means one customer's expensive query stalls ingestion for everyone, and the asymmetry matters: a slow query is an annoyance, while an ingestion gap is a permanent, visible hole in the customer's data.

Why it matters

The failure is not gradual. Ingestion falls behind, the buffer fills, and data is either dropped or delayed past the point of usefulness — during exactly the period the customer will later want to analyse, because whatever caused the load is what they are investigating.

Implementation patterns

• Different compute for write and read, with no shared pool. The scaling signals differ too: ingestion scales on event rate, query on concurrency.
• A durable queue in front of ingestion, so a storage slowdown becomes a delay rather than a loss and spikiness is absorbed before reaching the store.
• Per-customer query concurrency limits with a visible queue position, rather than unbounded parallelism that consumes the cluster.
• A separate pool for very expensive queries, so an unbounded scan does not sit alongside interactive dashboard refreshes.
• Pre-aggregation for repeated queries only. The distinguishing signal is repetition, not cost: an expensive query run monthly should stay a raw scan; a cheap query run ten thousand times a day is a candidate.
• Per-customer query cost attribution and quotas, which is the control that changes behaviour — analytical workloads have no natural ceiling, and technical limits produce complaints while visible cost produces changed query patterns.

Industry example

Product-analytics platforms such as PostHog and Amplitude ingest very large event volumes while customers run arbitrary segmentation over them. The architecture that survives is a columnar store with strong compression and partitioning for raw events, a small set of materialised aggregates covering the known-hot dashboard queries, and hard isolation between the two paths.

The tempting mistake is to pre-aggregate for exploration, which is by definition unpredictable — producing a large set of aggregates that answer questions nobody asks while the real question still requires a raw scan.

Failure scenarios

• Shared compute, so a customer's scan stalls ingestion.
• No ingestion buffer, converting a storage slowdown into data loss.
• Unbounded query concurrency, letting one customer consume the cluster.
• Pre-aggregation built for exploration, producing maintenance burden with no hit rate.
• No cost attribution, so expensive query patterns have no owner and no feedback.

Trade-offs

Isolation costs utilisation: two pools each need headroom, and some capacity is idle in each. Pre-aggregation costs maintenance, backfills on definition changes, and a set of artefacts that must be kept correct.

The counter-position is that the alternative is a platform whose ingestion reliability depends on customer query behaviour, which is not a property that can be promised to anyone. For a platform whose value is completeness of data, the isolation is not optional.

Interview question

"A customer runs a query scanning two years of events during your peak ingestion hour. Describe what happens in a shared architecture and in an isolated one, and tell me what the customer sees in each."