A search engine holds its index entirely in memory to guarantee predictable latency. What does that buy, what does it cost, and when is it the wrong choice?

What it buys

Predictable latency, which is a different property from low latency. A disk-backed index is fast when the working set is cached and slow when it is not, so its latency distribution has a long tail whose shape depends on access patterns. An in-memory index has essentially no such tail.

For interactive search — where the user is typing and expects results between keystrokes — predictability matters more than the average, because a 200ms outlier is visible in a way that a 5ms difference in the average is not.

It also simplifies the system enormously: no cache tier to manage, no eviction policy, no cold-start problem, no divergence between cached and uncached behaviour. Operational simplicity is a real architectural benefit and it is routinely undervalued.

What it costs

• Memory is the capacity limit, and memory is expensive. The dataset size that fits is the dataset size you can serve, full stop.
• Scaling is vertical until it is not. Adding a node means sharding, which reintroduces distributed complexity into a design chosen to avoid it.
• Restart cost. Rebuilding an in-memory index on restart takes time proportional to data size, which affects deployment strategy and failure recovery.
• Durability must come from elsewhere — a write-ahead log or a source of truth to rebuild from.

When it is the wrong choice

When the corpus exceeds affordable memory, or when the workload is analytical rather than interactive. A system scanning large historical ranges gains little from memory residency because it will be I/O bound regardless, and it will not fit.

It is also wrong when the query pattern is extremely skewed and the corpus is large: a disk-backed index with a good cache serves the hot subset at similar latency for a fraction of the memory, and the tail affects only rare queries.

The judgement being tested

This is a genuine architectural trade rather than a maturity difference. A simpler engine with predictable performance and a lower ceiling frequently produces better outcomes than a more powerful one that a team tunes badly — the ceiling is only a problem if you reach it, and the tuning burden is a cost paid every day.

The question to ask is where the corpus will be in two years and whether the team has search expertise. Those two answers decide it.