A platform ingests 40 TB of logs a month. Roughly 2% of queries are interactive incident searches that must return in seconds; the rest are audits and monthly investigations where minutes are fine. Full indexing of everything costs more than the compute fleet. Which storage design fits?

The deciding property

Two query populations with different latency requirements are being served by one storage design. Indexing is what makes a query fast and it is also what makes ingestion expensive: the index is built for every record at write time, whether or not anyone will ever query it. Paying interactive-grade indexing for the 98% of data that will only be read by a monthly audit is the whole of the waste.

Datadog's engineering writing on Husky makes the same point from the vendor side: limiting the compute available to queries that do not need low latency is described as the technical underpinning of its lower-cost logs tier. Tiering is not a discount, it is a different compute allocation for a different query class.

Why two tiers

• A fully indexed recent window - typically 7 to 15 days - covers incidents, because incident search is almost always recent. This is where the interactive 2% lives and where the index cost is justified.
• A queryable archive on object storage holds everything for the audit horizon, in columnar files with time-range and coarse-column pruning rather than a full inverted index. Queries scan; scans are slower and far cheaper, and minutes is within the requirement.
• One query interface over both, so an investigation does not need to know which tier answers it. Without this the archive is a dead letterbox that nobody queries and the compliance claim is theoretical.

Why the other options fail

• "Index everything and shorten retention to 7 days." It controls cost and breaks the requirement: audits need months. This is the most common accidental design, because retention is the easiest knob, and it is discovered when a regulator or a long-running fraud investigation asks for something from March.
• "Index 10% by sampling and discard the rest." Sampling is right for traces and wrong for logs used as evidence. A sampled audit trail is not an audit trail, and the one request you need is the one you dropped. Where sampling does belong, it must be biased - keep all errors, sample the routine - not uniform.
• "Keep metrics only." Metrics are aggregations decided at write time and cannot answer a question nobody anticipated, which is precisely what an incident is. This is a real design for a cost-constrained small system, and it means accepting that some incidents will be unexplainable.

What would flip the decision

When this is over-engineering

At a few hundred gigabytes a month, one indexed store with 30 days of retention costs less than the meeting about tiering. The two-tier design earns its keep when the indexing line exceeds the compute line, which is the signal to measure before building anything.