The Hardware Lottery in Architecture Choice

Prune 90% of a network's weights at random positions and you have removed 90% of the multiplications. Measure it on a GPU and it is frequently slower than the dense version. The FLOPs went away and the wall-clock did not, because a dense matmul maps onto tensor cores and a scattered one does not.

Sara Hooker named the general pattern: a research idea wins because it suits the available software and hardware, not because it is better than the alternatives (Hooker, 2020, The Hardware Lottery, arXiv:2009.06489, published in Communications of the ACM, December 2021). The essay's historical examples are about ideas cast as failures when they were really mismatches, and its warning is that domain-specialised accelerators make straying off the beaten path progressively more expensive.

The lottery is a measurable thing, not a metaphor

Three cases where the fit, rather than the idea, decided the outcome:

Unstructured sparsity lost to 2:4. Arbitrary sparsity patterns have better accuracy-per-remaining-weight than constrained ones. NVIDIA shipped hardware support for exactly one pattern, two non-zeros in every group of four, so 2:4 is what production models use (see N:M semi-structured sparsity). The winning pattern is the one with a datapath, not the one with the best Pareto curve.

State space models needed a kernel before they needed a better equation. Selective state space models are not parallelisable as a convolution, which is what made them expressive and what made them slow. Mamba's contribution was as much the hardware-aware selective scan, which keeps the recurrence in SRAM and never materialises the full state in HBM, as the selectivity mechanism itself (Gu and Dao, 2023, Mamba, arXiv:2312.00752). The architecture became viable when the kernel did.

Attention got a memory-aware rewrite rather than a cheaper approximation. A long line of subquadratic attention variants promised asymptotic wins and mostly lost in wall-clock to exact attention with a tiled, IO-aware implementation (Dao et al., 2022, FlashAttention, arXiv:2205.14135). Asymptotics lost to constants because the constants were about memory traffic.

What it means when you are choosing an architecture

The practical version of the lottery is that your architecture is only as fast as its worst-supported operator. A design that is 30% cheaper in FLOPs and hits an unfused fallback path in your runtime is slower, and the gap does not close by itself; it closes when someone writes the kernel.

This is why hardware-aware architecture search optimises measured latency rather than FLOPs (see hardware-aware architecture search), and why the useful question about a new architecture is not "is it better on paper" but "what has to exist for it to be fast, and does that exist on my target". For an idea to be adopted, the kernel, the runtime support and the quantisation story usually all have to arrive together.

When it breaks

The lottery is not a verdict. Calling something a hardware-lottery loser is a claim about the current ecosystem, and ecosystems move. Low-precision floating point was a niche research interest before FP8 and FP4 units existed; mixture-of-experts was a curiosity before fast all-to-all collectives made expert parallelism practical. An idea that lost once can win later, so treat the label as dated, not permanent.

It is also an excuse. Not every unadopted architecture is a lottery victim. Some ideas are simply worse, and "the hardware was not ready" is a comfortable story that is difficult to falsify. The test is whether someone has measured the idea on hardware that suits it, or built the kernel and shown the gap close. Without that, the claim is a hypothesis.

Co-design has its own lock-in. Designing an architecture around one vendor's current silicon is the trap one abstraction level up: it wins on this generation and constrains the next. Models built tightly around a specific tensor-core shape or interconnect topology have been expensive to port, and the cost lands years later, on someone else.