An A100 can execute 312 trillion half-precision operations per second but can only pull about 2 terabytes from memory in that same second. FlashAttention made attention fast not by computing less, but by refusing to touch memory.
A 70B model in FP16 needs 140 GB of memory it spends most of its time waiting to read. Dropping each weight to four bits cuts that to 35 GB, and for years that cut also broke the model. Here is what changed.
A single agent burns about 4x the tokens of a chat. A multi-agent system burns about 15x. The whole question of orchestration is whether the extra agents buy you accuracy worth more than that bill, and four named patterns decide …
Most teams treat prompt caching as a billing hack. The deeper consequence is that you can now ship product patterns - persistent agents, large-context tools, system-prompt-heavy UX - that were latency-prohibitive eighteen months ago.
Test-time compute makes models smarter but slower and pricier. Sleep-time compute moves the thinking to before the user asks, cutting the compute needed for the same accuracy by roughly 5x and the cost per query by 2.5x when cont…