Tagged “latency”

7 posts.

All Model Architecture19 Training & Alignment22 Inference & Serving18 Agents & Orchestration11 Reasoning & Evaluation26 Safety, Security & Governance7 Platforms & Practice20

Inference & Serving 24 min

Designing for a Collaborator That Is Sometimes Wrong: The Evidence Behind Human-AI Interaction Design

In a 2025 randomized trial, experienced developers using AI tools took 19% longer to finish their tasks while believing they had been 20% faster. Twenty-six years of human-AI interaction research explain the gap: an assistant's v…

interaction-design-for-ai product verification calibration ∑ ◫

Inference & Serving 24 min

Goodput, Not Throughput: The Metric That Decides Whether Your LLM Deployment Works

Two servers run the same model on the same GPUs. One reports 4,200 tokens per second and is unusable; the other reports 2,600 and feels instant. Throughput is a property of the server, latency is a property of the request, and th…

inference serving latency goodput ∑ ◫

Inference & Serving 4 min

Prompt caching is not just a cost optimisation - it changes what apps you can build

Most teams treat prompt caching as a billing hack. The deeper consequence is that you can now ship product patterns - persistent agents, large-context tools, system-prompt-heavy UX - that were latency-prohibitive eighteen months ago.

caching inference latency economics

Platforms & Practice 24 min

Query Understanding Is the Hidden Half of Search: From Broder's Taxonomy to LLM Query Rewriting

An LLM-written pseudo-document lifts BM25 by 15 nDCG@10 points on TREC DL 2019, and turns a 16 ms search into one that waits over two seconds for the model. Query understanding decides what the engine searches for before ranking …

query-understanding information-retrieval retrieval rag ∑ ◫

Inference & Serving 24 min

Running Language Models on a Phone: Memory Bandwidth, NPUs, and the Few-Billion-Parameter Ceiling

Phones ship NPUs rated in trillions of operations per second, yet the speed at which a reply appears is set by how fast LPDDR memory can hand a couple of gigabytes of weights to the processor, over and over. This post derives the…

on-device-and-edge-ai on-device-ai llm-inference memory-bandwidth ∑ ◫

Inference & Serving 21 min

Sleep-Time Compute: Thinking About the Context Before the Question Arrives

Test-time compute makes models smarter but slower and pricier. Sleep-time compute moves the thinking to before the user asks, cutting the compute needed for the same accuracy by roughly 5x and the cost per query by 2.5x when cont…

inference test-time-compute agents reasoning ∑ ◫

Inference & Serving 23 min

Speculative Decoding: How a Small Draft Model Makes Large Language Models Think Faster

Autoregressive decoding forces even the largest language models to emit one token at a time, bottlenecked by memory bandwidth rather than compute. Speculative decoding breaks this sequential constraint by letting a lightweight dr…

speculative-decoding llm-inference parallel-decoding draft-model ∑ ◫