Tagged “evaluation”

7 posts.

All Model Architecture13 Training & Alignment8 Inference & Serving11 Agents & Orchestration10 Reasoning & Evaluation7 Safety, Security & Governance3 Platforms & Practice11

Model Architecture 23 min

Context Rot: Why Bigger Context Windows Don't Mean Better Retrieval

A million-token window promises perfect recall of everything you feed it. Controlled tests on 18 frontier models show recall degrading steadily, unevenly, and well before the window fills, a pattern researchers now call context rot.

llm long-context rag context-engineering ∑ ◫

Reasoning & Evaluation 25 min

Error Bars for Evals: Why Most Benchmark Differences Are Noise

A 250-question benchmark carries a standard error of about three percentage points. Most of the model comparisons published on top of such benchmarks cannot distinguish the models they are comparing. Evaluations are experiments, …

evaluation benchmarks statistics mlops ∑ ◫

Platforms & Practice 10 min

Retrieval Is a Ranking Problem: Why Your RAG System Doesn't Need a Better Embedding Model

Teams tune the embedding model and the vector database, then wonder why answers are still wrong. Both are the least important parts of the stack. Retrieval is a two-stage ranking problem, and information retrieval solved the shap…

rag retrieval reranking chunking ∑ ◫

Reasoning & Evaluation 13 min

The reasoning-model bubble: when test-time compute stops paying

o3, R1 and Claude extended thinking are a real capability shift on a narrow slice of tasks. They are also being shoved into product surfaces that punish every property reasoning models exhibit - and the bill is starting to arrive.

reasoning test-time-compute economics evaluation ∑

Reasoning & Evaluation 3 min

What the bake-off taught us: classical ML is not dead, it is just under-attended

We pitted twelve sklearn algorithms head-to-head on a tabular dataset. The winner was not the most expensive one. It was not the most modern one. It was the one whose assumptions matched the data.

machine-learning tabular benchmarks evaluation

Reasoning & Evaluation 20 min

When the Judge Is Also a Player: LLM-as-Judge, Contamination, and Why Leaderboards Drift

A strong model grading other models looks like a free lunch for evaluation. It is not. Position, verbosity, and self-preference biases plus quietly leaked test sets mean a leaderboard number can move several points without any mo…

evaluation llm-as-judge benchmarks contamination ∑ ◫

Reasoning & Evaluation 14 min

Your eval pipeline is the moat, not your model choice

The model layer is commoditising and the answer flips every six months. The only durable advantage is the ability to A/B a model swap end-to-end in 48 hours and know whether it improved things for your users.

evaluation benchmarks llm-as-judge reliability