Tagged “mlops”

Platforms & Practice 24 min

A Model Version Is No Longer a File: Registries for Compound AI Systems

Model registries were built to version a trained artefact you own. An LLM application's behaviour comes from a hosted snapshot that retires on someone else's calendar, plus prompts, an index and tools you change weekly. The relea…

model-registry-and-versioning mlops llm-systems evaluation ∑ ◫

Reasoning & Evaluation 25 min

Error Bars for Evals: Why Most Benchmark Differences Are Noise

A 250-question benchmark carries a standard error of about three percentage points. Most of the model comparisons published on top of such benchmarks cannot distinguish the models they are comparing. Evaluations are experiments, …

evaluation benchmarks statistics mlops ∑ ◫

Training & Alignment 30 min

Feeding the Accelerator: Why the Training Loop Is a Storage Problem

A 75% idle GPU is usually not a compute problem. Across the configurations one VLDB study measured, DNN training spent between 10% and 70% of every epoch blocked on I/O, and the storage system was not busy. The data path has its …

training storage data-loading infrastructure ∑ ◫

Reasoning & Evaluation 24 min

Hidden Technical Debt, a Decade On: What Continuous Delivery for ML Actually Fixed

In 2015 a Google paper catalogued the ways machine learning systems rot, and a decade of MLOps tooling set out to pay that debt down. It paid down the debt that lives in pipelines and artefacts, and left the debt that lives in ju…

ci-cd-for-ml mlops evaluation-mlops deployment ∑ ◫

Platforms & Practice 23 min

Microsoft Azure AI Foundry: The Enterprise AI Development Platform

Microsoft has renamed its AI development platform three times in three years, from Azure AI Studio to Azure AI Foundry to Microsoft Foundry. Behind the branding churn is a genuinely ambitious consolidation: 1,900+ models, a manag…

azure ai-foundry enterprise-ai model-catalog ∑ ◫

Reasoning & Evaluation 24 min

The Leaderboard Is Not Your Corpus: Why Top-Ranked Embedding Models Disappoint in Production

Embedding models are chosen from a leaderboard more often than from an experiment, and the leaderboard now publishes training splits for its own test sets. Between contamination, task-family averaging and geometry no benchmark me…

embeddings retrieval-rag evaluation benchmarks ∑ ◫

Platforms & Practice 24 min

The Leak in Every Training Set: Feature Stores, Point-in-Time Joins, and the Train-Serve Contract

A fraud model can score perfect recall offline and block nothing in production, because its training join looked a few hours into the future. Feature stores exist to enforce one contract: a training row may only see what the serv…

feature-stores mlops feature-engineering data ∑ ◫

Reasoning & Evaluation 23 min

The Model Knows It Is Being Tested: Evaluation Awareness and the Limits of Behavioural Safety Evidence

On one synthetic honeypot evaluation, Claude Sonnet 4.5 said out loud that it suspected it was being tested in 80 to 100 percent of transcripts, against under 10 percent for its predecessor. When the internal representations behi…

safety-alignment evaluation interpretability red-teaming ∑ ◫

Reasoning & Evaluation 24 min

Your Improvement Is Inside the Noise: Seeds, Nondeterminism and the Reproducibility Problem in ML

Change one bit in one weight of a ResNet and, three epochs later, test accuracy differs by more than ten points. Training is a chaotic process, so a seed is not a control variable but a draw from a distribution. Most published an…

experiment-tracking-and-reproducibility statistics determinism mlops ∑ ◫