Digs

Searching X architecture returns material written to be found by that query, which means tutorials and vendor pages. The material worth having was written by an engineer explaining a real system to their peers, and it uses different words. A dig is what comes back from going after that layer on one topic: the architecture reconstructed across real systems, the decisions and the conditions that flip them, what actually broke, and numbers you can plan against. Every claim is graded, linked and dated.

88 field guides 2376 graded sources 412 postmortems 8 categories

Serving, inference, retrieval, agents, evaluation and model operations.

AI & LLM Systems 02 Oct 2026

Bounding what the agent may do

Eighteen months of CVE records, vendor security advisories, specification arguments and shipped enforcement code show how the industry actually bounds agent authority: a deterministic envelope the model cannot…

What surprised meIn four of seven CVE-grade incidents the human-approval gate was never fooled and never even shown; it was re-bound, because the thing the gate check…

29 sources · 16 organisations · 9 postmortems 31 min read
agentsprompt-injectionsandboxingleast-privilegemcp
AI & LLM Systems 26 Sep 2026

Keeping the training run alive

The only two complete public training logbooks (Meta's OPT-175B, BigScience's BLOOM) record what actually stops a big run: roughly two machine deaths a day, stalls that no timeout catches, one slow GPU taxing …

What surprised meThe most expensive failures in both logbooks were failures of the recovery machinery itself, including the cloud provider's support team deleting the…

23 sources · 12 organisations · 8 postmortems 28 min read
trainingfault-tolerancecheckpointinggpugoodput
AI & LLM Systems 24 Sep 2026

Everything that demanded the whole program got archived: ten years of Meta's ML platform

A decade of one company's answer to a problem every deployment path has: how much of a dynamic program you capture ahead of time, and what you do with the part you cannot. Read from defaults at release tags ra…

What surprised meIn PyTorch 2.8 the signature of torch.export.export changed from strict=True to strict=False, so the replacement for TorchScript repeated TorchScript…

24 sources · 11 organisations · 2 postmortems 30 min read
pytorchcompilersdeploymentdeprecationsupply-chain
AI & LLM Systems 18 Sep 2026

Keep the format dumb: ten years of Hugging Face, measured from its own releases

Reads a decade of one company's decisions as a single argument about where a capability belongs, in the file format every reader must parse or in the loader only you control. A reader leaves able to defend tha…

What surprised meThe v5 cleanup removed 113 TensorFlow and Flax model files and 17.7% of the library's lines while adding 29 model directories, but left occurrences o…

20 sources · 9 organisations · 5 postmortems 31 min read
hugging-facemodel-serializationsupply-chaindeduplicationdeprecation
AI & LLM Systems 15 Sep 2026

The fastest change in the stack is a block of text

A field guide to the change lifecycle of the production prompt, the highest-leverage, lowest-friction change surface in an LLM product. Built from the published incidents of OpenAI, xAI, DPD, Cursor and Air Ca…

What surprised meThe industry is running the same experiment in both directions at once: xAI re-coupled prompt changes to code's controls after three incidents in twe…

26 sources · 22 organisations · 4 postmortems 33 min read
promptsllm-operationschange-managementevalsrollback
AI & LLM Systems 08 Sep 2026

When the model quietly gets worse

A field guide to the production incident where answer quality drops while availability, latency and error dashboards stay green, because the change was in a router, a kernel, a precision mode, a prompt or a th…

What surprised meIn every documented incident the detector was a user, never an automated gate, because request-level monitoring is structurally blind to a quality re…

23 sources · 17 organisations · 4 postmortems 27 min read
llm-servingevaluationobservabilityinferencepostmortem
AI & LLM Systems 28 Aug 2026

The token stream stays up and the answers get worse

A field guide to the capacity decision in production LLM serving, reconstructed from the systems of Anthropic, OpenAI, Meta, DeepSeek, Moonshot AI, Character.AI and Perplexity, from three published incident re…

What surprised meThe failure that costs most is not the outage: Anthropic's postmortem and Meta's QCon talk independently describe inference bugs that degrade answer …

22 sources · 16 organisations · 3 postmortems 32 min read
llm-inferencekv-cachevllmcapacity-planningrouting