Checkpoint Formats and State Dicts

Everything except named tensors. The module class, the architecture config, the dtype intent, and the parallelism layout the tensors were sharded under all live outside the file. Registered buffers are included; anything held as a plain Python attribute is not, so a manually-cached RoPE table can disappear across a save and load with no warning. Loading is reconstruction, not restoration.

Ignoring the return value of load_state_dict. It returns missing_keys and unexpected_keys, and with strict=False it does not raise, so a renamed or mis-prefixed key leaves that parameter at its initialisation while everything else loads correctly. The model runs, produces plausible output, and is quietly broken. Inspect the returned lists and fail the load yourself if they are non-empty.

Safety and zero-copy loading. Pickle is an instruction stream whose GLOBAL and REDUCE opcodes import and call arbitrary symbols, so loading an untrusted checkpoint executes code from the file. Safetensors is a JSON header of names, dtypes, shapes, and byte offsets followed by raw bytes, interpreted by nothing. Because each tensor is a contiguous byte range, the file can be memory-mapped rather than deserialised into fresh allocations.

Optimiser state dominates. Adam holds two fp32 moments per parameter, so with fp32 master weights the optimiser state alone is roughly twice the model size, before scheduler state, data-loader position, and RNG state. A checkpoint that saves only weights produces a resume that looks successful and behaves like a fresh run with a warm initialisation, losing the momentum and second-moment estimates that were shaping every step.

DataParallel and DDP wrap the model and prefix every key with module.; torch.compile wraps it and adds _orig_mod.. Both stack, so keys accumulate prefixes depending on how the model was wrapped at save time. The remapping regexes that grow around a long-lived training script are load-bearing and untested: one that strips a segment too many converts a loud key error into a silent partial load.

Because a process killed mid-write leaves a truncated file that parses fine as bytes and fails only when someone tries to resume, potentially hours or days later. Write to a temporary path and rename, since rename is atomic on the same filesystem, keep more than one generation, and verify the newest checkpoint loads before deleting its predecessor. Corruption discovered at resume time is how a fault-tolerant run still loses a day.

Everything except named tensors. The module class, the architecture config, the dtype intent, and the parallelism layout the tensors were sharded under all live outside the file. Registered buffers are included; anything held as a plain Python attribute is not, so a manually-cached RoPE table can disappear across a save and load with no warning. Loading is reconstruction, not restoration.

Ignoring the return value of load_state_dict. It returns missing_keys and unexpected_keys, and with strict=False it does not raise, so a renamed or mis-prefixed key leaves that parameter at its initialisation while everything else loads correctly. The model runs, produces plausible output, and is quietly broken. Inspect the returned lists and fail the load yourself if they are non-empty.

Safety and zero-copy loading. Pickle is an instruction stream whose GLOBAL and REDUCE opcodes import and call arbitrary symbols, so loading an untrusted checkpoint executes code from the file. Safetensors is a JSON header of names, dtypes, shapes, and byte offsets followed by raw bytes, interpreted by nothing. Because each tensor is a contiguous byte range, the file can be memory-mapped rather than deserialised into fresh allocations.

Optimiser state dominates. Adam holds two fp32 moments per parameter, so with fp32 master weights the optimiser state alone is roughly twice the model size, before scheduler state, data-loader position, and RNG state. A checkpoint that saves only weights produces a resume that looks successful and behaves like a fresh run with a warm initialisation, losing the momentum and second-moment estimates that were shaping every step.

DataParallel and DDP wrap the model and prefix every key with module.; torch.compile wraps it and adds _orig_mod.. Both stack, so keys accumulate prefixes depending on how the model was wrapped at save time. The remapping regexes that grow around a long-lived training script are load-bearing and untested: one that strips a segment too many converts a loud key error into a silent partial load.

Because a process killed mid-write leaves a truncated file that parses fine as bytes and fails only when someone tries to resume, potentially hours or days later. Write to a temporary path and rename, since rename is atomic on the same filesystem, keep more than one generation, and verify the newest checkpoint loads before deleting its predecessor. Corruption discovered at resume time is how a fault-tolerant run still loses a day.