Flashcards
2,671 cards in 36 decks, written from the same source as the concepts. Space to flip, arrows to move, K and R to sort what you know from what you do not. Progress lives in your browser.
Training & Fine-Tuning
From raw web crawl to an aligned model — data, dynamics, scale and adaptation.
Pretraining Data Pipelines
Web-scale corpus construction, filtering, deduplication, decontamination and data mixtures.
Synthetic Data
Self-Instruct, distillation, self-play, and how to generate training data without collapsing.
Training Dynamics & Scaling
Learning-rate schedules, warmup, loss spikes, critical batch size, muP, and scaling laws.
Distributed Training
Data, tensor and pipeline parallelism, ZeRO/FSDP sharding, mixed precision, and offload.
Parameter-Efficient Fine-Tuning
LoRA, QLoRA, DoRA, adapters and soft prompts — adapting big models on small budgets.
Alignment & Post-Training
SFT, reward modelling, DPO/IPO/KTO/ORPO, model merging, and evaluating an aligned model.