Classical Supervised Learning
Linear and logistic regression, regularisation, margins, kernels, and the geometry underneath them.
8concepts
156flashcards
60minutes of reading
- 01 Linear Regression as Projection Least squares is orthogonal projection onto the column space of the design matrix, which explains the normal equations, the meaning of residuals, and why nobody who ships numerical code inverts the matrix.
- 02 Logistic Regression and the Log-Odds Why classification is modelled on the log-odds scale rather than the probability scale, why there is no closed-form solution, and what perfect separation does to the coefficients.
- 03 k-Nearest Neighbours and the Curse of Dimensionality The simplest classifier there is comes with a strong guarantee in low dimensions and a quiet collapse in high ones, because neighbourhoods stop being local and distances stop discriminating.
- 04 Cross-Validation and Model Selection Cross-validation estimates how well a learning procedure generalises, not how well your fitted model will, and the moment you use it to pick among models its estimate of the winner is optimistically biased.
- 05 Naive Bayes and Generative Classifiers Naive Bayes models how each class generates its features under an assumption that is almost always false, and still classifies well, because getting the ranking right is far easier than getting the probabilities right.
- 06 Ridge, Lasso and Elastic Net Why an $\ell_1$ penalty produces exact zeros while $\ell_2$ only shrinks, what each does to correlated features, and why regularisation is a bias-variance trade rather than a way to fix a bad model.
- 07 Margins, Kernels and the Support Vector Machine How maximising the distance to the nearest point gives a classifier that depends on a handful of examples, and how the dual formulation lets you work in an infinite-dimensional feature space without ever visiting it.
- 08 Probability Calibration: Platt Scaling and Isotonic Regression A classifier can rank perfectly and still report probabilities that are wrong by a factor of three, and post-hoc calibration fixes the numbers without touching the ranking, at a cost in held-out data that depends on which method you choose.