Logistic Regression and the Log-Odds

Fit a linear model directly to a 0/1 label and it will happily predict a probability of 1.4 for a confident case and -0.2 for another. The problem is not the linearity; it is that a linear function is unbounded and a probability is not. Logistic regression fixes the mismatch by keeping the linear part and moving it to a scale where being unbounded is fine.

That scale is the log-odds:

The odds \(p/(1-p)\) run from 0 to \(\infty\); their log runs over the whole real line, matching the range of a linear predictor exactly. Nothing about the model is a hack, and this is the same sigmoid that sits at the output of a binary classifier head in a neural network, with the layers below playing the role of learned features.

Reading a coefficient correctly

\(\beta_j\) is the change in log-odds per unit change in \(x_j\), holding other features fixed. Exponentiating gives the odds ratio: \(e^{\beta_j}\) multiplies the odds. A coefficient of 0.69 means the odds roughly double.

The effect on the probability is not constant, and this trips people up constantly. The derivative \(\partial p / \partial x_j = \beta_j\, p(1-p)\) is largest at \(p = 0.5\) and vanishes at either extreme. The same coefficient that moves a probability from 0.50 to 0.62 moves it from 0.98 to 0.99. Odds ratios are constant; probability differences are not, and quoting one while meaning the other is a routine source of overstated effects.

Maximum likelihood, and why there is no closed form

The log-likelihood for labels \(y_i \in \{0,1\}\) is

which is exactly the negative binary cross-entropy. Its gradient has a form worth memorising:

Errors times features, identical in shape to the linear regression gradient, except that \(p\) depends on \(\beta\) nonlinearly. That nonlinearity is why setting the gradient to zero yields transcendental equations with no closed-form solution.

The log-likelihood is concave, so there is a unique maximum whenever one exists and any sane optimiser finds it. Classical implementations use Newton's method, which for this model reduces to iteratively reweighted least squares: each step solves a weighted least squares problem with weights \(p_i(1-p_i)\). Large-scale implementations use L-BFGS or SGD instead, since the Hessian is \(p \times p\) and forming it is the expensive part.

When it breaks

Perfect separation sends coefficients to infinity. If a hyperplane separates the classes exactly, the likelihood increases without bound as \(\|\beta\| \to \infty\): pushing every predicted probability to 0 or 1 always improves it. The MLE does not exist. Software reports enormous coefficients, enormous standard errors, and convergence warnings that are frequently ignored. It is common with small samples, many categorical levels, or a feature that leaks the label. Any penalty solves it, and Firth's bias-reduced likelihood solves it while keeping a principled interpretation.

Calibration is not preserved under resampling. Downsampling the majority class, standard practice for imbalance, shifts the intercept by a known amount, \(\log(r)\) for downsampling rate \(r\), and leaves the slopes approximately unchanged. If you downsample and forget to correct the intercept, the ranking is fine and every predicted probability is wrong. That distinction matters when the output feeds a threshold or an expected-value calculation.

The linearity assumption is on the log-odds scale. A relationship that is monotone but S-shaped in log-odds space needs a spline or an interaction; the model has no capacity to discover it. This is exactly what gradient-boosted trees get for free, and it is much of why they win on tabular problems.

Coefficients are not feature importances. They are scale-dependent, they change when correlated features are added or removed, and their magnitudes are not comparable across features unless the inputs are standardised. Under collinearity they can flip sign while predictions stay stable, the same pathology as in linear regression.