Priors, Conjugacy and the Posterior

A new ranking model wins 7 of 10 head-to-head comparisons. The maximum likelihood estimate of its win rate is 0.70. Nobody believes 0.70. Every prior model in this family has landed between 0.45 and 0.55, and ten comparisons is thin evidence against that. Writing that belief down as a Beta(20, 20) prior and updating gives a posterior mean of \((20+7)/(40+10) = 0.54\), which is what a careful reader would have concluded anyway. The Bayesian machinery is a way of making that reasoning explicit and reproducible rather than a matter of taste.

Bayes' rule is \(p(\theta \mid x) \propto p(x \mid \theta)\,p(\theta)\). The likelihood is what the data say, the prior is what you knew, and the posterior is the combination. The interesting content is entirely in how much each contributes.

The prior as pseudo-data

For a Beta\((\alpha, \beta)\) prior on a probability with Binomial data, the posterior is Beta\((\alpha + s, \beta + n - s)\) for \(s\) successes in \(n\) trials. The update is addition. That makes the interpretation immediate: the prior behaves exactly like having already observed \(\alpha\) successes and \(\beta\) failures, and \(\alpha + \beta\) is its strength in units of observations.

This is the honest way to choose and to defend a prior. Beta(1,1) is uniform and worth two observations. Beta(20,20) is worth forty, which on a ten-observation experiment means the prior dominates, and that is a claim you should be willing to state out loud. Reporting a prior as a "weakly informative regularising choice" without saying how many observations it is worth hides the one number a reviewer needs.

The same reading works elsewhere. A Normal prior with variance \(\tau^2\) on a mean with known observation variance \(\sigma^2\) gives a posterior mean that is a precision-weighted average, with the prior contributing the equivalent of \(\sigma^2/\tau^2\) observations. Ridge regression is exactly a Gaussian prior on coefficients, so the penalty \(\lambda\) is a prior strength in disguise, and the lasso is a Laplace prior.

Conjugacy, and why it still matters

A prior is conjugate to a likelihood when the posterior stays in the same family. Beta-Binomial, Gamma-Poisson, Normal-Normal, Dirichlet-Multinomial: in each case the update is arithmetic on the parameters and the normalising constant is known.

Modern samplers make conjugacy unnecessary for correctness, and it remains valuable for three reasons. Closed forms are exact and instant, which matters inside a loop such as a bandit algorithm updating thousands of arms per second. They provide a check on a sampler that is supposed to reproduce a known answer. And the parameter arithmetic makes the prior's influence legible in a way an MCMC trace does not.

The trouble with "uninformative"

There is no prior that expresses no information. A uniform prior on a probability \(p\) is not uniform on the log-odds of \(p\), so flatness is a property of the parameterisation, not of the belief.

Jeffreys' prior, proportional to \(\sqrt{\det I(\theta)}\), is invariant under reparameterisation, which is the closest thing to a principled default. For the Binomial it is Beta(0.5, 0.5), which puts extra mass near 0 and 1, and it is the reason Jeffreys intervals behave sensibly at \(s = 0\) where the Wald interval collapses.

Improper priors, which do not integrate to one, are sometimes usable because the posterior is still proper. Sometimes it is not, and the failure is silent: a sampler runs, produces output, and the chain is exploring a distribution that does not exist. Checking propriety is not optional when the prior is improper.

When it breaks

With enough data, the prior stops mattering, and that is often the wrong comfort. The Bernstein-von Mises theorem says the posterior converges to a normal centred at the MLE, so priors wash out asymptotically. The regimes where Bayesian methods are chosen, rare events, small samples, hierarchical models with few observations per group, are exactly the regimes where washing out has not happened.

Priors on parameters are not priors on predictions. A prior that looks reasonable on a logistic regression's coefficients can imply that predicted probabilities are almost all near 0 or 1. Prior predictive simulation, drawing parameters from the prior and generating fake datasets, exposes this in minutes and is the single most useful diagnostic available before any data are used.

Model misspecification is not repaired by a prior. If the likelihood is wrong, the posterior concentrates confidently on the best wrong answer, and posterior intervals understate the error because they account for parameter uncertainty and not for structural uncertainty.

Conjugacy is a property of a pair, not of a prior. A Beta prior is conjugate to a Binomial likelihood and not to a logistic regression with covariates. Chasing conjugacy in a model that does not admit it distorts the model to fit the mathematics, which is the wrong direction of accommodation now that samplers exist.