Probabilistic Forecasts and Quantile Loss
Why a point forecast is insufficient for any decision involving asymmetric costs, how pinball loss trains quantiles directly, and what makes a distributional forecast well calibrated.
A point forecast of 100 units answers a question nobody asked. The decision is how much to stock, and that depends on what it costs to be short versus to be long. If stockouts cost five times as much as excess inventory, the right order is well above the mean, and no point forecast can tell you how far above.
Training a quantile directly
The pinball loss for quantile level \(\tau\) is
It penalises under- and over-prediction asymmetrically, and its minimiser is exactly the \(\tau\)-quantile of the conditional distribution. At \(\tau = 0.5\) it reduces to absolute error, whose minimiser is the median.
This is the mechanism that matters: no distributional assumption is required. Train the same model with several \(\tau\) values, or one model with several outputs, and you get an empirical predictive distribution without ever specifying a family. This is why quantile regression dominates practical probabilistic forecasting over parametric alternatives.
The newsvendor connection makes the decision explicit: with underage cost \(c_u\) and overage cost \(c_o\), the optimal order quantity is the \(c_u/(c_u + c_o)\) quantile of demand. The business ratio names the quantile, and the model produces it.
Evaluating a distributional forecast
Calibration asks whether stated probabilities are honest: does the 90th percentile forecast get exceeded 10 percent of the time? It is checkable per quantile and is the first thing to verify.
Sharpness asks how narrow the intervals are. A forecast predicting a very wide interval is perfectly calibrated and useless, so calibration alone is not enough.
The principle is to maximise sharpness subject to calibration. Continuous ranked probability score combines both into one number and reduces to absolute error for a point forecast, which makes it comparable across deterministic and probabilistic models. Winkler and interval scores serve a similar role for specific intervals.
When it breaks
Quantile crossing. Independently fitted quantiles can produce a 90th percentile below the 80th, which is incoherent. Monotonic constraints, joint fitting, or post-hoc sorting all address it, and the last is the crude fix that is usually adequate.
Extreme quantiles need data that does not exist. Estimating a 99th percentile requires many observations in the tail, and for a short series there are none. Extreme quantiles are extrapolation whatever the method, and reporting them with the same confidence as central ones overstates what the data supports.
Marginal quantiles are not a joint distribution. A forecast giving each horizon's quantiles says nothing about the correlation across horizons, so the probability that demand is high for three consecutive weeks cannot be read off it. Decisions over a horizon need sample paths, not marginals.
Downstream systems often discard it. A carefully calibrated distribution consumed by a planning system that takes only the mean has wasted the effort, and the integration is usually the harder half of delivering probabilistic forecasting.
12 flashcards for this concept
Click a card to reveal the answer.