Forecast Evaluation

MASE, pinball loss, Winkler score, Diebold-Mariano, rolling vs expanding window

MAPE is the default metric most teams use, and it has three distinct failure modes that make it actively misleading. It's undefined when actuals are near zero (division by zero). It systematically biases selection toward models that underpredict, because over-forecasts generate larger percentage errors than under-forecasts of equal absolute magnitude. And it's not comparable across series with different scales. MASE solves all three by normalising against the in-sample naïve forecast — scale-free, symmetric, defined when Y_t = 0, and interpretable (MASE < 1 means you beat the naïve baseline). But metric choice is the second problem. The first is backtesting design: a rolling-origin evaluation that faithfully simulates production conditions is the only reliable proxy for live performance. Fitting normalisation scalers on the full dataset including the test period is ubiquitous and inflates reported performance by 5-15% — not from overfitting the model, but from the preprocessing pipeline implicitly seeing the future.

Key points

• MAPE fails in three distinct ways. Division by zero: any series with zero values on some days breaks it — daily new user counts, SKU-level sales with stockouts, any count series. Asymmetric penalty: over-forecasting by 50% produces a larger MAPE contribution than under-forecasting by 50% — MAPE is minimised by systematically underforecasting, biasing model selection toward conservative forecasters even when over-forecasting is equally costly. Scale dependence: MAPE on a series with mean 10 versus mean 10,000 are not comparable, making cross-series performance comparison meaningless.
• MASE (Hyndman & Koehler 2006) solves all three MAPE failure modes. MASE = MAE / (MAE of in-sample naïve forecast). MASE < 1 means the model beats the naïve baseline on average. Scale-free: the normalisation makes MASE comparable across series with different scales. Symmetric: over- and under-forecasting of equal magnitude contribute equally. Defined when Y_t = 0: the denominator uses historical naïve errors which are well-defined even when actuals are zero. The M4 competition chose MASE as the primary metric — treat it as the default.
• Probabilistic forecast evaluation captures what point metrics miss. A point forecast of 100 with ±5 uncertainty and ±500 uncertainty have identical point forecast errors but completely different decision implications for inventory or capacity planning. Pinball (quantile) loss: L_q(y, ŷ) = q(y - ŷ) if y ≥ ŷ, else (1-q)(ŷ - y). Averaging over all quantiles gives CRPS — the gold standard for distributional forecasts. Winkler score penalises interval width plus extra penalty for actuals outside the interval, directly measuring calibration and sharpness.
• Diebold-Mariano test: statistically tests whether two forecasters have equal expected loss. Test statistic based on d_t = L(e₁_t) − L(e₂_t), using Newey-West standard errors for autocorrelated loss differentials. Both models must be evaluated on the same test set at the same horizon. For a portfolio of 1000 series: compute per-series MASE difference d_i, run a paired t-test or Wilcoxon signed-rank on {d_i}. Report effect size (median MASE difference) alongside p-value — a statistically significant difference of 0.001 MASE is operationally irrelevant.
• Rolling vs expanding window: expanding window uses all available history (training window grows over time) — correct for stationary DGPs where more data always helps. Rolling window uses a fixed training size — simulates non-stationary environments where only recent data is relevant. Rolling gives more evaluation cutoffs per total dataset length. Start with expanding; switch to rolling if model performance degrades systematically on more recent cutoffs (a signal of non-stationarity in the DGP).
• Lookahead bias is the most common way backtesting results fail to transfer to production. Four forms: (1) Using future covariate values at forecast time — the most obvious form. (2) Fitting normalisation (mean, std) on the full dataset including the test period — the scaler "knows" the future level. (3) Feature lag calculation errors without proper temporal shifts — feature at time t accidentally includes values from t+1. (4) Selecting the model with best test MAPE and reporting it as unbiased — the test set has been implicitly used for selection. The fix: strict temporal split, all preprocessing fitted on training data only, a validation set for model selection, and the test set touched exactly once for final unbiased evaluation.
• Lookahead bias magnitude: fitting a Z-score normaliser on training + test combined inflates performance because test-period values are included in the mean and standard deviation. If the test period has a higher mean (trending series), the full-dataset normalisation partially de-trends the test values, making the model appear more accurate than it would be in production. Empirically, this inflates reported MASE by 5-15% for non-stationary series. Always fit preprocessing on training data only and apply to test using training-fitted parameters.

Recap

• MAPE fails 3 ways: undefined at Y=0, penalises over- more than under-forecasting, not scale-comparable.
• MASE = MAE / naïve-MAE — scale-free, symmetric, defined at zero; <1 beats naïve. Default metric (M4).
• Probabilistic eval: pinball/quantile loss → CRPS; Winkler score penalises width + misses.
• Diebold-Mariano tests equal expected loss; across many series use paired t-test/Wilcoxon on per-series MASE.
• Effect size > p-value: a significant 0.001 MASE gap is operationally irrelevant.
• Expanding window (stationary DGP) vs rolling window (non-stationary); rolling gives more cutoffs.
• Lookahead bias inflates MASE 5-15%: fit ALL preprocessing on train only; touch test set exactly once.

Check your understanding