The Forecast Failure Zoo: Six Silent Killers of Time Series Models
Time series models fail in ways that break silently. ARIMA assumes stationarity — it doesn't fail; it forecasts a trend that reverses. Prophet assumes additive seasonality — it doesn't fail; it underfits multiplicative patterns. LSTM assumes sufficient data — it doesn't fail; it memorises noise. Here are the six failure modes, the production signals that indicate each, and how to catch them before deployment.
Forecasting models don't fail loudly. They fail quietly, producing plausible numbers that are systematically wrong. The six failure modes below are the ones that kill forecasting projects in production.
1. Seasonality assumption violations — ARIMA assumes stable seasonality
ARIMA learns the seasonal pattern from historical data and assumes it will persist. When seasonality shifts — amplitude changes, frequency changes, or seasonality disappears entirely — ARIMA extrapolates the learned pattern blindly.
Production signal: your model forecasts with consistent MAPE during training, but its error spikes at the onset of a new season. January is typically high-volume, but a pandemic year shows flat demand. The model forecast January as high; actual is flat.
Detection: run autocorrelation analysis (ACF) on the residuals. If residuals show the same seasonal pattern the model was supposed to capture, seasonality is changing.
Fix: don't assume seasonality is stable. Use Prophet with `yearly_seasonality=False` for series where annual seasonality is expected to shift. Better: use a rolling forecasting window and retrain monthly.
2. Non-stationarity ignored — the model fits the trend instead of the pattern
ARIMA assumes the series is stationary (after differencing). If you fit ARIMA to a non-stationary series without proper differencing, the model fits the overall trend rather than the deviations around the trend.
Production signal: your forecast follows the historical trend perfectly through the holdout period, then diverges. If historical data was rising, the model forecasts continued rise — even when the series mean-reverts.
Detection: ADF test (Augmented Dickey-Fuller) on the residuals. If p > 0.05, residuals are non-stationary — your model didn't capture the non-stationarity correctly.
Fix: difference the series until it passes the ADF test (p < 0.05). The number of differences required becomes the d parameter in ARIMA(p,d,q).
3. Structural breaks treated as noise — when the regime changes permanently
A structural break is a permanent shift in the level, trend, or pattern of a series. New market entrant. Product launch. Regulation change. Macro event. Models trained before the break will forecast as if the old regime continues.
Production signal: your model's forecast diverges from actuals immediately after a structural break. Mean Absolute Percentage Error jumps 30%+. The break is unambiguous in hindsight but invisible during backtest because your training set didn't include it.
Detection: use the Chow test or visual inspection. Plot your series with a sliding window of recent data. If the recent distribution looks different from the long-term distribution, a break may have occurred.
Fix: retrain on post-break data only. Alternatively, use Prophet, which has built-in changepoint detection and can weight pre-break data lower. Or use a structural time series model that explicitly models breaks.
4. Evaluation metric mismatch — optimising RMSE when operations cares about MAPE
RMSE penalises large errors heavily. MAPE penalises percentage errors. MAE treats all errors equally. If you optimise one metric but operations monitors another, you've solved the wrong problem.
Production signal: your model reports excellent RMSE in backtest, but operations says it's missing large absolute errors by 20% or more.
Detection: compute multiple metrics on the same holdout set. If RMSE and MAPE rankings of candidate models are different, the metric choice matters for your use case.
Fix: choose the metric that aligns with business costs. If missing a 100-unit spike is 10× worse than missing a 10-unit spike, use RMSE. If missing by 20% is uniformly costly, use MAPE. Define the metric before backtesting, not after.
5. Look-ahead bias in walk-forward validation — the model already saw the future
A common backtest mistake: fit the model once on all historical data, then generate "forecasts" on historical windows. The model has already seen those windows during training. Your backtest is not honest.
Production signal: your backtest MAPE is 3%, production MAPE is 12%. No model change, no data change. The backtest was measuring how well the model remembers the past, not how well it predicts the future.
Detection: plot your forecasts against actuals using expanding window cross-validation. If the forecast lags the actual by exactly the seasonal period, you're not forecasting — you're copying last season's values.
Fix: use expanding or rolling window cross-validation. At each step, fit on all data up to time T, forecast forward to T+H, then move T forward by one period and repeat. This is the only honest backtest.
6. Insufficient data for seasonality capture — LSTM memorises noise instead
Neural forecasting models need thousands of observations to learn seasonal patterns. A 2-year time series with daily data (730 observations) has only 2 full annual cycles. An LSTM trained on this will overfit to noise rather than learn the seasonal pattern.
Production signal: your LSTM trains to near-zero loss on the training set, but its forecast on a fresh holdout year is 10× worse than a simple Prophet model. The LSTM has memorised the training years, not learned a generalizable pattern.
Detection: check your data volume. Count complete seasonal cycles. < 2 cycles: seasonal models are risky. < 3: validate heavily. 5+ cycles: seasonal models are appropriate.
Fix: start with simpler models (ARIMA, Prophet) that have explicit seasonal components and lower data requirements. Reserve neural models for scenarios with thousands of observations and complex nonlinear dependencies.
The production checkpoint:
Before deploying a forecast model: (1) Plot the series and identify obvious structural breaks or seasonality changes. (2) Run stationarity tests. (3) Backtest with a proper walk-forward validation where the model never sees future data. (4) Verify your evaluation metric aligns with what operations actually cares about. (5) Validate that your holdout MAPE is within 20% of your cross-validation MAPE. If not, you have a problem — likely one of the six above.
Practice this in Time Series to diagnose failures and apply the right preventive checks for your data.
```python import numpy as np
def walk_forward_cv(series, model_fn, horizon=7, min_train=90): """Walk-forward validation — the only honest time series backtest. Never lets the model see future data. Each fold: fit on [0:t], predict [t:t+h].""" errors = [] for t in range(min_train, len(series) - horizon): train = series[:t] actual = series[t : t + horizon] model = model_fn(train) pred = model.predict(horizon) mape = np.mean(np.abs((actual - pred) / (np.abs(actual) + 1e-9))) * 100 errors.append(mape)
cv_mape = np.mean(errors) # Sanity check: if holdout MAPE >> cv_mape, you likely have leakage or non-stationarity return {'cv_mape': round(cv_mape, 2), 'n_folds': len(errors)}
# Red flag: cv_mape=4.2% but holdout_mape=38.1% → one of the 6 failure modes above ```