Why Your Forecast Was Wrong Before It Ran: The 8 Silent Killers
A time series model that looks excellent in backtesting routinely fails in production. The backtest metric was real — the error was introduced before the model ever ran. Most forecast failures are data and evaluation failures, not model failures. Here are the 8 patterns that kill forecasts before deployment.
Forecasting failures are mostly diagnosed wrong. When a forecast misses badly, the instinct is to try a different model: ARIMA instead of Prophet, LSTM instead of ARIMA. Usually that's the wrong fix. The model wasn't the problem.
Here are the 8 patterns that corrupt forecasting pipelines before the model gets involved.
1. Target leakage in temporal features
You're predicting sales tomorrow. One of your features is "average sales over the past 7 days" — computed using the 7 days before the prediction date. In training, you compute this lazily using the full history. In production, the pipeline that materialises this feature runs at 06:00 UTC, including transactions that came in at 23:55 the previous night.
Result: training features are computed with a slightly different time boundary than production features. The model learns patterns that don't exist in the production data. Your MAPE looks fine in backtest; it degrades 15% in production.
```python import pandas as pd from sklearn.linear_model import Ridge
def make_temporal_features(df, target_col, lag_days=[7, 14, 28]): """Safe temporal feature engineering — no future leakage.""" df = df.sort_values('date').copy() for lag in lag_days: # shift(lag) ensures we only use data from lag days ago df[f'target_lag_{lag}d'] = df[target_col].shift(lag) # Rolling mean must be shifted by 1 to avoid using today's value df['rolling_mean_7d'] = df[target_col].shift(1).rolling(7).mean() return df
# Common mistake — leaks the target into features: # df['rolling_mean'] = df[target_col].rolling(7).mean() # BUG: uses today # df['lag_1'] = df[target_col].shift(0) # BUG: is today
# Safe train/test split for time series — never shuffle cutoff = pd.Timestamp('2024-01-01') train = df[df['date'] < cutoff] test = df[df['date'] >= cutoff] # strictly after — no overlap ```
2. Non-stationarity ignored at training time
Most classical forecasting methods assume stationarity: the statistical properties of the series (mean, variance, autocorrelation) don't change over time. Most real-world series aren't stationary. Retail sales trend upward. Energy consumption has decade-long cycles. Advertising spend has quarterly budget patterns.
If you fit an ARIMA to a non-stationary series without differencing, the model is fitting to the trend rather than the patterns. It will forecast the trend to continue — and mean-revert when it doesn't.
Always test stationarity with ADF or KPSS before model selection. Always apply the differencing order that makes the series stationary before passing it to a classical model.
3. Structural breaks treated as noise
A structural break is a permanent shift in the level or trend of a series — a new market entrant, a product line change, a regulation, a macro event. Models trained before the break will forecast as if the old regime continues. Models trained after the break may not have enough data to characterise the new regime.
Prophet has built-in changepoint detection. For other models: detect breaks with the Chow test or Bai-Perron algorithm. Segment your training data at the break point — data before the break is either excluded or given lower weight.
4. Evaluation metric mismatch
RMSE penalises large errors heavily. MAPE breaks when the actual is near zero. MAE treats all errors equally regardless of scale. Symmetric MAPE (SMAPE) has its own pathologies.
The right metric depends on your use case: if an underforecast and overforecast have equal cost, MAE. If large errors are catastrophically costly (safety-critical applications), RMSE. If you care about percentage errors and your series never goes to zero, MAPE.
The mistake: optimising RMSE in training when operations cares about MAPE, or reporting MAPE on a series with near-zero values where the denominator explodes.
5. Look-ahead bias in the evaluation window
Your backtest generates forecasts for each week using a model fit on all prior data. But did you re-fit the model at each step of the walk-forward validation, or did you fit it once on the full history and generate "forecasts" with the model that already saw the future?
A model fit once on full history then evaluated over historical windows has already seen the "future" of those windows during training. Your backtest numbers are not honest. Always use expanding window or rolling window cross-validation where the model is genuinely blind to future data.
6. Cold start on sparse series
Your inventory model works well for your top-100 SKUs. It fails for the 4,000 long-tail SKUs with 2–3 transactions per month. Classical time series models need at least 2–3 seasonal cycles of history to characterise seasonality. For sparse series, fitting a model per series is both computationally wasteful and statistically unsound.
Solutions: hierarchical forecasting (borrow statistical strength from similar series), global models (one neural network trained on all series simultaneously), or default-to-category-average for series below a sparsity threshold.
7. Feature pipeline drift after deployment
Your forecast model uses 6 external features (weather, economic indicators, competitor prices). These features are fetched from third-party APIs. One API changes its response schema in month 3. Your feature pipeline silently fills that feature with NULL, which gets imputed to the mean. The model continues running and producing numbers — slightly wrong ones, for the next 8 months, until a quarterly review catches it.
Validation: apply schema checks and null-rate monitoring to every external feature, not just the model's predictions. Alert if null rate for any feature exceeds 2x its training baseline.
8. Not accounting for intermittency
A series that is zero 70% of the time (a product that doesn't sell most days) should not be forecast with a model designed for continuous data. Standard RMSE will be dominated by the zero periods. Standard models will forecast small positive values for the zero periods (systematic positive bias).
Correct approach: Croston's method (separate demand interval and demand size models), or zero-inflated distributions, or — for very sparse series — just forecast the probability of a non-zero period and the expected size conditional on non-zero.
The meta-pattern
Six of these eight failures are detectable before you train any model: by auditing the feature computation logic, checking stationarity, verifying your evaluation methodology, and profiling your series for sparsity and structural breaks. Spend 30% of your forecasting project on this audit before touching model selection. You'll find problems that no model architecture can fix.