Neural Forecasting
N-BEATS, N-HiTS, TFT, PatchTST, when transformers lose to MLPs, foundation models
Electricity demand forecasting is the benchmark problem for neural time series methods: 1-year history, predict 24 hours ahead. A sequence of Transformer-based papers — Informer, Autoformer, FEDformer — each claimed state-of-the-art on this benchmark. Then a 2023 paper (Zeng et al.) showed that a single linear layer applied to the flattened lookback window outperforms all of them. The reason exposes a structural flaw: Transformer attention is permutation-equivariant. Attention scores are computed from pairwise content similarity — dot products of embeddings — not from temporal position. Shuffle the timestamps and performance barely changes. A model that ignores temporal order cannot model autocorrelation, trend, or seasonality.
Positional encodings are added but don't fix this — they make position part of the content, which still allows position-insensitive mixing.
N-BEATS and N-HiTS sidestep this entirely. N-BEATS uses MLP stacks with a doubly-residual architecture: each block produces a backcast (reconstruction of its input) and a forecast; the backcast is subtracted before the next block, so each block models only what previous blocks couldn't explain. N-HiTS extends this with hierarchical multi-rate sampling — a coarse stack captures trends from downsampled series, a fine stack captures high-frequency variation — which is the right inductive bias for the 24-hour-ahead problem where trend and daily cycle operate at different timescales.
The Temporal Fusion Transformer (TFT) is competitive despite using attention because it preserves temporal structure everywhere else. An LSTM encoder processes the sequence before attention, variable selection networks gate which features matter per timestep, and attention operates on LSTM outputs rather than raw timestamps. TFT wins when rich covariates exist — static entity features, known future inputs — and N-HiTS wins without them.
NOT this. "Transformer-based models always outperform ARIMA and classical methods." N-BEATS, TFT, and other deep learning methods win on long-horizon multi-step forecasting with many series. On short, well-behaved univariate series with fewer than 2 years of history, ARIMA with seasonal decomposition frequently wins. The M4 and M5 competitions showed that feature-engineering with LightGBM beats most neural methods on the aggregate leaderboard. The right question is not which neural architecture to use but whether any neural model beats a well-tuned statistical baseline — that comparison must include DLinear.
Key points
- Transformer self-attention is permutation-equivariant: it computes scores from content similarity, not temporal position, so shuffling timestamps barely changes performance on ETT benchmarks. This is why DLinear — a single linear layer on the flattened lookback window — outperforms Informer, Autoformer, and FEDformer. The architectural complexity of these Transformers does not compensate for discarding temporal order. DLinear is the mandatory baseline for any neural forecasting paper claim. If a proposed model does not beat DLinear, the complexity is not justified.
- N-BEATS's doubly-residual architecture gives each MLP block a smaller, well-defined task: model only what prior blocks failed to explain. Each block produces a backcast (reconstruction of its input window) and a forecast. The backcast is subtracted via a residual connection before the next block sees the input — subsequent blocks work on harder residuals. N-HiTS extends this with hierarchical multi-rate sampling: coarse stacks downsample the input to capture slow trends; fine stacks operate at full resolution for high-frequency patterns. N-HiTS outperforms N-BEATS on horizons beyond 96 steps and handles the daily-plus-weekly double seasonality of the electricity problem naturally.
- Foundation models (TimeGPT, MOIRAI) win exactly one scenario: cold start with fewer than 30 observations per series. With 2+ years of in-domain history, local statistical or global neural models trained on your data dominate zero-shot foundation models. Foundation models are calibrated to their pretraining distribution — pharmaceutical sales patterns (approval spikes, generic entry cliffs), industrial sensor data, or niche domain series that differ from the training corpus show degraded zero-shot performance. The global-vs-local decision is separate: global models (N-BEATS, TFT) train across series jointly and benefit from cross-series patterns, but only when series are related enough for transfer to help.
Transformer self-attention ignores temporal order by design, which is why DLinear outperforms Informer on standard benchmarks — always include DLinear as a baseline before claiming any neural forecasting win. N-HiTS retains temporal inductive bias through hierarchical multi-rate MLP stacks and beats N-BEATS on long horizons; TFT is competitive only when rich covariates exist. Foundation models win exactly one scenario: cold start with fewer than 30 observations per series.
Recap
- Transformer attention is permutation-equivariant — ignores temporal order; shuffling timestamps barely changes MSE.
- DLinear (one linear layer) beats Informer/Autoformer/FEDformer — mandatory baseline before any neural claim.
- N-BEATS doubly-residual: each block backcasts + forecasts, subtracts before next → smaller task per block.
- N-HiTS adds hierarchical multi-rate sampling — beats N-BEATS beyond 96-step horizons, handles double seasonality.
- TFT competitive only with rich covariates.
- Foundation models (TimeGPT, MOIRAI) win exactly one case: cold start, <30 obs/series.
- With 2+ years of in-domain data, local/global models trained on your data dominate zero-shot.
Check your understanding
Q1. A colleague reports Informer beats ARIMA by 15% lower MSE on ETT. Which TWO checks are appropriate before trusting this result?
- A) Compare against DLinear, which Zeng et al. showed outperforms Informer on the ETT benchmark, and verify both models use identical lookback windows and forecast horizons in the comparison.
- B) Confirm the test split was not used anywhere during Informer's hyperparameter tuning; leakage from the test set into tuning would inflate the reported 15% MSE improvement over ARIMA artificially.
- C) Accept the result immediately as-is — a 15% MSE reduction is large enough to be practically significant on its own and unlikely to be explained away by evaluation methodology differences at all.
- D) The Informer result is simply impossible — Zeng et al. definitively proved Transformer architectures can never outperform linear models on any time series benchmark, so the colleague must have erred.
Q2. You need to forecast hourly electricity demand for 1000 substations, 168 hours ahead (one week). You have 3 years of historical data per substation. TFT vs N-HiTS vs SARIMA — which do you use and why?
- A) SARIMA per-substation is the correct choice here; 3 years of hourly data provides more than sufficient history for reliable parameter estimation, and statistical methods stay more interpretable for utility operators.
- B) TFT is always the single best choice for multi-step forecasting with multiple series, because its variable selection network automatically identifies the most informative features without any manual engineering.
- C) N-HiTS is the strongest start: hierarchical pooling handles the 168h horizon well, a global model beats 1000 SARIMAs, SARIMA can't handle double seasonality. Use TFT with covariates; keep a SARIMA baseline.
- D) Ensemble all three approaches equally; with 3 years of data per substation the computation cost of fitting all three models is negligible, and an equal-weight ensemble always outperforms any individual model chosen alone.
Q3. You train a TimeGPT zero-shot model on a new domain (pharmaceutical sales) without fine-tuning. It performs worse than Holt-Winters. When would you expect a foundation model to outperform statistical methods, and what limits them?
- A) Foundation models always outperform statistical methods given enough compute budget; the underperformance observed here indicates a bug in the TimeGPT API call rather than any genuine model limitation.
- B) Outperform when: cold start (<30 obs), heterogeneous portfolios, or domain well-represented in pretraining. Here: pharma patterns differ from pretraining; 3yr data favors Holt-Winters instead.
- C) Foundation models outperform only on strictly univariate series; with multivariate pharmaceutical sales data the zero-shot performance will always be structurally degraded compared to univariate statistical methods.
- D) Fine-tune TimeGPT on the full 3 years of pharmaceutical data; zero-shot performance is always poor by design, but after fine-tuning foundation models uniformly outperform statistical methods regardless of domain.
Q4. What is the doubly residual architecture in N-BEATS and why does it improve forecasting over a plain deep MLP?
- A) Doubly residual means N-BEATS uses two separate residual streams — one for an AR component and one for an MA component — directly analogous to ARMA, which is why it outperforms plain MLPs on time series.
- B) Doubly residual refers to applying dropout twice per block, once after each fully-connected layer, which provides stronger regularisation than standard single-dropout MLPs and reduces overfitting on short series.
- C) Each block produces a backcast b̂_t and forecast f̂_{t+h}; the backcast is subtracted before the next block (x_{t+1}=x_t−b̂_t), forecasts sum across blocks. Each block only learns what prior blocks couldn't explain.
- D) Doubly residual means N-BEATS applies a residual connection at both the block level and the stack level; the block connection handles short-range patterns while the stack handles long-range trends instead.
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →