Anomaly Detection: Isolation Forest, Autoencoders, and Statistical Baselines

Anomaly detection is the umbrella problem: fraud, network intrusion, industrial equipment failure, rare disease, data pipeline errors. The right algorithm depends on the data type (tabular, time series, image), the anomaly definition (global outlier, local outlier, contextual outlier), and the label availability. Isolation Forest works when you have no labels. Autoencoders work when structure is complex. Statistical baselines work when interpretability matters.

Anomaly detection is inherently asymmetric: anomalies are rare by definition, often have no examples to learn from, and may be structurally novel — different from anything in the training set. This rules out standard supervised classification and motivates unsupervised and semi-supervised approaches.

The three types of anomalies

Global outlier: a point that is extreme relative to the entire dataset. A transaction of $100,000 when typical transactions are $50-200. Point anomaly in multivariate space. Local outlier: a point that is anomalous relative to its neighbourhood but not globally. A temperature of 40°C is normal in July but anomalous in January. Contextual anomaly. Collective anomaly: a sequence of points that is anomalous only together. Individual steps of a complex fraud that are each normal individually but the combination is suspicious.

Statistical baselines: where to start

Z-score: flag points more than k standard deviations from the mean. Assumes Gaussian distribution. Fast, interpretable, fails for skewed distributions. IQR method: flag points below Q1 - 1.5*IQR or above Q3 + 1.5*IQR. Robust to outliers (IQR itself is resistant to extreme values). For multivariate data: Mahalanobis distance generalises the z-score to multiple dimensions using the covariance structure: d(x) = sqrt((x - μ)^T Σ^-1 (x - μ)). Points with large Mahalanobis distance are multivariate outliers. Assumes elliptical distribution; computed from SVD or LU decomposition.

Isolation Forest

Isolation Forest (Liu et al., 2008) exploits the observation that anomalies are isolated: they require fewer random binary splits to separate from the rest of the data than normal points. Algorithm: build an ensemble of random isolation trees. Each tree recursively partitions the data by randomly choosing a feature and a random split value. The anomaly score for a point is the average path length across all trees to isolate it — shorter path = more anomalous.

Advantages: works on raw tabular features with no distributional assumptions, scales to large datasets (each tree is O(n log n)), requires no labels. Hyperparameters: n_estimators (more = more stable), contamination (expected anomaly fraction, used to set the threshold). Weakness: fails on high-dimensional data where random splits lose their discriminative power, and struggles with local outliers (anomalies that are only unusual in a small region of the space).

Local Outlier Factor (LOF)

LOF (Breunig et al., 2000) computes the local density of each point relative to its k-nearest neighbours. Points in lower-density regions than their neighbours receive high LOF scores. LOF captures local anomalies that Isolation Forest misses. Disadvantage: O(n^2) for naive implementation; expensive on large datasets.

Autoencoder-based anomaly detection

Train an autoencoder (see Post 62) to reconstruct normal data. Anomalies reconstruct poorly — high reconstruction error. This approach works well for: high-dimensional data (images, time series, logs) where statistical distances are meaningless, detecting novelty rather than statistical outliers, and settings with semi-supervised labels (train on clean normal data, detect deviations).

Reconstruction error threshold is set on a validation set of normal data (e.g., flag the top 1% highest reconstruction error at validation time). The autoencoder approach extends naturally to VAEs (reconstruction error + KL divergence as the anomaly score) and to time series (LSTM autoencoders reconstruct sequences; anomalous windows have high error).

One-Class SVM

Trains a boundary around the normal data in feature space. Points outside the boundary are anomalies. Kernel trick allows nonlinear boundaries. Scales poorly with dataset size (O(n^2) kernel matrix). Generally dominated by Isolation Forest for tabular data but useful when a tight, nonlinear boundary is needed.

Evaluation without ground truth

Anomaly detection without labels requires indirect evaluation: inject synthetic anomalies into held-out data and measure detection rate; use domain expert review of flagged cases (precision); measure coverage of known anomalies from historical incident reports. When labels are available for a test set: AUC-ROC (class-imbalanced), precision-recall AUC (more informative for rare anomalies), and AUCPR (area under precision-recall curve).

Try on Colab: use the KDD Cup 1999 network intrusion dataset. Train Isolation Forest and LOF on the normal traffic subset. Evaluate AUC-ROC on the test set including both normal and attack traffic. Then train an autoencoder on normal traffic, compute reconstruction error, and set the threshold at the 99th percentile of validation error. Compare all three methods' precision and recall at the same operating point.