Gaussian Mixture Models
EM algorithm, soft assignments, model selection with BIC
You have per-transaction purchase amounts x. Some transactions are 5 dollars (app purchases), some are 50 dollars (subscriptions), some are 500 dollars (enterprise). The distribution of purchase amounts is trimodal — three distinct subpopulations. K-means would split them by which of 3 centroids is nearest. But a single $26 transaction sits between the $5 and $50 clusters — does it belong to the 5-dollar cluster or the 50-dollar cluster? GMM computes a responsibility for each component, worked out below.
Gaussian Mixture Model:
$P(x) = Σₖ πₖ N(x | μₖ, Σₖ)$
where πₖ is the mixing weight (Σπₖ = 1), N(x | μₖ, Σₖ) is the k-th Gaussian component evaluated at a single scalar transaction amount x — not an aggregate over a customer's several purchases. Responsibility for component k at a point x is:
$r_k(x) = πₖN(x|μₖ,Σₖ) / Σⱼ πⱼN(x|μⱼ,Σⱼ)$
Worked example: take the $5 cluster as (π=0.45, μ=5, σ=8) and the $50 cluster as (π=0.35, μ=50, σ=12). At x=26:
$N(26|5,8) ≈ 0.00159 and N(26|50,12) ≈ 0.00450$
so πₖN(x|μₖ,Σₖ) gives:
$0.45×0.00159 ≈ 0.00072 (the $5 cluster) and 0.35×0.00450 ≈ 0.00157 (the $50 cluster)$
Normalizing those two:
$0.00072/(0.00072+0.00157) ≈ 31% and 0.00157/(0.00072+0.00157) ≈ 69%$
— gives responsibilities of about 31% for the $5 cluster and 69% for the $50 cluster for that one $26 transaction. A customer with several transactions gets one responsibility vector per transaction, not a single blended number for the customer.
Fitted via EM, which alternates two steps until log-likelihood stops improving: the E-step computes every point's responsibility r_k(x) for every component using the formula above; the M-step then re-estimates πₖ, μₖ, Σₖ as the responsibility-weighted mixing fraction, mean, and covariance over all points. Each E/M pair provably does not decrease the log-likelihood, but the landscape is multimodal, so EM only hill-climbs to whichever local optimum is nearest its starting point.
Covariance types control cluster shape: full (each component has a different Σₖ — arbitrary ellipsoids), tied (all components share one Σ), diag (Σₖ is diagonal — axis-aligned ellipsoids, fewer parameters), spherical (Σₖ = σₖ²I — each component is K-means-like). Full covariance is most expressive but requires the most parameters. Spherical alone is not enough to equal soft K-means — it also has to be tied (one shared σ² across every component, not a separate σₖ² per component); spherical-and-tied covariance is equivalent to soft K-means, and as σ→0 it reduces further to hard K-means's plain Euclidean nearest-centroid rule. Untied (per-component) spherical covariance still lets each cluster use its own scale, so it does not collapse to that rule.
Bayesian Information Criterion for model selection: BIC = k·log(n) - 2·log(L̂). Lower is better. Penalizes model complexity. Fit GMMs with K=1 to 20, plot BIC, take the minimum.
NOT-this: "GMM is just soft K-means." Soft K-means is GMM with spherical, tied covariance (one shared σ² across every component, not per-component σₖ²) and equal mixing weights. Full GMM with full covariance can fit elliptical clusters of any orientation and size — qualitatively different from K-means geometry. The covariance matrix is the key structural difference.
Key points
- Use BIC to select the number of components — fit K=1 to 20, take the K with minimum BIC. This is more principled than the elbow method and penalizes overfitting. If BIC keeps decreasing past K=20, your data does not have well-defined Gaussian components — try HDBSCAN or examine whether a distributional assumption is appropriate at all.
- Trap: initializing GMM randomly on high-dimensional data leads to component collapse — a component gets assigned 0 weight and disappears. Use K-means to initialize the component means before running EM. Sklearn's GaussianMixture does this by default with init_params=`'kmeans'`. Multiple restarts (n_init > 1) further reduce the risk of degenerate solutions.
- Diagnostic: plot the learned Gaussian components as ellipses over the data. If components heavily overlap (high uncertainty for all points), K is too large. If data points lie clearly between component centers with low probability under all components, K is too small. Soft assignment probabilities near 0.5 across all components for most points indicate that the Gaussian assumption may not match your data geometry.
GMM lifts three K-means restrictions at once — soft assignments, elliptical clusters, log-likelihood as the fit criterion — but EM only finds a local optimum, so run multiple restarts and use BIC to choose K.
Recap
- Soft probabilistic assignment: $P(x)=\Sigma\pi_k\,N(x|\mu_k,\Sigma_k)$, fitted by EM.
- Lifts three K-means limits: soft membership, elliptical clusters, log-likelihood as fit criterion.
- EM alternates E-step and M-step: E-step computes each point's responsibility $r_k(x)=\pi_k N(x|\mu_k,\Sigma_k)/\Sigma_j\pi_j N(x|\mu_j,\Sigma_j)$ per component; M-step re-estimates $\pi_k,\mu_k,\Sigma_k$ as the responsibility-weighted mixing fraction/mean/covariance.
- Covariance type controls shape: full (any ellipsoid), tied, diag, spherical (spherical + tied = soft K-means).
- EM only finds a local optimum — use multiple restarts.
- Select K by BIC ($k\log n - 2\log\hat{L}$, lower better), fit K=1–20, take the minimum.
- Init means with K-means to avoid component collapse (sklearn default).
- Overlapping ellipses = K too large; points between centers = K too small.
Check your understanding
Q1. EM for GMM is guaranteed to not decrease log-likelihood at each step, yet it often converges to a poor local optimum. Which two of the following correctly explain this?
- A) The guarantee is about local monotonicity, not global optimality — the log-likelihood landscape is genuinely multimodal
- B) EM hill-climbs to the nearest local maximum from its starting point, so run multiple restarts and keep the best result
- C) The guarantee is actually incorrect — EM can decrease log-likelihood whenever covariance matrices become singular
- D) Poor local optima only occur with diagonal covariance; switching to full covariance guarantees the global optimum always
Q2. You fit a GMM with K=5 and diagonal covariance on 10,000 points in 50 dimensions. BIC keeps decreasing as you increase K from 1 to 20. What does this suggest and what do you do?
- A) BIC always decreases monotonically with K in every case — the correct stopping rule is when AIC and BIC finally disagree
- B) The data genuinely has more than 20 natural groups here — always keep increasing K until BIC itself starts increasing
- C) Switch to full covariance instead — diagonal covariance is causing BIC to badly underestimate its penalty term here
- D) True cluster count may exceed 20, or non-Gaussianity needs many parts — try PCA first, remove outliers, or use HDBSCAN
Q3. A customer segmentation GMM has a component with mixture weight π_k = 0.001 that collapses to a tiny covariance (singular matrix). What happened and how do you fix it?
- A) This is normal, expected behaviour — components with small mixture weights always have small covariances by definition
- B) The component latched onto a few isolated points and drove its covariance to zero — fix with regularisation or Bayesian GMM
- C) Increase the number of EM iterations further — degenerate components always resolve themselves eventually with more training
- D) Switch to full covariance — diagonal covariance always produces degenerate components whenever mixture weights are small
Q4. How do you use GMM for density estimation and anomaly detection? What determines the anomaly threshold?
- A) GMM anomaly detection strictly requires labelled anomalies to set any threshold — it cannot ever operate fully unsupervised
- B) Use the number of mixture components as the threshold — points in components with under 5% mixture weight are anomalies
- C) Compute log p(x) per sample after fitting; flag very low log-likelihood; set threshold by percentile or labelled data
- D) GMM density estimation only ever works for anomaly detection when K=1 — multiple components make the threshold ambiguous
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →