Bayesian Inference
Prior, likelihood, posterior, conjugate priors, MCMC
MLE and MAP give you a single best-guess set of parameters. But a single point estimate throws away everything you know about parameter uncertainty — and uncertainty is precisely what matters when data is scarce, when you need calibrated predictions, or when you are making sequential decisions. Bayesian inference maintains a full probability distribution over parameters: the posterior P(θ|data) ∝ P(data|θ)P(θ). This distribution captures what you know and what you do not know. The problem is the denominator: P(data) = ∫ P(data|θ)P(θ)dθ. This integral marginalises over all possible parameters — and in high dimensions it is almost never tractable. Conjugate priors are special cases where the posterior is in the same family as the prior, giving closed-form updates without any integration. When conjugacy fails, you have two options: MCMC samples from the posterior without computing the denominator by exploiting the fact that acceptance ratios cancel it out; variational inference approximates the posterior with a tractable family by minimising KL divergence. Both approaches trade exactness for tractability in different ways.
Key points
- Posterior ∝ likelihood × prior. The normalising constant P(data) = ∫ P(data|θ)P(θ)dθ is the marginal likelihood — rarely tractable because the integral is over all possible parameter values. MCMC exploits the fact that acceptance ratios in Metropolis-Hastings cancel this constant, making exact posterior sampling possible without computing it.
- Conjugate prior: Beta-Binomial, Dirichlet-Multinomial, Normal-Normal, Gamma-Poisson. Conjugacy gives closed-form sequential updates: observe data, update the hyperparameters algebraically. Beta(α, β) updated with k successes from n trials gives Beta(α+k, β+n−k). α and β act as pseudo-counts. Conjugate updates are the only Bayesian inference that scales to real-time streaming.
- Posterior predictive P(x_new|X) = ∫ P(x_new|θ)P(θ|X)dθ averages predictions over the full posterior rather than using a point estimate. It gives wider, more honest uncertainty than a MAP prediction. The posterior predictive is what a calibrated Bayesian model actually reports — not the mode of the posterior.
- MCMC: Metropolis-Hastings proposes θ' from a proposal q(θ'|θ) and accepts with probability min(1, P(θ'|X)q(θ|θ')/[P(θ|X)q(θ'|θ)]). The ratio of posteriors cancels the intractable denominator P(X). Diagnosis: R-hat ≈ 1 across multiple chains, effective sample size (ESS) large relative to chain length, trace plots that look like fuzzy caterpillars.
- Variational inference approximates the posterior P(θ|X) with a tractable distribution q(θ) by minimising KL(q‖P(θ|X)) = maximising the ELBO. It is faster than MCMC but biased: reverse KL causes q to collapse to a single mode of the true posterior, missing multimodality. VAEs use variational inference where the encoder is the approximate E-step.
Bayesian inference gives you a distribution over parameters, not a point. That distribution is the right answer when calibrated uncertainty matters — for small data, sequential updating, or uncertainty-aware decisions. The cost is that the posterior is almost never tractable in closed form, which is the entire reason MCMC and variational inference exist.
Recap
- Posterior ∝ likelihood × prior; it's a full distribution over params, not a point — captures uncertainty.
- The denominator P(data)=∫P(data|θ)P(θ)dθ is the intractable part — this is why MCMC and VI exist.
- Conjugate priors give closed-form updates: Beta(α,β)+k/n → Beta(α+k, β+n−k); α,β act as pseudo-counts.
- MCMC (Metropolis-Hastings) samples the posterior — the acceptance ratio cancels the intractable P(data).
- MCMC diagnostics: R-hat ≈ 1, large ESS, trace plots like fuzzy caterpillars.
- Variational inference approximates the posterior by minimising KL(q‖P) = maximising the ELBO — fast but mode-seeking.
- Posterior predictive $P(x_{new}|X)=\int P(x_{new}|\theta)P(\theta|X)d\theta$ averages over the posterior — wider, honest uncertainty.
Check your understanding
Q1. You have posterior P(θ|data) ∝ N(θ; 2, 1) × N(θ; 4, 1). What is the posterior distribution?
- A) The product of N(θ;2,1) and N(θ;4,1) gives N(θ; 3, 2) — averaging the means and summing the variances. Multiplying two Gaussians is treated like adding independent noise sources: variances add, σ*² = σ₁²+σ₂² = 1+1 = 2, and the mean is the arithmetic average, μ* = (μ₁+μ₂)/2 = (2+4)/2 = 3. The posterior is N(θ; 3, 2), with the pooled variance reflecting both sources of uncertainty.
- B) The product is N(θ; 6, 0.5) — means multiply and variances halve. When combining two evidence sources, the posterior has mean μ₁·μ₂/(μ₁+μ₂) and variance σ₁²σ₂²/(σ₁²+σ₂²) = (1·1)/(1+1) = 0.5. Because the likelihood and prior both push the estimate toward larger values, the posterior mean 6 reflects the combined, reinforcing evidence rather than an average of the two means.
- C) The product of two Gaussians N(μ₁,σ₁²) and N(μ₂,σ₂²) is N(μ*, σ*²) where σ*² = σ₁²+σ₂² = 2 and μ* = μ₁+μ₂ = 6. The posterior is N(θ; 6, 2). Multiplying probability densities is like multiplying their sufficient statistics directly: means add and variances add, mirroring how natural parameters of the Gaussian exponential family combine under a product of densities.
- D) Multiplying two Gaussians: P(θ|data) ∝ N(θ;2,1)·N(θ;4,1) ∝ exp(−[(θ−2)²+(θ−4)²]/2) = exp(−(θ−3)²/1) = N(θ; 3, 1/2). In general, N(μ₁,σ₁²)·N(μ₂,σ₂²) ∝ N(μ*, σ*²) where 1/σ*² = 1/σ₁²+1/σ₂² and μ* = σ*²(μ₁/σ₁²+μ₂/σ₂²). The posterior mean is a precision-weighted average, and it is precision (1/variance) that sums across Gaussians, not variance itself.
Q2. Which two of the following statements about approximate Bayesian inference methods are correct?
- A) MCMC constructs a Markov chain whose stationary distribution equals the posterior P(θ|data); the Metropolis-Hastings acceptance ratio min(1, P(θ'|data)/P(θ|data)) cancels the intractable normalising constant P(data), so the chain converges to the true posterior without ever computing that integral directly.
- B) Variational inference approximates the posterior with a tractable family q_φ(θ) by minimising the reverse KL divergence D_KL(q‖P(θ|data)), which is equivalent to maximising the ELBO; this scales to large models like VAEs but is biased and tends to collapse onto a single mode of a multimodal posterior.
- C) Both MCMC and variational inference require first computing the normalising constant P(data) = ∫P(data|θ)P(θ)dθ exactly before any sample can be drawn or any bound optimised — this shared prerequisite is exactly why both methods scale poorly past a few hundred parameters.
- D) Exact inference is intractable only when using non-conjugate priors — with any conjugate prior (Beta-Binomial, Normal-Normal, Gamma-Poisson), neither MCMC nor variational inference is ever necessary, because the posterior is always available in closed form regardless of model size.
Q3. What is a conjugate prior? Give one example and explain why conjugacy is computationally useful.
- A) A conjugate prior is a distribution from the same exponential family as the likelihood. Formally: if likelihood P(data|θ) is in the exponential family and prior P(θ) has the same base measure, the posterior is in the same exponential family. Example: Gaussian prior + Gaussian likelihood → Gaussian posterior. Computational usefulness: (1) Closed-form posterior via natural parameter updates. (2) Sequential updating: each observation updates the natural parameter vector additively. (3) The marginal likelihood is always the ratio of normalising constants — computable analytically, avoiding any numerical integration step.
- B) A conjugate prior is a distribution where the posterior has the same functional form as the prior: if prior P(θ) is in family F and posterior P(θ|data) ∝ P(data|θ)P(θ) is also in F, then F is conjugate to that likelihood. Example: Beta prior + Bernoulli likelihood → Beta posterior; Beta(α,β) updated with s successes in n trials gives Beta(s+α, n−s+β) — still a Beta distribution. Computational usefulness: (1) closed-form posterior, no numerical integration; (2) sequential updating — each observation just increments α or β; (3) analytic mean α/(α+β) and mode, no sampling or variational inference required.
- C) A conjugate prior is a prior that is invariant under the likelihood — P(θ|data) = P(θ) when data provides no information. The term 'conjugate' refers to the dual relationship between the prior and likelihood: the prior is conjugate if multiplying by the likelihood leaves the distribution in the same family. Example: any distribution is conjugate to a uniform likelihood — since the posterior equals the prior when the likelihood is flat. True conjugacy, where the posterior gets genuinely updated hyperparameters, occurs only for the Gaussian-Gaussian pair among common families.
- D) A conjugate prior requires the posterior to be a scaled version of the prior — P(θ|data) = c·P(θ) for some normalising constant c that depends only on data. This means conjugate priors are always proper (integrable) distributions because c is finite for any dataset. Example: Gamma prior + Poisson likelihood → Gamma posterior with updated rate and shape parameters. Computational usefulness: the update rule is c = P(data), the marginal likelihood, which provides a free estimate of model evidence at no extra computational cost.
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →