Bayesian Inference

Likelihood, prior, posterior, conjugate priors, predictive distribution, sequential updating

You flip a coin 3 times and get H, H, H. The frequentist MLE gives P(H) = 3/3 = 1.0 — the model is certain the coin always lands heads. But you only have 3 flips. The model has fit the data perfectly and is telling you something obviously wrong. This is what happens when you collapse inference to a single point estimate without tracking uncertainty.

Bayesian inference solves this by maintaining a full distribution over the unknown parameter rather than collapsing to one value. The update rule is: posterior ∝ likelihood × prior. With a uniform prior Beta(1, 1) — encoding no prior knowledge about the coin — and likelihood P(data | θ) = θ³, the posterior is Beta(4, 1).

The posterior mean is 4/5 = 0.8, not 1.0. The prior has pulled the estimate away from the degenerate MLE, encoding the reasonable belief that most coins are somewhere near fair.

The key mechanism: the prior's influence is inversely proportional to the amount of data. With 3 flips, the posterior is 0.8 — meaningfully different from MLE. With 300 flips and 300 heads, the posterior mean is 301/302 ≈ 0.99 — nearly identical to MLE. The likelihood dominates and the prior washes out. This is the correct behavior: priors matter when data is scarce and become irrelevant when data is abundant.

The practical cost of this framework is the denominator in Bayes' theorem: p(θ | X) = p(X | θ) p(θ) / p(X), where p(X) = ∫ p(X | θ) p(θ) dθ requires integrating the likelihood over the entire parameter space. For conjugate models, this integral is analytic. For everything else — which is most real models — it has no closed form, and the entire ecosystem of approximate inference methods (MCMC, variational inference, Laplace approximation) exists to avoid or approximate this integral.

NOT this. "Bayesian methods are computationally intractable." For conjugate prior-likelihood pairs, the posterior is analytic — no integration required. Beta + Binomial, Gaussian + Gaussian, Dirichlet + Categorical all yield closed-form posteriors. For non-conjugate problems, variational inference reframes the posterior computation as an optimization problem, and MCMC constructs a Markov chain whose stationary distribution is the posterior. Bayesian deep learning with MC Dropout adds a single forward pass with dropout active at test time. The intractability is real but narrower than it appears: it applies only to the exact normalizing constant, and every major approximate inference method has a principled strategy for working around it.

Key points

• Bayes' theorem: p(θ|X) = p(X|θ)p(θ) / p(X). The normaliser p(X) = ∫ p(X|θ)p(θ)dθ requires integrating the likelihood over the entire parameter space, which has no closed form for most real models. This is why posterior ∝ likelihood × prior is the workable form — the intractable constant drops out. It also why every approximate inference method (MCMC, VI, conjugate priors) exists: each is a different strategy for avoiding or approximating that integral.
• Likelihood p(X|θ) is a function of θ given fixed data — it scores how well θ explains what you observed. It is not a probability distribution over θ and does not integrate to 1 over θ. Conflating likelihood with probability over parameters leads to a specific error: treating the parameter value that maximises the likelihood as the most probable one, ignoring the prior. MLE is valid; calling the MLE the "most probable" parameter is Bayesian reasoning without the prior.
• Conjugate priors are a computational shortcut: choose a prior from a family where the posterior stays in the same family after multiplying by the likelihood, giving closed-form updates with no integration. Beta(α,β) + Binomial data → Beta(α + successes, β + failures). Gaussian prior on μ + Gaussian likelihood → Gaussian posterior. Dirichlet + categorical → Dirichlet. This convenience comes at a cost: the conjugate family constrains what shapes your prior can take, which may not match your actual beliefs — the mathematical convenience is real but the resulting prior may not be.
• The correct Bayesian predictive distribution integrates over the full posterior: p(x*|X) = ∫ p(x*|θ)p(θ|X)dθ. Plugging in the MAP estimate instead — the common shortcut — treats a point estimate as if it were certain, systematically underestimating predictive uncertainty. In low-data regimes, this underestimate is large enough to make wrong decisions. The correct predictive distribution is wider, especially where the posterior is spread out.
• Bayesian A/B testing with Beta-Binomial: after observing conversions, p_A and p_B have Beta posteriors. P(A beats B) = ∫∫ 1[p_A > p_B] p(p_A|data) p(p_B|data) dp_A dp_B — computable analytically or by Monte Carlo. This is a direct probability statement about which variant is better, not a p-value. You stop when P(A > B) clears your decision threshold (e.g., 95%). No null hypothesis needed — the question is framed exactly as "how likely is it that A is better?"
• Credible intervals and confidence intervals answer different questions. A 95% Bayesian credible interval [L, U] means P(θ ∈ [L,U] | data) = 0.95 — a direct probability statement about where the parameter is. A 95% frequentist confidence interval means that if you repeated the experiment many times, 95% of the resulting intervals would contain the true θ. For any single computed interval, the true θ either is or isn't in it. Credible intervals are what stakeholders intuitively mean when they ask "how likely is it that the true value is in this range?" — confidence intervals don't answer that question.
• Sequential Bayesian updating is the natural model for streaming systems: the posterior from today becomes the prior for tomorrow. With Beta-Binomial, after day 1 you have Beta(α₁, β₁); feed it new data on day 2 and you get Beta(α₁ + new_successes, β₁ + new_failures). No reprocessing of historical data. The prior from yesterday is a sufficient summary of everything observed so far — this is what makes it computationally attractive for systems that must update continuously.
• Prior sensitivity is the thing practitioners skip and then regret. In low-data regimes, the prior dominates — your conclusions are largely determined by what you assumed before seeing any data. In high-data regimes, the likelihood takes over and the prior washes out. Always sanity-check: re-run with a more diffuse prior. If the posterior shifts substantially, you do not yet have enough data to draw firm conclusions — the result is prior-driven, not data-driven.
• MAP (Maximum A Posteriori) estimate: θ_MAP = argmax_θ [log p(X|θ) + log p(θ)]. This is regularised MLE — L2 regularisation corresponds to a Gaussian prior, L1 to a Laplace prior. MAP is often the right production choice (fast, no integration), but it collapses the posterior to a point and discards all information about posterior shape. Using a MAP estimate for predictions is the same as ignoring posterior uncertainty.
• Priors that look uninformative often aren't. A uniform prior over θ ∈ [0,1] looks neutral but assigns equal probability to CTR = 0.01 and CTR = 0.99 — which may be a strong prior in a context where rates above 20% are implausible. A uniform prior over log(θ) implies a very different belief. Always ask what your prior implies about the quantities you actually care about, not just the parameterisation you happened to write down.

Recap

• Posterior ∝ likelihood × prior. 3 flips HHH: MLE = 1.0 (degenerate); Beta(1,1) prior → Beta(4,1), posterior mean 0.8.
• Prior's pull is inversely proportional to data: dominates when scarce, washes out when abundant (300/300 → 0.99 ≈ MLE).
• The hard part is the normaliser $p(X)=int p(X|θ)p(θ)dθ$ — analytic only for conjugate pairs; MCMC/VI/Laplace exist to dodge it.
• Conjugate priors = closed-form updates: Beta+Binomial, Gaussian+Gaussian, Dirichlet+Categorical — convenient but constrains prior shape.
• MAP = regularised MLE (L2 ↔ Gaussian prior, L1 ↔ Laplace); collapses posterior to a point, discards uncertainty.
• Predictive distribution integrates over the posterior — plugging in MAP underestimates uncertainty, badly in low-data.
• Credible ≠ confidence: credible = direct $P(θ∈[L,U]|data)$; confidence = long-run coverage, silent on any single interval.

Check your understanding