Bayesian Inference: Prior, Likelihood, Posterior, and When to Use It

Frequentist statistics treats parameters as fixed unknowns. Bayesian statistics treats them as random variables with probability distributions. The posterior — your updated belief after seeing data — is derived from the prior (what you believed before) and the likelihood (how probable the data is under each parameter value). This framework quantifies uncertainty, incorporates prior knowledge, and naturally handles small data regimes.

Bayesian inference is a framework for updating beliefs in light of evidence. Instead of asking "what is the parameter?" it asks "what is the probability distribution over possible parameter values, given what I have observed?" This distinction has significant practical implications for how you quantify uncertainty and make decisions.

Bayes' theorem

P(θ | D) = P(D | θ) * P(θ) / P(D)

Posterior ∝ Likelihood × Prior. In words: your belief about the parameter θ after seeing data D is proportional to how probable the data is under θ (likelihood) times your prior belief about θ (prior). P(D) is a normalisation constant.

The prior: encoding domain knowledge

The prior P(θ) represents your beliefs about θ before seeing data. It can encode domain knowledge ("I know this coefficient is probably positive"), regularisation ("weights should be small — use a Gaussian prior," which is equivalent to L2 regularisation in MAP estimation), or ignorance ("I have no idea — use a flat or weakly informative prior"). Priors matter most when data is scarce. With large data, the likelihood dominates and the prior washes out — frequentist and Bayesian estimates converge.

The likelihood: how the data informs the parameter

P(D | θ) is the probability of observing the data given the parameter value. For a coin with bias θ, if you observe 7 heads in 10 flips, the likelihood is θ^7 * (1-θ)^3. The likelihood is maximised at θ=0.7 (maximum likelihood estimate). Bayesian inference does not stop at the maximum — it computes the full posterior distribution, which accounts for uncertainty when the sample is small.

Conjugate priors: tractable closed-form posteriors

In general, computing the posterior requires integrating P(D | θ) * P(θ) over all θ — an integral that is often intractable. Conjugate priors are chosen so that the prior and posterior have the same distributional form, making the posterior analytically computable. Examples: Beta prior with Binomial likelihood → Beta posterior. Normal prior with Normal likelihood → Normal posterior (with updated mean and variance). Dirichlet prior with Multinomial likelihood → Dirichlet posterior. Conjugate pairs are the analytical workhorses of Bayesian inference and appear in bandit algorithms, naive Bayes classifiers, and topic models (LDA).

MAP estimation: the connection to regularisation

Maximum a posteriori (MAP) estimation finds the mode of the posterior: θ_MAP = argmax_θ log P(D | θ) + log P(θ). This is regularised maximum likelihood: the log prior acts as a regularisation term. Gaussian prior (P(θ) ∝ exp(-λ||θ||^2)) → L2 regularisation. Laplace prior → L1 regularisation. MAP gives a point estimate; full Bayesian inference retains the entire posterior distribution.

MCMC: sampling when posteriors are intractable

For complex models, the posterior has no closed form. Markov Chain Monte Carlo (MCMC) approximates the posterior by constructing a Markov chain that converges to the posterior distribution. Metropolis-Hastings: propose a new θ' from a proposal distribution; accept with probability min(1, P(θ'|D)/P(θ|D)). Running this chain produces samples from the posterior after the chain mixes.

Modern probabilistic programming languages (PyMC, Stan, Pyro) implement MCMC and variational inference, making Bayesian modelling accessible without deriving samplers by hand.

Bayesian credible intervals vs frequentist confidence intervals

A Bayesian 95% credible interval [a, b] means: given the data, P(a ≤ θ ≤ b | D) = 0.95. This is what most people intuitively mean when they say "95% confidence interval." A frequentist 95% confidence interval means: if you repeated the experiment many times and computed the interval each time, 95% of those intervals would contain the true parameter. These are not the same statement, and the Bayesian interpretation is often more natural for decision-making.

When to use Bayesian methods in production

Bayesian approaches shine in: small data regimes where the prior provides meaningful regularisation, uncertainty quantification (prediction intervals rather than point estimates), online/sequential updating (Thompson sampling for bandits), and hierarchical models (partial pooling across groups — e.g., estimating conversion rates for 1000 products with varying sample sizes). They are more expensive computationally than frequentist methods and harder to communicate to stakeholders. For large datasets where uncertainty quantification is less critical, frequentist maximum likelihood is simpler and sufficient.

Try on Colab: use PyMC to infer the conversion rate of two web page variants (A/B test) from observed clicks/impressions. Use a Beta(1,1) (uniform) prior. Plot the posterior distributions for both variants. Compute P(variant B > variant A) by sampling from both posteriors. Compare the Bayesian result to a frequentist chi-squared test — they should agree when n is large but give different uncertainty estimates for n=20.