MLE vs MAP Estimation

Q1. For a Gaussian likelihood with unknown mean μ and fixed variance σ², derive the MLE estimate for μ given data {x₁,...,xₙ}.

• A) MLE: maximise log-likelihood ℓ(μ) = −n/2·log(2πσ²) − (1/2σ²)Σ(xᵢ−μ)². Setting ∂ℓ/∂μ = 0 gives Σ(xᵢ−μ) = 0, so μ_MLE = Σxᵢ — forgetting to divide by n, leaving the raw sum instead of the sample mean. Because the sum grows with n while the true optimum does not, this "estimate" diverges as more data is collected, the opposite of what a consistent estimator should do.
• B) Maximise L(μ) = Π (1/√(2πσ²))exp(−(xᵢ−μ)²/(2σ²)). Taking log: ℓ(μ) = const − (1/2σ²)Σ(xᵢ−μ)². Setting ∂ℓ/∂μ = 0: −(1/2σ²)·(−2)Σ(xᵢ−μ) = 0, so Σxᵢ − nμ = 0, giving μ_MLE = (1/n)Σxᵢ = x̄. But this is a biased estimator: E[μ_MLE] = μ−σ²/n due to Jensen's inequality applied to the log. The unbiased estimator requires a Bessel correction: μ_unbiased = x̄·n/(n−1).
• C) L(μ) = Π (1/√(2πσ²))exp(−(xᵢ−μ)²/(2σ²)). Log-likelihood: ℓ(μ) = −n/2·log(2πσ²) − (1/2σ²)Σ(xᵢ−μ)². Maximise: ∂ℓ/∂μ = (1/σ²)Σ(xᵢ−μ) = 0, giving μ_MLE = (1/n)Σxᵢ = x̄ — the sample mean. Maximising log-likelihood here is equivalent to minimising Σ(xᵢ−μ)², so MLE = least squares = sample mean, which is why MSE is the natural loss under a Gaussian noise assumption.
• D) MLE for Gaussian mean requires maximising L(μ) = Π exp(−(xᵢ−μ)²/(2σ²)). The log-likelihood is ℓ(μ) = −Σ(xᵢ−μ)²/(2σ²), a concave quadratic in μ. Setting ∂ℓ/∂μ = Σ(xᵢ−μ)/σ² = 0 gives μ_MLE = median({xᵢ}), since the sum of absolute deviations from the median is minimised, not the sum of squared deviations. The sample mean would instead be the MLE under a Laplace (L1) likelihood, not a Gaussian one.

Q2. MLE for a Bernoulli distribution gives P̂(X=1) = (number of 1s)/(total samples). Now add a Beta(α,β) prior. Which two of the following are correct?

• A) Likelihood: L(p) = Π pˣⁱ(1−p)^{1−xᵢ} = p^s(1−p)^{n−s} where s=Σxᵢ. Prior: Beta(α,β): π(p) ∝ p^{α−1}(1−p)^{β−1}. Posterior ∝ p^{s+α−1}(1−p)^{n−s+β−1} ~ Beta(s+α, n−s+β). MAP maximises the log-posterior (s+α−1)log p + (n−s+β−1)log(1−p): p_MAP = (s+α−1)/(n+α+β−2). The Beta prior adds α−1 pseudo-observations of class 1 and β−1 of class 0; for α=β=1 this reduces to MLE=s/n.
• B) The Beta posterior has mean (s+α)/(n+α+β), so MAP = mean = (s+α)/(n+α+β). The MAP of a Beta distribution equals its mean because Beta is symmetric around its mean in every case. For α=β=1 (uniform prior), MAP = s/n = MLE. The denominator n+α+β adds α+β pseudo-observations rather than α+β−2, because the normalising constant of the Beta prior itself contributes exactly 1 pseudo-observation per parameter.
• C) The Beta(α,β) posterior is Beta(s+α, n−s+β). The MAP is the mode: (s+α−1)/(n+α+β−2). This equals MLE plus a Bayesian correction term: s/n + (α−β)/(n(n+α+β−2)). For large n, the correction term vanishes entirely and MAP converges to the raw count s/n. For α=β, MAP equals MLE exactly at every n — a perfectly symmetric prior never biases the estimate away from the raw count.
• D) The Beta(α,β) posterior also has mean (s+α)/(n+α+β), distinct from the MAP mode (s+α−1)/(n+α+β−2); the two coincide only in special cases such as when the posterior itself is symmetric (s+α = n−s+β) or in the n→∞ limit. Reporting the posterior mean instead of the mode gives a different but equally valid Bayesian point estimate — choosing between MAP and posterior mean is a modeling decision, not something MAP itself requires.

Q3. A linear regression model's MSE loss is equivalent to maximum likelihood under what distributional assumption? What assumption does L1 loss correspond to?

• A) MSE corresponds to a Uniform likelihood: P(yᵢ|xᵢ,θ) = Uniform(ŷᵢ−ε, ŷᵢ+ε) for some fixed ε. Minimising MSE finds the θ that keeps all residuals within ±ε, treating every in-range residual as equally likely. L1 loss corresponds instead to a Gaussian likelihood with heavier tails — specifically, a Student-t distribution with 1 degree of freedom, i.e. a Cauchy distribution.
• B) MSE loss Σ(yᵢ−ŷᵢ)² ∝ −log L(θ) under a Gaussian likelihood P(yᵢ|xᵢ,θ) = N(ŷᵢ,σ²) — minimising MSE = maximising Gaussian log-likelihood. L1 loss Σ|yᵢ−ŷᵢ| corresponds to a Laplace likelihood P(yᵢ|xᵢ,θ) = (1/2b)exp(−|yᵢ−ŷᵢ|/b). Laplace has heavier tails, so L1 penalises outliers less than L2. L1 gives median regression; L2 gives mean regression.
• C) MSE corresponds to Gaussian noise with variance proportional to ŷᵢ (heteroskedastic): P(yᵢ|xᵢ,θ) = N(ŷᵢ, |ŷᵢ|·σ²). This is why MSE is claimed to be unstable for regression near zero — the likelihood is undefined when ŷᵢ=0. L1 loss instead corresponds to constant-variance Gaussian noise P(yᵢ|xᵢ,θ) = N(ŷᵢ, σ²), supposedly more robust since it does not assume variance scaling with the prediction.
• D) MSE corresponds to Gaussian noise P(yᵢ|xᵢ,θ) = N(ŷᵢ, σ²) only when the noise is additive and independent across all samples. L1 loss corresponds instead to a Poisson likelihood: P(yᵢ|xᵢ,θ) = exp(−ŷᵢ)ŷᵢ^{yᵢ}/yᵢ!, appropriate for count data. L1 is claimed to be more robust than MSE because Poisson has lighter tails than Gaussian, assigning less probability mass to large residuals.