Joint Distributions & Independence

Joint PDF/PMF, marginals, conditional distributions, covariance

You are building a credit scoring model. Two features: age (continuous) and missed_payments (count, discrete). You want to know P(age > 30, missed_payments ≥ 2). These variables are not independent — older borrowers tend to have longer credit histories and different payment patterns. You cannot multiply P(age > 30) × P(missed_payments ≥ 2) and get the right answer. You need the joint distribution P(age, missed_payments).

Joint distribution P(X, Y): for discrete variables, a 2D table of probabilities summing to 1. For continuous variables, a 2D density f(x, y) integrating to 1. Marginal distribution: integrate or sum out the other variable. P(X = x) = Σ_y P(X = x, Y = y). Conditional distribution: P(Y = y | X = x) = P(X = x, Y = y) / P(X = x). This is the Bayes denominator — the mechanism behind every probabilistic classifier.

Independence: X and Y are independent if and only if P(X, Y) = P(X) × P(Y) for all values. In ML, Naive Bayes assumes all features are conditionally independent given the label. This is almost always false, but the classification decisions can still be correct even when the probability estimates are wrong — independence of errors in different directions can cancel.

Covariance: Cov(X, Y) = E[(X - μ_X)(Y - μ_Y)] = E[XY] - E[X]E[Y]. Positive covariance means the variables tend to move together. Negative means they move oppositely. Zero means no linear relationship — NOT the same as independence. Correlation: ρ = Cov(X, Y) / (σ_X · σ_Y). Bounded in [-1, 1]. For jointly Gaussian variables, zero correlation implies independence — for any other distribution, it doesn't, as the next paragraph shows.

NOT this. Correlation = 0 does not mean the variables are independent. This is only true for jointly Gaussian random variables. For any other distribution, zero linear correlation is compatible with strong nonlinear dependence. Let X ~ Uniform(-1, 1) and Y = X². Then Cov(X, Y) = 0 by symmetry, but Y is completely determined by X — perfect deterministic dependence. Mutual information captures any dependence; correlation captures only linear dependence.

Key points

• Always compute the joint distribution (or its sample estimate) before assuming independence. A quick scatter plot or correlation matrix is a fast first check, but it only catches linear relationships — nonlinear dependence needs mutual information or a rank-based test. Ignoring dependence leads to probability estimates that are systematically wrong — the Naive Bayes independence violation is not a theoretical concern, it produces miscalibrated probabilities that cannot be used for risk-sensitive decisions.
• Trap: treating conditional probabilities as symmetric. P(fraud | transaction > $10K) ≠ P(transaction > $10K | fraud). Getting the conditioning direction wrong produces confident wrong answers. The base rate of each event determines which direction of conditioning gives useful information. Draw the causal structure first, then condition.
• Diagnostic: if a model's predicted probabilities are miscalibrated — predicted 0.8 but true frequency is 0.4 — check whether correlated features are creating double-counting of information. This is the Naive Bayes independence violation made explicit. When two features carry the same signal (high correlation), treating them as independent doubles the effective evidence, pushing predictions toward the extremes.

Recap

• Joint P(X,Y) is required when features are dependent — can't multiply P(X)×P(Y) unless independent.
• Marginal = sum/integrate out the other variable; conditional $P(Y|X)=P(X,Y)/P(X)$ is the Bayes denominator.
• Independence ⟺ P(X,Y)=P(X)P(Y) for all values. Naive Bayes assumes this — usually false.
• Cov(X,Y)=E[XY]−E[X]E[Y]; ρ = Cov/(σ_X σ_Y) ∈ [−1,1]. Captures linear relationship only.
• Zero correlation ≠ independence: X~Uniform(−1,1), Y=X² has Cov=0 but Y fully determined by X.
• Correlated features double-count evidence in Naive Bayes → miscalibrated, over-confident probabilities.
• For real independence use mutual information or a rank test, not Pearson correlation.

Check your understanding