Random Variables & Distributions
PMF, PDF, expectation, variance, common distributions
You are predicting whether a transaction is fraud. The outcome is 0 or 1 — but the transaction amounts are continuous. Two completely different mathematical objects. Without the right vocabulary, you will confuse notation and make probability statements that are incoherent — like asking P(X = 3.14159) for a continuous variable, which is always exactly 0.
Random variables are the bridge between raw data and probability theory. A random variable is a function that assigns a real number to each outcome in a sample space. The type — discrete or continuous — determines which mathematical machinery applies, and mixing them up silently produces wrong answers.
Discrete random variables: P(X = k) is a valid statement. The PMF (probability mass function) sums to 1 over all k. Bernoulli(p) is your fraud indicator — 1 with probability p, 0 otherwise. Binomial(n, p) counts frauds in n transactions. Poisson(λ) counts events in a fixed time window. Geometric(p) counts trials until the first fraud — P(X=k) = (1-p)^{k-1}p for k=1,2,3,.... Any PMF built from a constant ratio r, like Geometric's (1-p) or a (1/2)^k pattern, needs two series identities to solve for its normalizing constant and its mean: Σ_{k=1}^∞ r^k = r/(1-r), and Σ_{k=1}^∞ k·r^k = r/(1-r)² — both valid for |r| < 1.
Continuous random variables: P(X = 3.14) = 0 exactly — a single point has measure zero in a continuous space. What you can compute is P(a ≤ X ≤ b) = ∫_a^b f(x)dx. The PDF (probability density function) integrates to 1 over the real line. The Gaussian N(μ, σ²) is continuous. The Exponential distribution measures time between events.
Expected value: E[X] = Σ x·P(X=x) for discrete, ∫ x·f(x)dx for continuous. This is the probability-weighted average — the long-run mean if you drew forever. Variance: Var(X) = E[(X - E[X])²] = E[X²] - (E[X])². The standard deviation σ = √Var(X) is in the same units as X — interpretable. The gap E[X²] - (E[X])² is always non-negative, and zero only when X is constant. Never report a model's mean prediction without its variance — the variance is what tells you whether that mean is trustworthy.
NOT this. Probability and statistics are not interchangeable terms. Probability reasons forward from a known model to predictions about data. Statistics reasons backward from observed data to inferences about the model. Random variables live in probability. Estimators live in statistics. Confusing the direction leads to conditioning on the wrong thing and drawing the wrong conclusions. Asking P(X = 3.14) for transaction amounts is not a rounding question — it is a category error about the type of the variable.
Key points
- Always verify whether a variable is discrete or continuous before writing a probability statement. P(X = x) means something for discrete variables and nothing for continuous ones — for continuous X, P(X = x) = 0 for every single x. The mathematical framework (PMF vs PDF) determines what operations are valid. Writing a PMF for a continuous variable is not approximately wrong, it is completely wrong.
- Trap: confusing E[f(X)] with f(E[X]). Jensen's inequality: for a convex function, E[f(X)] ≥ f(E[X]). The expected loss of a model is not the loss at the expected parameter. Concretely: E[X²] ≥ (E[X])², with equality only when X is constant. This matters for Bayesian prediction — the mean of a distribution over predictions is not the prediction at the mean of the distribution.
- Diagnostic: if you compute a probability that exceeds 1.0 or is negative, you have misidentified the variable type or mixed PMF and PDF formulas. Check whether your probability statement requires summing (discrete) or integrating (continuous). A PDF value f(x) can exceed 1 — it is a density, not a probability. Only the integral of f over an interval is a probability.
Discrete and continuous random variables require completely different probability machinery. Misidentifying the type produces probability statements that are not just inaccurate but meaningless — P(X = x) for a continuous variable is always exactly 0, no matter how precisely you specify x.
Recap
- Discrete vs continuous is a type, not a detail: $P(X=x)$ is valid for discrete, always 0 for continuous.
- Discrete = PMF sums to 1; continuous = PDF integrates to 1. A PDF value can exceed 1 (it's a density).
- Expectation = probability-weighted average: $E[X]=\sum x\,P(X{=}x)$ or $\int x f(x)dx$.
- Variance = $E[X^2]-(E[X])^2$, always ≥ 0, zero only when X is constant. σ shares X's units.
- Jensen: for convex f, $E[f(X)] \geq f(E[X])$ — expected loss ≠ loss at the mean parameter.
- Probability reasons forward (model→data); statistics reasons backward (data→model). Don't mix directions.
- Sanity check: a probability >1 or <0 means you mixed PMF and PDF formulas or misread the type.
Check your understanding
Q1. A discrete random variable X has PMF P(X=k) = C × (1/2)^k for k=1,2,3,... Find C and compute E[X].
- A) C=1/2, E[X]=4. Find C: Σ_{k=1}^∞ C·(1/2)^k = C·2 = 1, so C=1/2 — this treats the geometric series Σ(1/2)^k as summing to 2 instead of 1, an off-by-factor error common when the index starts at k=1. E[X] = Σ k·(1/2)^{k+1} = 4, carrying the extra (1/2) factor through the shifted-index derivative identity.
- B) C=1, E[X]=4. Find C: Σ_{k=1}^∞ C·(1/2)^k from k=1 equals C·1=1, giving C=1 by the standard geometric series identity a/(1−r). E[X]=Σ k·(1/2)^k. Using Σ k·r^k = r/(1-r)² at r=1/2: E[X]=(1/2)/(1/4)=2. But since the sum starts at k=1, not k=0, we add 1 to shift the index: E[X]=2+1=4.
- C) Find C: Σ_{k=1}^∞ C·(1/2)^k = C·(1/2)/(1−1/2) = C = 1, so C=1. E[X] = Σ_{k=1}^∞ k·(1/2)^k. Using Σ k·r^k = r/(1−r)² at r=1/2: E[X] = (1/2)/(1/4) = 2. Sanity check: P(X=1)=1/2, P(X=2)=1/4 — weight concentrates near small k, so E[X]=2 sits just above the mode.
- D) C=2, E[X]=2. Find C: Σ_{k=1}^∞ C·(1/2)^k = 1 requires C=2, since Σ(1/2)^k=1/2 — mistakenly treating the infinite geometric sum as equal to just its first term (1/2)¹, instead of applying the full formula a/(1−r) with a=r=(1/2), which correctly gives 1. E[X] = Σ k·(1/2)^k, dropping C since a normalizing constant cancels in the mean, giving (1/2)/(1/4) = 2 — matching the value expected for a rapidly decaying PMF.
Q2. X ~ N(0,1) and Y = X². Which TWO of the following statements about Y are true?
- A) Y = X² follows a chi-squared distribution with 1 degree of freedom, Y ~ χ²(1), because χ²(1) is defined exactly as the square of a single standard normal random variable — the base case of the general χ²(k) = sum of k independent squared standard normals.
- B) E[Y] = E[X²] = 1, since E[X²] = Var(X) + (E[X])² = 1 + 0 = 1 for X ~ N(0,1); this matches the χ²(1) distribution's mean, which always equals its degrees of freedom k=1 by the standard chi-squared moment formula relating mean directly to k.
- C) Y = X² follows a half-normal distribution, since squaring removes the sign of X and the half-normal is defined as the distribution of |X| for X ~ N(0,1); squaring a half-normal variable is mathematically equivalent to squaring the original normal.
- D) E[Y] = E[X²] = 2, because the second moment of N(0,1) equals the sum of variance and squared mean, σ² + μ² = 1 + 1 = 2, following the general second-moment identity applied to a distribution with unit variance and unit mean.
Q3. Why is E[f(X)] ≠ f(E[X]) in general? When does equality hold?
- A) E[f(X)] ≠ f(E[X]) because the expectation operator distributes over sums but not over arbitrary functions. Equality holds when f is monotone (strictly increasing or decreasing) since monotone functions preserve the ordering of outcomes. For non-monotone f like f(x)=x², the inequality direction depends on the sign of x.
- B) E[f(X)] ≠ f(E[X]) whenever f is nonlinear, because the average of f(X) over the distribution of X is taken before applying f, whereas f(E[X]) applies f to the mean first. Equality holds only when X is symmetric — symmetric distributions ensure the Jensen gap cancels. For asymmetric X, E[f(X)] > f(E[X]) always.
- C) E[f(X)] ≠ f(E[X]) in general because of sampling variability in a finite dataset. Equality only holds in the limit n→∞, when the law of large numbers ensures the sample mean converges to E[X] and the continuous mapping theorem then guarantees f(X̄) converges to f(μ) as well.
- D) Jensen's inequality: E[f(X)] ≥ f(E[X]) for convex f, ≤ for concave f. Equality holds iff f is linear (affine) or X is constant. Example: E[X²] ≥ (E[X])², equality only when Var(X)=0. In ML: expected loss of averaging predictions differs from loss at the average prediction — mean is optimal only under squared loss.
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →