Probability Fundamentals

Sample spaces, Bayes' theorem, conditional probability

An email arrives. Is it spam? You know before opening it that 30% of all email is spam — that is your prior. You open it and find the word "FREE" five times. Words like "FREE" appear five times more often in spam than in legitimate email. How do you combine what you already knew with this new evidence?

The naive move is to just count: look at all emails containing "FREE" in your dataset and compute the fraction that were spam. This ignores the prior and breaks when your dataset is small or skewed. What you actually want is a principled update: start from P(spam) = 0.3, observe the word "FREE," and revise. The mechanism for this is conditional probability. P(spam|FREE) asks: among emails that contain "FREE," what fraction are spam? This is not the same as P(FREE|spam), which asks: among spam emails, what fraction contain "FREE?" Those two quantities are different, and confusing them is the base rate fallacy.

To go from the latter to the former, you need Bayes' theorem: $P(spam|FREE) = P(FREE|spam) \cdot P(spam) / P(FREE)$. The numerator multiplies the likelihood — how often "FREE" appears in spam — by the prior probability of spam. The denominator, $P(FREE)$, normalises so everything sums to 1. It equals $P(FREE|spam) \cdot P(spam) + P(FREE|ham) \cdot P(ham)$: the law of total probability.

This is the mechanism behind every probabilistic ML system. The prior $P(spam) = 0.3$ encodes what you knew before. The likelihood $P(FREE|spam)$ is what your model learns from data. The posterior $P(spam|FREE)$ is what you actually want — your updated belief after seeing evidence.

Concretely: suppose "FREE" appears in 10% of spam emails but only 2% of legitimate ones — a 5x likelihood ratio, matching what you observed. Then $P(FREE) = P(FREE|spam) cdot P(spam) + P(FREE|ham) cdot P(ham) = 0.10 imes 0.3 + 0.02 imes 0.7 = 0.044$, and $P(spam|FREE) = (0.10 imes 0.3) / 0.044 = 0.03 / 0.044 approx 0.68$. A single word moved your belief from a 30% prior to a 68% posterior.

NOT this. Most people think $P(A|B) = P(B|A)$. They do not. A test that correctly identifies 99% of sick patients — $P(positive|disease) = 0.99$ — does not mean a positive result means you are 99% likely to have the disease. Suppose the disease affects only 1 in 1000 people ($P(disease) = 0.001$) and the test also has a 1% false-positive rate ($P(positive|no\ disease) = 0.01$, i.e. 99% specificity). Then $P(positive) = 0.99 \times 0.001 + 0.01 \times 0.999 \approx 0.0110$, so $P(disease|positive) = 0.00099 / 0.0110 \approx 9\%$. The false positive rate, applied to the large healthy population, swamps the true positives. Dropping the prior — treating $P(positive|disease)$ as $P(disease|positive)$ — is the error. The formal rules: conditional probability $P(A|B) = P(A \cap B)/P(B)$. Independence: $P(A \cap B) = P(A)P(B)$. Chain rule: $P(A,B,C) = P(A)P(B|A)P(C|A,B)$.

Key points

• Use it when you know the likelihood but want the posterior. Any time your model gives you $P(data|hypothesis)$ but you need $P(hypothesis|data)$, Bayes' theorem is the exact conversion. In spam filtering: your trained model gives you $P(word|spam)$; Bayes gives you $P(spam|word)$. Without the prior, you cannot make this conversion.
• The production trap: ignoring the base rate. A fraud detection model that catches 99% of actual fraud (99% recall) sounds great. But if only 0.1% of transactions are fraudulent and the model also has a 1% false-positive rate, then out of 10,000 transactions: about 10 are truly fraudulent (0.1% of 10,000), of which the model catches roughly 9.9 (99% recall). But that same 1% false-positive rate applied to the ~9,990 legitimate transactions flags about 100 of them too. So of the ~110 total flagged alerts, only ~9.9 are real fraud — realized precision is about 9%, not 99%. The prior probability of fraud determines whether a high-recall model is operationally useful. Always report precision *at the operating base rate*, not just recall on a balanced test set.
• The diagnostic: check whether your prior and likelihood are on the same scale. If $P(spam) = 0.3$ but your spam filter was trained on a 50/50 balanced dataset, the likelihood ratios are calibrated for a different prior. Recalibrate with Platt scaling or isotonic regression before multiplying priors by likelihoods. Symptoms of miscalibration: model confidence of 90% but actual accuracy of 60% on live traffic.
• Counting orderings: the binomial coefficient. Flip a biased coin three times with $P(H) = 0.7$. "Exactly 2 heads" can happen as HHT, HTH, or THH — three distinct orderings, each with probability $0.7^2 imes 0.3^1 = 0.147$. There are $inom{3}{2} = 3$ such orderings (choose which 2 of the 3 flips are heads), so $P( ext{exactly 2 heads}) = 3 imes 0.147 = 0.441$. This is the binomial PMF: $P(k heads in n flips) = inom{n}{k}p^k(1-p)^{n-k}$ — count the orderings, then multiply by the probability of any single ordering.

Recap

• Bayes = the update rule: $P(spam|FREE) = P(FREE|spam)\,P(spam)/P(FREE)$. Prior × likelihood ÷ evidence.
• Posterior ≠ likelihood: $P(A|B) \neq P(B|A)$. Confusing them is the base rate fallacy.
• Rare disease trap: 99% sensitive test, 0.1% prevalence → $P(disease|+) \approx 9\%$. False positives swamp true ones.
• Evidence = law of total probability: $P(FREE) = P(FREE|spam)P(spam) + P(FREE|ham)P(ham)$ normalises to 1.
• Report precision at the real base rate, not on a balanced test set — 99% recall + 1% false-positive rate + 0.1% fraud prevalence → realized precision ≈ 9%, not 99%.
• Recalibrate before multiplying: a 50/50-trained model's likelihoods don't match a 30% prior. Platt/isotonic first.

Check your understanding