BM25 and TF-IDF: Why Sparse Retrieval Still Beats Neural Search in Many Cases

Every team building semantic search eventually asks: should we replace BM25 with dense retrieval? The honest answer is often no. BM25 has been the backbone of Google, Elasticsearch, and Solr for decades. It handles exact keyword matches, rare terms, and out-of-distribution queries better than neural models. Understanding what BM25 computes — and where it fails — is the prerequisite for knowing when dense retrieval actually helps.

Before neural search, information retrieval was dominated by sparse bag-of-words models. TF-IDF (1970s) and BM25 (1994) formalised the intuition that a word is important if it appears frequently in this document but rarely across all documents. They remain competitive baselines and are the default in production search systems.

TF-IDF: the intuition

Term Frequency-Inverse Document Frequency scores a term t in document d in corpus C. TF(t,d) = count of t in d, or a log-normalised version. IDF(t) = log(N / df(t)), where N is the total number of documents and df(t) is the number containing t. TFIDF(t,d) = TF(t,d) × IDF(t).

The IDF term down-weights terms that appear in many documents ("the", "a", "is" appear everywhere and carry no signal) and up-weights rare, discriminative terms. A query's score for a document is the sum of TFIDF weights for matching query terms.

BM25: fixing TF-IDF's two weaknesses

BM25 (Robertson et al., 1994) is the probabilistic extension that addresses two problems with raw TF-IDF. Problem 1: raw TF grows without bound — a document mentioning "python" 100 times should not score 100× higher than one mentioning it once. BM25 saturates TF: TF_sat = TF * (k1 + 1) / (TF + k1 * (1 - b + b * dl/avgdl)). k1 controls saturation speed (typically 1.2–2.0); the saturation kicks in after a few mentions.

Problem 2: longer documents match more terms by chance, not because they are more relevant. The document length normalisation term (1 - b + b * dl/avgdl) penalises long documents, where dl is the document length and avgdl is the mean length across the corpus. b=0.75 is the standard setting.

Why BM25 still wins in many production settings

Exact match: if a user searches "python 3.12 changelog", BM25 exactly matches "python", "3.12", "changelog" as tokens. A dense retrieval model may map "python" near "snake" or near "Django" and miss exact version matches. BM25 never misses exact keyword matches. Rare terms: BM25 gives high IDF to rare tokens (product codes, technical strings, names). Dense models see rare terms as noise or OOV tokens. Out-of-domain queries: BM25 requires no training and generalises perfectly to new vocabularies. Dense models fail on terms not seen at training time.

In the BEIR benchmark (heterogeneous IR evaluation across 18 datasets), BM25 outperforms many dense retrieval models on out-of-domain datasets. The lesson: dense models trained on MS MARCO do not generalise to biomedical or legal search without domain-specific fine-tuning.

Inverted index: how BM25 scales

BM25 is efficient because of the inverted index: a data structure mapping each term to a posting list — the list of (document_id, TF) pairs for documents containing that term. Query scoring: for each query term, look up its posting list, score each matching document, merge and rank. With a precomputed inverted index, BM25 can search a billion documents in milliseconds. Dense retrieval requires ANN search which is slower for very large corpora and requires keeping large embedding matrices in memory.

Hybrid search: sparse + dense

The production answer is rarely pure BM25 or pure dense retrieval. Hybrid search runs both: BM25 for exact/keyword matching, dense retrieval for semantic matching, then fuses the result lists. Reciprocal Rank Fusion (RRF) is the simplest fusion: score = Σ_m 1/(k + rank_m), summing over retrieval methods, where k=60 is standard. RRF is surprisingly effective and does not require calibrated scores from each method.

Elasticsearch's new semantic search uses ELSER (a learned sparse model) that produces expanded sparse representations — essentially learning which additional terms to add to the BM25 index. This hybrid approach achieves dense-retrieval quality with BM25-like serving efficiency.

Try on Colab: use rank_bm25 (Python library) to index Wikipedia abstracts (10K sample). Compare retrieval quality on a set of factoid queries against sentence-transformer dense retrieval. Count how often each method is better. You should find BM25 wins on exact-match and rare-term queries; dense retrieval wins on paraphrase queries ("cardiovascular disease" vs "heart attack"). Implement RRF fusion and measure improvement.