Topic Modeling

LDA mechanics, NMF, choosing K, coherence vs perplexity, production limits

You have 100,000 customer-support tickets and no time to read them. "App crashes at checkout," "payment keeps declining," "can't log in after the update," "no sound on video calls" — somewhere in that pile are a handful of recurring themes, and you want to find them *without* labelling every ticket by hand. Topic modeling does exactly this: it reads the whole corpus and discovers the hidden themes automatically, just from which words tend to show up together.

The core idea rests on one observation: words that belong to the same theme keep appearing together. "Payment," "declined," "card," and "refund" cluster in billing tickets; "crash," "freeze," "update," and "restart" cluster in bug reports. So a topic is really just *a group of words that travel together*, and a document is usually a *blend* of a few topics at once — a ticket might be 70% billing, 30% bug.

LDA: the classic recipe

Latent Dirichlet Allocation (LDA) is the workhorse. Its picture of the world: every topic is a bag of words with different weights (the "billing" topic leans heavily on "payment," "card," "declined"), and every document is a mixture of a few such topics. LDA starts from the finished documents and works *backwards* — given only the words it can actually see, it figures out what set of topics, and what per-document blend, most plausibly produced them. You tell it K, the number of topics to look for; it hands back the topics and each document's mix.

The alternatives, and when they win

NMF (non-negative matrix factorization) factors the word-count table into "documents × topics" and "topics × words," keeping everything positive so the pieces add up rather than cancel out — it is faster than LDA and often better on short documents. BERTopic takes a more modern route: it turns each document into a meaning-based embedding (from a model like BERT), clusters those, and reads off each cluster's characteristic words. Because it works on *meaning* rather than raw word matches, it shines on short, messy text — it knows "crash" and "freeze" are related, where LDA just sees two unrelated words.

The one hard choice: how many topics?

There is no free lunch on picking K. Ask for too few topics and you get vague mega-themes that blur real distinctions; ask for too many and you get near-duplicate, hair-splitting topics nobody can act on. It is tempting to lean on perplexity (a statistical fit score), but that is a trap: perplexity almost always keeps "improving" as you add topics, so it will happily push you toward far too many. Coherence (do a topic's top words actually belong together?) is the better guide — it peaks at a sensible K and then falls as topics start to fragment. But no number settles it. The real test is human: can a domain expert put a clear one-word label on *every* topic without hedging? The right K is the largest one where that is still true.

(One practical note: LDA lives or dies on preprocessing. Strip out stop words and ultra-common terms first, or every topic ends up dominated by "the," "data," and "please," no matter how you tune it.)

Inside LDA: the priors and how it's fit

LDA is a *generative* story with two knobs worth naming. Each document draws a document-topic distribution and each topic a topic-word distribution, both from Dirichlet priors controlled by α and β. α controls how many topics a document typically mixes: small α → each document is dominated by one or two topics; large α → documents spread across many. β controls topic sparsity in words: small β → each topic concentrated on a few words. Because you only observe the words, LDA *infers* the hidden distributions backward, using either collapsed Gibbs sampling (repeatedly reassign each word to a topic based on the current assignments of all others until it stabilises) or variational inference (optimise a tractable approximation to the true posterior). You don't need the math to use LDA, but knowing α/β and "it's Bayesian inference over hidden topic assignments" is standard interview fare.

NMF, mechanically

NMF is the linear-algebra cousin. Take the document-term matrix V (usually TF-IDF weighted) and factor it into two non-negative matrices: V ≈ W × H, where W is documents×topics and H is topics×words. The non-negativity is the whole point — because nothing can subtract, topics combine *additively*, giving a parts-based representation (a document is a sum of topics, not a cancellation of them) that tends to be more interpretable. It's faster and more stable than LDA and often better on short text, where LDA's sparse word co-occurrence starves its statistics.

BERTopic's fine print

BERTopic is powerful but has real knobs and caveats. It depends heavily on the embedding model you choose (and its language/domain — a general English model does poorly on medical or non-English text). Its clustering step (usually HDBSCAN) is sensitive to parameters and produces an explicit outlier topic (-1) for documents it can't cluster — which can swallow a large fraction of your corpus if tuned wrong. And because clustering is stochastic, topics can shift between runs (instability), so pin seeds and check reproducibility. It's often the best on short messy text — but "often," not "always."

Choosing K, more fully

Coherence is the headline metric, but round it out. Plot the coherence curve over K and take a peak, then cross-check with topic diversity (are the top words across topics distinct, or do topics overlap?), the duplicate-topic rate (how many near-identical topics did you get?), and the domain-labelability test (can an expert name every topic?). The final filter is business actionability — a mathematically-fine K that produces topics nobody can *do anything with* is the wrong K. The best K is the largest one that's still coherent, diverse, and actionable.

Evaluating topics beyond one number

Topic quality is multi-dimensional. Topic coherence (top words belong together) and topic diversity (topics don't repeat) are the automated pair. The word-intruder task is the human gold standard: insert one random word into a topic's top words and see if a person can spot it — if they can, the topic is coherent. Also weigh downstream usefulness (do the topics improve a task you care about?) and stability (do you get similar topics across different seeds and data samples?). A topic model that changes completely on a re-run isn't trustworthy no matter its coherence.

In production, topics drift

Topic models aren't fit-once artifacts. Real corpora drift — new products, new slang, new issues appear, so a model trained last quarter slowly stops matching today's tickets. Plan a retraining cadence, and build new-topic detection (a rising share of outlier/-1 documents or a spike in low-coherence assignments signals an emerging theme). The topics also need human naming and a taxonomy governance process so labels stay consistent as the model is retrained, plus monitoring of topic volume over time (a topic suddenly surging is often the real business signal you wanted). The model finds themes; keeping them meaningful over months is an operational job.

Key points

• Use BERTopic over LDA for short texts (tweets, support tickets, product reviews) — BERT embeddings capture synonymy and semantic relationships that word co-occurrence statistics miss. LDA sees "crash" and "fail" as different words. BERTopic knows they are semantically related. For short texts where individual words carry insufficient co-occurrence signal, embedding-based methods dominate word-count-based methods.
• Trap: not preprocessing aggressively before LDA. Remove stop words, apply stemming or lemmatization, remove words appearing in fewer than 5 or more than 80% of documents. LDA without preprocessing produces incoherent topics dominated by frequent function words regardless of K or the number of training iterations.
• Diagnostic: for each topic, look at the top 10 words and ask "can I give this topic a one-word label?" If you cannot, the topic is incoherent — reduce K or improve preprocessing. If all topics look similar (sharing words like "data," "result," "method"), add those high-frequency terms to the stop list and reduce K, because the model has carved one broad topic into near-duplicate components.
• Know LDA's priors and inference, NMF's factorisation, and BERTopic's caveats. LDA draws document-topic and topic-word distributions from Dirichlet priors: α controls how many topics per document, β controls topic word-sparsity, and it's fit by collapsed Gibbs sampling or variational inference. NMF factors the (TF-IDF) matrix V ≈ W×H with non-negativity, giving a fast, stable, parts-based representation that often beats LDA on short text (sparse co-occurrence starves LDA). BERTopic depends on the embedding model and language/domain, produces an explicit outlier topic (-1) that can swallow the corpus if mis-tuned, and is unstable across runs — pin seeds.
• Pick K with multiple signals, evaluate topics multi-dimensionally, and plan for drift. Choose K from the coherence curve plus topic diversity, duplicate-topic rate, expert labelability, and business actionability — perplexity keeps improving with K and misleads. Evaluate with coherence, diversity, the human word-intruder task, downstream usefulness, and stability across seeds/samples (a model that changes on re-run isn't trustworthy). In production, topics drift, so set a retraining cadence, build new-topic detection (rising outlier/-1 share), maintain human naming and taxonomy governance, and monitor topic volume over time — a surging topic is often the real signal.

Recap

• Topic = words that travel together; document = a blend of a few topics.
• LDA: Dirichlet priors (α = topics per doc, β = word sparsity), inferred backward via Gibbs / variational inference.
• NMF: $V\approx W\times H$ non-negative, parts-based, faster and better on short text.
• BERTopic: embed → cluster → read off words; wins on short messy text but depends on embedding model, has -1 outlier topic, unstable across runs.
• Perplexity misleads (keeps improving with K); coherence peaks at a sensible K.
• Real test: can an expert one-word-label every topic? Preprocess aggressively — strip stop words.
• Topics drift in production — retraining cadence, new-topic detection, taxonomy governance.

Check your understanding