Semantic Search: The Full Architecture from Query to Results
Search is not a single model. It is a pipeline: query understanding, candidate retrieval, ranking, and result presentation. Each stage has its own failure modes and ML decisions. A query for "cheap flights nyc" must be understood (intent: price-sensitive travel), expanded ("NYC" → New York City airports), retrieved (fast), re-ranked (personalised), and deduplicated. This is how Google, LinkedIn, and Amazon search actually work.
Production search is a multi-stage pipeline where each stage produces a smaller, higher-quality candidate set. The overall goal is to take a user query — messy, ambiguous, often incomplete — and return the most relevant documents from a billion-scale corpus in under 100ms. No single model can do this. Each stage makes a different trade-off between recall, precision, and latency.
Stage 1: Query understanding
Before retrieval, the query must be understood. This involves multiple sub-tasks that typically run in parallel.
Intent classification: what type of query is this? Navigational (user wants a specific page), informational (user wants to learn something), transactional (user wants to buy something). Intent changes how results are ranked — a navigational query should surface the exact URL; a transactional query should surface product pages.
Entity recognition and linking: "Taylor Swift Eras Tour" contains an entity (Taylor Swift), a sub-entity (Eras Tour), and an implicit intent (concert tickets, tour dates). Entity linking maps surface forms to canonical entities in a knowledge graph.
Query rewriting and expansion: "cheap flights nyc" → "affordable flights New York City JFK LGA EWR". Expansion enriches the query with synonyms and related terms to improve recall. Spelling correction: "macbook probook" → "macbook pro". These are often small fine-tuned models (BERT-based sequence classifiers or seq2seq models) running in under 10ms.
Stage 2: Candidate retrieval
Given the processed query, retrieve hundreds to thousands of candidates from the full corpus. For modern semantic search this is typically hybrid: BM25 for exact matching + dense retrieval for semantic matching (see Posts 70 and 79). Multiple retrieval paths run in parallel and are merged: keyword-based retrieval, semantic retrieval, personalised retrieval (items the user has engaged with previously), trending content retrieval.
Stage 3: Ranking
Score the retrieved candidates with a feature-rich model. Features include: query-document text similarity (dense embedding cosine similarity), BM25 score, user engagement history with the document, document quality signals (click-through rate, dwell time, freshness, authoritative source signals), personalisation features (user interests, location, language). The ranker is a LambdaRank model or a transformer cross-encoder (slower but more accurate — it processes the query and document together in one forward pass, enabling full attention between them).
Cross-encoders: unlike bi-encoders (two-tower) that encode query and document separately, cross-encoders encode the concatenated (query, document) pair. This allows full attention between query and document terms — much more expressive but cannot precompute document embeddings. Used only in the final ranking stage over hundreds of candidates, not retrieval over millions.
Stage 4: Diversity and deduplication
Top-ranked results often cluster around the same content: 5 news articles about the same event, 3 product listings for the same item. Maximal Marginal Relevance (MMR) re-ranks to maximise both relevance and diversity: score_mmr(d) = λ * rel(d) - (1-λ) * max_{d' in selected} sim(d, d'). Near-duplicate detection removes documents with > 70% content overlap.
Stage 5: Result presentation
Which format? Blue links (web search), product cards (e-commerce), inline answer boxes (knowledge panel for "what is the capital of France"), rich snippets (star ratings, price, availability). The format choice itself is a model decision — does this query have a direct answer (show a snippet) or does it require exploration (show diverse blue links)?
Query performance prediction: knowing when you'll fail
Query Performance Prediction (QPP) estimates retrieval quality before retrieving. Low-resource queries (rare entities, new product launches, misspellings) are predicted to have low retrieval quality — the system can fall back to safer defaults or trigger additional retrieval paths. QPP is done with statistical features of the query (IDF of query terms, query clarity score).
Try on Colab: use Haystack (open-source search framework) to build a two-stage semantic search pipeline on a Wikipedia subset. Stage 1: BM25 retrieval (top-100). Stage 2: cross-encoder reranking (top-10). Query a set of factoid questions. Compare MRR@10 for: BM25 only, dense retrieval only, BM25 + cross-encoder reranking. The two-stage pipeline should outperform either single-stage approach by 5-10 MRR points.