Embeddings: What It Means to Represent Meaning as Geometry

Word2Vec discovered that word meaning has geometric structure: King - Man + Woman ≈ Queen. This is not a coincidence or a trick — it is a direct consequence of how the embedding was trained. The distributional hypothesis predicts it; the training objective enforces it. Understanding this makes the jump to contextual embeddings (BERT, GPT) and to dense retrieval natural.

An embedding is a mapping from a discrete object (a word, a user, a product) to a point in a continuous vector space. The power of embeddings comes from the geometry that emerges in that space: objects with similar properties occupy similar regions. This geometry is not imposed — it is learned from patterns in data. Understanding how and why it emerges makes it easier to use embeddings correctly and to debug failures.

The distributional hypothesis

You shall know a word by the company it keeps. Words that appear in similar contexts have similar meanings. "Dog" and "cat" both appear near "pet," "fur," "veterinarian," "home." "Bank" appears near "money" and "loan" in some contexts, and near "river" and "shore" in others. The distributional hypothesis says that the patterns of co-occurrence in a large corpus capture semantic relationships.

Word2Vec operationalises this hypothesis as a prediction task. Skip-gram: given a word, predict the surrounding context words. CBOW: given context words, predict the centre word. In both cases, the embeddings are trained as the weight matrix of a shallow neural network. Words with similar contexts receive similar gradient updates and converge to similar embedding vectors.

Why linear arithmetic works: King - Man + Woman = Queen

After training on a large corpus, word vectors have a property that seems magical: vector arithmetic on word embeddings captures semantic relationships. king - man + woman ≈ queen. paris - france + italy ≈ rome. Why?

If "man" and "king" appear in similar contexts except for gender-related words, their embeddings will be similar in most dimensions and differ in a gender-direction. "woman" and "queen" have the same relationship. The gender difference is a consistent direction in the embedding space because gendered words consistently co-occur with gender-related context words. The arithmetic works because the gender direction is approximately the same for (man, king) as for (woman, queen) — the distributional structure enforces parallel geometry.

This is not a special property of Word2Vec — it is a property of any representation learned from distributional patterns at sufficient scale.

The limitation: one embedding per word

Word2Vec assigns a single vector to each word. "Bank" (financial) and "bank" (river) share the same embedding — a compromise between the two senses that represents neither well. For downstream tasks requiring contextual understanding, this is a significant limitation.

Contextual embeddings: different contexts, different vectors

ELMo (2018), BERT (2018), and GPT (2018) replaced static word embeddings with contextual embeddings: the same word receives a different embedding depending on the sentence it appears in. The embedding of "bank" in "the river bank" and in "the bank account" are different vectors produced by running the sentence through a deep model (bidirectional LSTM or Transformer) and reading the hidden state at the word's position.

Contextual embeddings capture polysemy by construction. They are also richer: they encode not just word identity but syntactic role, discourse position, and local context. Transfer learning with contextual embeddings (fine-tune a pretrained BERT for a downstream task) became the dominant paradigm in NLP from 2018 onward.

Dense retrieval: embeddings for search

Once you have sentence embeddings, you can do semantic search: embed a query, embed all candidate documents, find the nearest neighbours. This is dense retrieval, as opposed to sparse retrieval (BM25, TF-IDF). Dense retrieval finds semantically similar documents even with no lexical overlap — "cardiac arrest" and "heart attack" are near neighbours in a good embedding space even though they share no words.

The key engineering challenge: approximate nearest-neighbour search at scale. Libraries like FAISS index millions of embeddings and return approximate nearest neighbours in milliseconds. The accuracy/speed tradeoff in ANN is controlled by the index type (HNSW for high accuracy, IVF for speed) and the number of probe cells.

Try on Colab: train Word2Vec (gensim) on a text corpus. Visualise the top 200 most frequent words in 2D using UMAP or t-SNE. Semantic clusters (countries, professions, emotions) should be visible as spatial clusters. Then test vector arithmetic: find the 5 nearest neighbours to king - man + woman. Compare the result with a contextual model: embed the same words from a BERT sentence and see if the arithmetic still holds.