Word2Vec: Negative Sampling, Embedding Geometry, and Why King − Man + Woman = Queen

Word2Vec (2013) was the moment NLP became geometry. Before it, words were discrete tokens. After it, words were points in a space where direction and distance had meaning. Every embedding system since — GloVe, BERT, GPT, CLIP — is a successor to this idea. Understanding word2vec means understanding why the geometry works, not just that it does.

Word2Vec (Mikolov et al., Google, 2013) is the paper that created the modern notion of a word embedding — a dense vector representation of a word that captures semantic and syntactic relationships through geometric proximity. It is the ancestor of every embedding system in use today.

Before word2vec: one-hot encodings

Before 2013, NLP typically represented words as one-hot vectors: a vector of length |vocabulary| with a 1 in the position of the word and 0 elsewhere. These are orthogonal — every pair of words has zero cosine similarity regardless of semantic relationship. "Cat" and "kitten" are no more similar than "cat" and "airplane." You cannot generalise from training examples about cats to test examples about kittens.

The distributional hypothesis

"You shall know a word by the company it keeps" — Firth, 1957. Words that appear in similar contexts tend to have similar meanings. "Bank" appears near "money," "loan," "finance" in financial contexts and near "river," "fish," "shore" in geographic contexts. This polysemy is a problem for static embeddings but the core insight — context defines meaning — is correct and powerful.

Skip-gram: the word2vec training objective

Skip-gram model: given a target word w_t, predict the context words within a window of size k. For each target word, we define the positive context words (those that appear within the window) and learn embeddings that maximise the probability of context words given the target. The probability: P(w_c | w_t) = exp(v_{w_c}ᵀ u_{w_t}) / Σ_{w'} exp(v_{w'}ᵀ u_{w_t}), where u_{w_t} is the target embedding and v_{w_c} is the context embedding. The denominator sums over the entire vocabulary — O(|V|) per gradient step, which is too expensive for vocabulary sizes of millions.

Negative sampling: the computational trick

Instead of computing the full softmax, negative sampling trains a binary classifier: does this (target, context) pair come from real co-occurrence or from random sampling? Objective: for each positive pair (w_t, w_c), maximise log σ(v_{w_c}ᵀ u_{w_t}) + Σ_{k=1}^{K} E_{w_neg~P_n}[log σ(-v_{w_neg}ᵀ u_{w_t})]. The first term pushes the positive context word's embedding to be similar to the target. The K negative terms push K randomly sampled words (from the unigram^{3/4} distribution) to be dissimilar. This is O(K) per step instead of O(|V|). Typical K = 5-20. The unigram^{3/4} distribution downweights frequent words (reducing their dominance as negatives) and upweights rare words.

CBOW vs Skip-gram

CBOW (Continuous Bag of Words): predict the target word from the average of context embeddings. Faster training, slightly worse for rare words (averaging loses the individual context structure). Skip-gram: predict context words from the target. Slower to train but better on rare words. Skip-gram is generally preferred.

Why the geometry works

After training, the learned embeddings capture semantic analogies through vector arithmetic: king - man + woman ≈ queen. This works because the embedding space has learned consistent directions for semantic relationships. The direction man→king is approximately the same as woman→queen because co-occurrence patterns around "man" vs "woman" modify "king" vs "queen" in consistent ways. These are not directions we engineered — they emerged from the training objective. The phenomenon is more surprising than it sounds and is not fully theoretically understood.

Word2Vec as implicit matrix factorisation

Levy & Goldberg (2014) showed that skip-gram with negative sampling is implicitly factorising the matrix M_{ij} = PMI(wᵢ, wⱼ) - log k, where PMI is pointwise mutual information and k is the number of negatives. GloVe (Pennington et al., 2014) makes this explicit: it directly fits word vectors to the log co-occurrence matrix, using weighted least squares. GloVe is faster to train and produces similar quality embeddings.

From word2vec to modern embeddings

Word2Vec produces static embeddings: each word has one vector regardless of context. "Bank" has one embedding that tries to average over financial and geographic usages. BERT (Post 67) produces contextual embeddings: the embedding for "bank" depends on the sentence it appears in. The attention mechanism computes context-aware representations. CLIP (Post 69) extends the co-occurrence idea to image-text pairs: images and captions that co-occur should have similar embeddings.

Interview questions on this topic

"Explain the king - man + woman = queen analogy. Why does this work geometrically?" — The embedding space learns a consistent direction for 'royalty' and a consistent direction for 'gender.' The vector (king - man) captures the royalty direction for males; adding woman's embedding yields a point near queen because the royalty-female neighbourhood is near queen. The analogy works when the relationship is consistent and well-represented in the training corpus.

"What is the difference between word2vec and TF-IDF? When would you use each?" — TF-IDF is a sparse, high-dimensional, count-based representation. It captures word importance per document but not semantic similarity. Word2Vec is dense, low-dimensional, distributed. Two words with identical TF-IDF fingerprints could be semantically unrelated; two words with nearby word2vec embeddings are semantically similar. Use TF-IDF for exact keyword matching and search; word2vec/embeddings for semantic similarity and as features for downstream models.

"Why does negative sampling use a 3/4 power of the unigram distribution?" — It empirically outperforms the raw unigram and uniform distributions. The 3/4 power reduces the dominance of very frequent words (which would otherwise be sampled too often as negatives and provide low-information training signal) and increases the relative sampling rate of rare words.

"What are the failure modes of static word embeddings like Word2Vec?" — Polysemy (one vector per word regardless of sense). Poor handling of rare words (few co-occurrences → noisy vectors). No subword information (misspellings or morphological variants are unknown). Cultural and corpus biases are encoded and amplified. BERT and fastText (subword) address the last two.

Try on Colab: train word2vec using gensim on the text8 corpus (Wikipedia subset). Extract embeddings for 'man', 'woman', 'king', 'queen', 'paris', 'france', 'london', 'england'. Verify king-man+woman≈queen and paris-france+england≈london using cosine similarity. Visualise 50 words using TSNE — observe that semantically related words cluster together.