Self-Attention: What Q, K, and V Are Actually Doing

Attention is described as "letting every token look at every other token." That is true but incomplete. The mechanism is a soft database lookup — and once you see it that way, the roles of Q, K, and V, the scaled dot product, and the reason multi-head attention works all become concrete rather than mysterious.

Imagine a database: you send a query, the database matches it against keys, and returns weighted values. Attention is this operation made differentiable and applied to sequence positions. That is the whole idea. Everything else is implementation detail.

The three projections: Q, K, V

Given an input sequence of token embeddings, self-attention projects each embedding three ways. The query (Q) projection asks: what information am I looking for? The key (K) projection asks: what information do I contain? The value (V) projection asks: what should I actually return if someone attends to me?

These three projections are separate learned weight matrices applied to the same input. A token at position i produces Q_i, K_i, V_i. The query of position i is compared against the keys of all positions j to produce attention scores: score(i,j) = Q_i · K_j.

Why the dot product measures relevance

A dot product is large when two vectors are aligned — when they point in similar directions in the embedding space. Training pushes the query projections and key projections into a space where semantically relevant pairs align and irrelevant pairs do not. The dot product is the cheapest function that captures this alignment.

The scaling factor: why divide by sqrt(d_k)

With high-dimensional query and key vectors, dot products grow in magnitude proportionally to the dimension d_k. Large dot products push the softmax into very sharp distributions — near one-hot, with near-zero gradients almost everywhere. Dividing by sqrt(d_k) keeps the dot products in a range where softmax produces useful gradients.

Softmax turns scores into weights

After scaling, softmax converts scores into a probability distribution over positions: attention_weights(i,j) = softmax(Q_i · K_j / sqrt(d_k)). The output for position i is the weighted sum of all value vectors: output_i = Σ_j attention_weights(i,j) * V_j.

This is where "every token attends to every other token" comes from. But the attention is soft — even if position i mostly attends to position k, it still gets a small contribution from all other positions. The model learns which positions to weight highly through training.

Multi-head attention: running the lookup h times

Single-head attention uses one Q/K/V projection. Multi-head attention uses h separate sets of projections, runs attention independently on each, and concatenates the results. Why? Because a single attention head can only capture one type of relationship at a time. One head might learn syntactic relationships (subject-verb agreement); another might learn coreference (pronouns pointing back to nouns); another might learn positional proximity. Running h heads in parallel and concatenating gives the model the capacity to capture multiple relationship types simultaneously.

The computational cost is managed by projecting to d_k = d_model / h rather than d_model. The total parameter count and FLOPs are similar to a single full-dimensional head.

What the model learns in the attention weights

Probing trained attention patterns reveals structure: heads in early layers often attend locally (nearby tokens), while heads in later layers attend to semantically related tokens regardless of distance. This is not explicitly programmed — it emerges from the training objective. The model discovers that capturing both local syntax and long-range semantics is useful for predicting the next token.

The quadratic complexity problem

Computing Q · K^T for a sequence of length n produces an n×n attention matrix. Memory and compute scale as O(n^2). For n = 512 this is fine. For n = 100,000 (long documents, whole codebases) it becomes the primary bottleneck. Efficient attention variants (Longformer, FlashAttention, sliding window attention) are all solutions to this quadratic bottleneck while preserving the expressiveness of the attention mechanism.

Try on Colab: implement scaled dot-product attention in PyTorch from scratch — three linear projections, dot product, scale, softmax, weighted sum. Then compare your output against torch.nn.MultiheadAttention on the same input. Extract and visualise the attention weight matrix for a short sentence. See which token pairs get high weights.