Contrastive Loss Functions
NT-Xent, InfoNCE, temperature τ, negative mining, uniformity and alignment
In a batch of 512 images, each image has one positive pair—a second augmented view of itself—and 1,022 negatives: every other image in the batch. The NT-Xent loss for positive pair (i,j) is: -log[exp(sim(zᵢ,zⱼ)/τ) / Σ_{k≠i} exp(sim(zᵢ,zₖ)/τ)]. That formula looks mechanical until you understand what each piece does. The cosine similarities in the denominator are all the "wrong" answers the model must learn to reject. The temperature τ controls how much the model focuses on the hardest wrong answers versus treating all negatives equally.
Temperature is the hyperparameter practitioners misset most. Set τ = 0.07 (SimCLR's default) and the model concentrates gradient on the negatives closest to the anchor—useful, informative signal. Set τ = 0.01 and exp(sim/0.01) overflows: exp(1/0.01) ≈ 2.7×10^43, the softmax denominator saturates, gradients explode, training collapses to NaN within 100 steps. Set τ = 1.0 and every negative gets nearly equal weight, slowing convergence to a crawl because the signal cannot distinguish hard from easy negatives. The useful operating range is 0.07–0.2, and knowing why that range exists is more useful than memorizing it.
More negatives is not just empirically better—it is formally justified. NT-Xent is an instantiation of InfoNCE, a lower bound on mutual information I(X;C). The bound tightens as the number of negatives increases. At 512 images per batch, each anchor has 1,022 negatives; at MoCo's queue of 65,536, each anchor has 65,536 negatives. The mutual information lower bound is tighter, the discrimination task is harder, and the encoder must learn finer-grained features to identify the correct positive. This is the formal reason large batches help SimCLR and the formal reason MoCo's queue was designed to decouple negative count from batch size.
Hard negatives—items close to the anchor in embedding space but not actually the same image—carry far more gradient than easy negatives already pushed far apart. A model that has learned basic discrimination finds easy negatives uninteresting: they contribute near-zero gradient. Hard negatives force the model to learn fine-grained distinctions. But hard negatives increase false negative risk: two cat images in the same batch are treated as negatives even though they should cluster together. Debiased contrastive loss estimates and subtracts this same-class contribution from the denominator.
NOT this. The intuition "just lower τ to make training harder" is wrong and dangerous. τ < 0.05 causes numerical overflow before any useful learning occurs. The useful range of harder training is 0.07–0.12, not 0.01. Stabilization techniques (log-sum-exp, gradient clipping, warmup) can extend this slightly, but the fundamental constraint is the exponential—smaller τ raises every similarity to a power that overflows floating point.
Key points
- NT-Xent is a classification problem over one positive and N−1 negatives, and the InfoNCE bound tightens as N grows—this is why more negatives is formally justified, not just empirically observed. For each positive pair (i,j), the loss is -log[exp(sim(zᵢ,zⱼ)/τ) / Σ_{k≠i} exp(sim(zᵢ,zₖ)/τ)]. At N=512, the bound is loose and discrimination is relatively easy. At N=65,536 (MoCo's queue), the model must learn genuinely fine-grained features to identify the correct positive among 65,536 wrong answers.
- Temperature τ controls gradient concentration: too low causes numerical overflow, too high produces indiscriminate signal—the useful range is 0.07–0.2. At τ=0.07, gradients concentrate on hard negatives closest to the anchor. At τ=0.01, exp(sim/0.01) overflows (exp(100) ≈ 2.7×10^43), producing NaN loss within 100 steps. At τ=1.0, all negatives contribute nearly equal gradient and the model cannot distinguish hard from easy. When contrastive training is unstable, check τ first.
- Hard negatives accelerate convergence but increase false negative risk—the two problems trade off directly, and debiased contrastive loss is the tool for managing the tradeoff. Hard negatives (items close to the anchor but not the same image) carry large gradient and force fine-grained discrimination. But two cat images in the same batch get treated as negatives, generating gradient that pushes similar semantics apart. Debiased loss estimates the same-class fraction and corrects the denominator, recovering unbiased gradient signal.
Temperature τ is the most consequential contrastive hyperparameter: it controls whether gradients concentrate on the hardest negatives (low τ, overflow risk below 0.05) or spread uniformly over all negatives (high τ, slow convergence). More negatives tightens the InfoNCE mutual information lower bound—so methods like MoCo's 65,536-entry queue exist not as engineering convenience but as a formal improvement in what is being optimized.
Recap
- NT-Xent = classify 1 positive among N−1 negatives: -log[exp(sim(zᵢ,zⱼ)/τ) / Σ_{k≠i} exp(sim(zᵢ,zₖ)/τ)].
- More negatives = tighter InfoNCE bound: batch 512 → 1,022 negs; MoCo queue → 65,536 negs, formally harder.
- Temperature τ range 0.07–0.2: τ=0.01 overflows (exp(100)≈2.7×10⁴³) → NaN in ~100 steps; τ=1.0 → indiscriminate, slow.
- Check τ first when contrastive training is unstable.
- Hard negatives speed convergence but raise false-negative risk: two cats in a batch pushed apart.
- Debiased loss estimates the same-class fraction and corrects the denominator.
Check your understanding
Q1. SimCLR with a batch size of 256 gives poor results. Increasing to 4096 dramatically improves performance. Explain the mechanism.
- A) Larger batches improve gradient stability purely through better Monte Carlo estimates of the mean gradient direction, which drives SimCLR's improvement
- B) At N=256, each anchor has 510 negatives — the InfoNCE bound is loose; at N=4096 it has 8190, the bound tightens, forcing finer features; MoCo decouples negatives from batch size via a queue
- C) Larger batches simply increase the probability that true positive pairs appear together in the same batch, which is the core requirement NT-Xent depends on
- D) Batch size affects only wall-clock training speed, not representation quality — the accuracy improvement at 4096 must come from longer effective training time
Q2. You set temperature τ = 0.01 (very low) and training collapses to NaN loss after 100 steps. What happened?
- A) Very low temperature causes the model to ignore all negatives entirely, making the loss degenerate to exactly zero without any learning occurring
- B) At τ=0.01, exp(sim/τ) overflows (exp(100)≈2.7×10⁴³), causing NaN in softmax; fix with τ≥0.05 and gradient clipping
- C) τ=0.01 is simply too small a number to distinguish between positive and negative pairs, causing the model to assign uniform similarity to every pair
- D) Very low temperature causes the L2 normalization step applied to embeddings to fail numerically, and that normalization failure is the true source of the NaN
Q3. Explain why debiased contrastive loss is needed and sketch how it corrects for false negatives.
- A) Debiased contrastive loss is needed because standard NT-Xent mistakenly treats the anchor sample itself as one of its own negatives, which must be corrected
- B) Standard NT-Xent treats same-class examples as false negatives; debiased loss estimates and subtracts that fraction from the denominator
- C) Debiased contrastive loss addresses the false-negative problem by discarding every example that comes from the same data domain as the current anchor
- D) False negatives in contrastive learning are only ever a problem at small batch sizes; debiased loss becomes unnecessary once batch size exceeds roughly 4096
Q4. Select the two true statements about the relationship between uniformity and alignment in NT-Xent.
- A) Maximizing uniformity alone pushes all embeddings apart on the hypersphere, without regard to which pairs are positives
- B) Maximizing alignment alone pulls all points together and can collapse the whole embedding space to a single point
- C) Uniformity and alignment are two hyperparameters that get set once and never change during the course of training
- D) NT-Xent optimizes only alignment; uniformity is claimed to emerge automatically once alignment converges to zero
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →