Collapse-free SSL Without Negatives

BYOL, Barlow Twins, VICReg — why they don't collapse, scale limitations

Contrastive methods need negatives to prevent representational collapse, but negatives introduce the false negative problem—treating same-class images as push-away targets. BYOL's 2020 paper claimed to remove negatives entirely by using a stop-gradient and a momentum EMA target network, with an asymmetric predictor. The paper could not explain analytically why this did not collapse. The real answer arrived in the follow-up literature: BYOL prevents collapse because BatchNorm computes statistics across the batch, making each sample's representation a function of all other samples in the batch. This implicit cross-sample interaction acts as an implicit negative mechanism. Replace BatchNorm with LayerNorm—per-sample normalization, no cross-batch interaction—and BYOL collapses immediately.

The running example makes this concrete. BYOL uses an online network updated by backprop plus a target network updated by EMA (m≈0.996). An additional predictor MLP maps online representations to target representations. The loss minimizes L2 distance between predictor(online(view1)) and stop_gradient(target(view2)). Stop-gradient on the target branch is necessary—without it both networks collapse together by setting everything to zero. But stop-gradient alone is not sufficient: the collapse resistance comes from BatchNorm, and removing it with a larger batch (65,536) actually breaks BYOL because BatchNorm statistics converge to population statistics, eliminating the noisy cross-sample interaction that provides the implicit negative mechanism.

Barlow Twins takes a more direct approach: push the cross-correlation matrix between two views' normalized feature vectors toward identity. The invariance term (diagonal → 1) makes each feature dimension correlate with itself across views. The redundancy reduction term (off-diagonal → 0) forces different feature dimensions to decorrelate with each other—this is the explicit collapse-prevention mechanism, directly enforcing dimensional diversity without any negatives, stop-gradients, or momentum.

NOT this. "Removing negatives means you can use smaller batches freely" is incomplete. BYOL does work at batch size 256–512, unlike SimCLR. But at very large batches—65,536—BYOL collapses because BatchNorm statistics stabilize and the implicit negative mechanism disappears. The batch size freedom is real within a range; it does not extend to arbitrarily large batches.

Key points

• BYOL's real collapse-prevention mechanism is BatchNorm, not stop-gradient: BatchNorm computes statistics across the batch, creating implicit cross-sample interactions that prevent all embeddings from collapsing to a constant. Replace BatchNorm with LayerNorm—per-sample normalization—and BYOL collapses immediately. Stop-gradient is necessary to prevent both networks from trivially collapsing together, but it is not the mechanism that prevents the constant-representation solution. This finding reframed how BYOL's collapse resistance was explained, though it is not the final word: a follow-up study from BYOL's own authors showed group normalization plus weight standardization also avoids collapse without any batch statistics, so cross-batch interaction helps but is not strictly required.
• Barlow Twins prevents collapse explicitly via redundancy reduction: the off-diagonal terms of the cross-correlation matrix are pushed toward zero, forcing different feature dimensions to encode distinct information. The invariance term (diagonal → 1) pushes each feature to correlate across views. Without the redundancy reduction term, all feature dimensions could encode the same scalar function—the diagonal constraint alone allows dimensional collapse. The off-diagonal constraint is the collapse-prevention mechanism, and it is interpretable and debuggable in a way that BYOL's BatchNorm mechanism is not.
• At production scale with ViTs, contrastive methods outperform negative-free methods because explicit negatives provide a curriculum of hard discrimination that implicit BatchNorm mechanisms cannot match at large batch sizes. At batch size 65,536, BatchNorm statistics converge to population statistics, eliminating BYOL's implicit cross-sample interactions. The collapse resistance weakens exactly when scale increases. For production-scale ViT pretraining, MoCo v3 or CLIP-style contrastive outperforms BYOL.
• VICReg prevents collapse with an explicit third mechanism, neither BYOL's implicit BatchNorm trick nor Barlow Twins' cross-correlation-to-identity target: a variance term directly penalizes any embedding dimension whose per-batch standard deviation drops below a threshold. Alongside variance, VICReg adds an invariance term (MSE between the two views' embeddings, like BYOL/Barlow Twins' alignment goal) and a covariance term (off-diagonal covariance pushed toward zero, the same redundancy-reduction idea as Barlow Twins' off-diagonal constraint). Because the variance term directly penalizes collapse rather than relying on batch statistics or a specific cross-correlation target, VICReg can apply its three losses asymmetrically per branch and doesn't depend on BatchNorm at all — it collapse-proofs itself by construction rather than by side effect.

Recap

• Negatives prevent collapse but cause false negatives: same-class images treated as push-away.
• BYOL setup: online (backprop) + target (EMA m≈0.996) + predictor MLP, minimize L2 to stop-gradient target.
• Real collapse-preventer is BatchNorm, not stop-gradient: cross-batch stats = implicit negatives; swap to LayerNorm and BYOL collapses immediately.
• Stop-gradient necessary but not sufficient — without it both nets collapse to zero.
• BYOL breaks at batch 65,536: BatchNorm stats stabilize → implicit-negative mechanism vanishes.
• Barlow Twins = explicit fix: cross-correlation → identity; diagonal→1 (invariance), off-diagonal→0 (redundancy reduction = collapse prevention).
• At ViT/production scale, contrastive (MoCo v3 / CLIP) beats negative-free.

Check your understanding