SimCLR

Data augmentation, projection head, large-batch contrastive learning, SimCLRv2

A batch of 256 images becomes 512 views. Each image passes through ResNet-50, producing a 2048-dimensional representation h. A 2-layer MLP maps h to a 128-dimensional z. NT-Xent loss runs on z. At fine-tuning time, the MLP is thrown away and a linear layer attaches to h. That discard is the central puzzle of SimCLR: something you train and then immediately delete turns out to be critical.

The key to SimCLR's performance is not the architecture and not the loss—it is the augmentation composition. Random crop plus color jitter plus Gaussian blur forces the encoder to produce representations invariant to photometric and spatial perturbations. Ablate random cropping and accuracy drops sharply. Ablate color jitter and it drops further. Two random crops of the same 224×224 image can overlap by as little as 10% of the original area; the encoder must recognize the same object despite seeing drastically different local regions, which means texture matching fails and semantic encoding becomes necessary.

The projection head protects the encoder from a destructive force. Contrastive loss pressures representations to be invariant to augmentations. Color invariance is useful for the pretext task but harmful downstream—it destroys medical imaging features, defeats texture-based recognition, and removes color cues. The head takes this damage so the encoder does not have to. The encoder retains more information precisely because the loss does not reach it directly. This is not a subtle effect: removing the head and applying NT-Xent directly to h drops linear probe accuracy by roughly 10 percentage points.

SimCLR's practical bottleneck is the large-batch dependency. Good performance requires 4,096–8,192 batch size, which demands 32+ TPU cores for 100 epochs. This is not a fundamental constraint of contrastive SSL—it is an artifact of using in-batch negatives. MoCo solves this structurally with a momentum encoder and queue, decoupling negative count from batch size entirely.

NOT this. "You need a huge batch size for contrastive learning" is wrong—MoCo achieves the same representation quality at batch size 256 on a single 8-GPU machine. SimCLR's large-batch requirement is specific to the in-batch negative design, not to contrastive learning as a paradigm. Understanding why MoCo's queue solves the problem that SimCLR's batch size created is the conceptual test.

Key points

• SimCLR's augmentation pipeline is the performance driver: random crop plus color jitter plus Gaussian blur encodes a prior about which invariances representations should have, and ablating any one transformation drops accuracy sharply. Two random crops of the same 224×224 image can overlap by as little as 10% of the original area. The encoder must recognize object identity across drastically different local regions—local texture matching fails, semantic encoding becomes necessary. The augmentation strategy is not a hyperparameter; it is a design decision about what information the representation should preserve.
• The projection head is always discarded at fine-tuning because it absorbs the destructive invariances that contrastive loss imposes—color, crop, blur—protecting the encoder from losing information useful downstream. NT-Xent pressures representations to be augmentation-invariant. Color invariance helps the pretext task but destroys features that matter downstream. The head takes this damage; the encoder does not. Remove the head and apply loss directly to the encoder output: linear probe accuracy drops ~10 points because the encoder is now forced to discard the same information.
• SimCLR's TPU-scale requirement is an artifact of in-batch negatives, not a fundamental property of contrastive SSL—MoCo's queue decouples negative count from batch size and achieves comparable performance at batch size 256. At N=256, each anchor has 510 negatives; the InfoNCE bound is loose. At N=4096 each anchor has 8190 negatives; the bound tightens and the encoder learns finer distinctions. MoCo achieves 65,536 negatives without a large batch by maintaining a queue of keys from a slowly-updating momentum encoder. Same information-theoretic benefit, fraction of the compute cost.

Recap

• Augmentation is the performance driver: random crop + color jitter + Gaussian blur; ablate any one and accuracy drops sharply.
• Random crop forces semantics: two crops can overlap ~10% → texture matching fails, object identity must be encoded.
• Projection head is discarded at fine-tuning — it absorbs destructive invariances (color/crop/blur); apply NT-Xent directly to the encoder and linear probe drops ~10 points.
• Encoder (2048-dim) stays rich; head (128-dim) compresses to augmentation-invariant.
• Large-batch need (4,096–8,192) is an in-batch-negatives artifact, not fundamental — MoCo hits the same quality at batch 256 on 8 GPUs.
• Domain matters: in medical imaging color is diagnostic, so color jitter hurts — pick invariances the domain allows.

Check your understanding