Contrastive Learning: How CLIP Aligns Images and Text Without Labels

Self-supervised learning eliminates the need for labels by defining the training objective from the data structure itself. Contrastive methods do this by pulling together representations of similar pairs and pushing apart dissimilar ones. CLIP applies this to 400 million image-text pairs from the internet, producing multimodal embeddings that enable zero-shot classification, image search, and generation guidance.

The bottleneck in supervised learning has always been labels. Collecting ImageNet-scale labelled data required years and millions of dollars. Self-supervised learning removes this bottleneck by defining the training signal from unlabelled data itself — the task is chosen so that solving it requires learning useful representations.

SimCLR: contrastive learning for vision

SimCLR (Chen et al., 2020) is the clearest expression of the contrastive idea for images. Given an image, apply two random augmentations (crop, colour jitter, blur) to produce two views of the same image. These two views form a positive pair — they should have similar representations. All other images in the batch form negative pairs — they should have dissimilar representations.

The contrastive loss (NT-Xent): for a positive pair (i, j), maximise the cosine similarity of their representations relative to all other pairs in the batch. L = -log[exp(sim(z_i, z_j)/τ) / Σ_{k≠i} exp(sim(z_i, z_k)/τ)]. Temperature τ controls how sharply the distribution peaks. The model learns representations where augmented views of the same image are nearby and different images are far apart.

This works because augmentations remove information that should not matter (exact crop position, colour temperature) while preserving information that should (object identity, shape). The model is forced to be invariant to augmentations — which means it must capture the invariant content.

The representation quality

After pretraining with contrastive loss on ImageNet without labels, SimCLR representations (extracted with a linear probe) achieve within 7% of supervised ResNet accuracy on ImageNet classification. With fine-tuning, the gap closes further. This was a landmark result: competitive visual representations learned without any manual labels.

CLIP: contrastive pretraining across modalities

CLIP (Radford et al., 2021) applies contrastive learning across modalities: the positive pairs are (image, caption) pairs from the internet. The model jointly trains an image encoder (ViT or ResNet) and a text encoder (Transformer). For a batch of N image-text pairs, the N correct pairings are positive; the N^2 - N incorrect pairings are negative. The loss maximises similarity of matched pairs relative to mismatched ones.

Trained on 400 million image-text pairs from the web, CLIP learns a shared embedding space where images and their descriptions are nearby. Zero-shot classification becomes prompt engineering: to classify an image into k categories, encode all category names as text ("a photo of a dog"), encode the image, take the nearest text embedding — no fine-tuning needed.

Why CLIP generalises

CLIP's representations generalise to tasks never seen in training because the internet descriptions provide semantic supervision for a vast range of visual concepts. Unlike ImageNet-trained models that learn 1000 specific classes, CLIP learns continuous associations between visual content and language. On the ObjectNet benchmark (specifically designed to test out-of-distribution generalisation), CLIP significantly outperforms ImageNet-supervised models.

CLIP in production

CLIP embeddings are widely used in: semantic image search (embed the query text, find nearest-neighbour image embeddings), content moderation (detect NSFW or policy-violating images by measuring similarity to risk-describing text), recommendation (align user query embeddings with item image embeddings), and Stable Diffusion (CLIP text encoder drives the conditioning in latent diffusion models — the text prompt is processed by CLIP's text encoder to guide the denoising process).

DINO and MAE: self-supervised without negatives

Subsequent work showed negatives are not required. DINO (Caron et al., 2021) uses a teacher-student setup where the student matches the teacher's representations under different augmentations. MAE (He et al., 2022) masks 75% of image patches and trains a ViT to reconstruct them — no negatives, no contrastive loss, just reconstruction from partial context. Both produce representations competitive with CLIP's for vision tasks. The common thread: a pretext task that requires understanding global image structure.

Try on Colab: use the openai/clip-python package. Load CLIP ViT-B/32. Embed 100 images from CIFAR-100. Embed the class name strings ("a photo of a {class_name}"). For each image, compute cosine similarity to all 100 class text embeddings and pick the top-1. Report zero-shot accuracy. Then compare to a fine-tuned ResNet-18 on 500 labelled CIFAR-100 examples — observe the zero-shot vs. few-shot trade-off.