Downstream Adaptation from SSL

Linear probing, fine-tuning, prompt tuning, adapters, catastrophic forgetting, representation evaluation

MAE achieves 68% ImageNet linear probe accuracy. SimCLR achieves 70%. On this metric, SimCLR wins. Full fine-tuning reverses the result: MAE reaches 83.6%, SimCLR reaches approximately 76%. The same pretrained representations, evaluated two ways, produce opposite rankings. This is not a measurement error—it is a direct counterexample to the assumption that linear probe accuracy predicts fine-tuning performance, and understanding why the divergence exists is more useful than memorizing the numbers.

The divergence comes from what each pretraining objective preserves. SimCLR's contrastive loss organizes representations around object-level similarities—which invariances to augmentation the model should learn. Representations become linearly separable by ImageNet category because the loss explicitly pushes same-class views together and different-class views apart. But the same loss discards fine-grained local detail: color invariance, crop invariance, texture invariance are all baked in. MAE's pixel reconstruction objective preserves that local detail because reconstruction at 75% masking requires encoding spatial relationships, texture, and pixel-level structure. This information is not linearly organized by ImageNet category—it is distributed across the representation in ways a frozen linear layer cannot access. A fine-tuned nonlinear network can access it, which is why MAE's fine-tuning performance exceeds SimCLR's.

The adaptation strategy is not purely a compute decision. Full fine-tuning on small labeled datasets risks catastrophic forgetting: the downstream supervised loss overwrites pretrained weights in a few hundred steps. The gradient signal from 1,000 labeled examples is insufficient to provide stable weight updates across the full model depth—each gradient step in the wrong direction is not corrected by subsequent steps because the dataset is too small to cover the loss landscape adequately. Fine-tuning on fewer than 10,000 examples without regularization is high risk. With 1,000 labeled examples and an MAE pretrained ViT, the right strategy is LoRA or adapters—not full fine-tuning—with a low learning rate and early stopping.

Layer-wise learning rate decay (LLRD) encodes a prior about what each layer should do. Lower layers encode general features (edges, textures, spatial relationships) that should not change for a new downstream task. Upper layers encode task-specific representations that should adapt. Scaling the learning rate by d^{L-l} (d ≈ 0.75) gives near-zero rates to lower layers and base rates to upper layers. This is standard practice for BERT and ViT fine-tuning precisely because it substantially reduces catastrophic forgetting without sacrificing adaptation capacity in the layers that need it.

NOT this. "Linear probe accuracy predicts fine-tuning performance" is directly refuted by MAE vs SimCLR. The linear probe measures whether downstream task structure is linearly separable in the pretrained space. MAE's representations are richer than SimCLR's but less linearly organized—a distinction the linear probe cannot detect. When choosing between SSL models for fine-tuning, run both evaluations; the linear probe alone will mislead.

Key points

• Linear probe accuracy does not predict fine-tuning performance: MAE achieves 68% linear probe but 83.6% fine-tuned accuracy; SimCLR achieves 70% linear probe but only ~76% fine-tuned—the same representations rank opposite ways under the two evaluation protocols. MAE's pixel reconstruction preserves fine-grained local information not linearly organized by ImageNet category. A frozen linear layer cannot access it; a fine-tuned nonlinear network can. SimCLR's contrastive loss organizes representations linearly by object-level similarity but discards color, texture, and local detail. Linear probe measures linear separability, not representation richness.
• Full fine-tuning on small labeled datasets risks catastrophic forgetting: downstream gradients from insufficient data overwrite pretrained weights faster than stable weight updates can form. With fewer than 10,000 examples, gradient signals are too sparse to cover the loss landscape. Each update overwrites pretrained structure without correction. With 1,000 labeled examples, LoRA or adapters with a low learning rate (1e-5 to 5e-5) and layer-wise learning rate decay preserve pretrained representations while adapting the parts that need to change.
• LoRA approximates weight updates as ΔW = AB (A ∈ ℝ^{d×r}, B ∈ ℝ^{r×d}, r << d), training only A and B while keeping W frozen—preserving pretrained representations while providing task-specific adaptation capacity at ~1% of full fine-tuning parameter count. At inference, W_new = W + AB merges the LoRA update with no latency overhead. This is why LoRA is the default for adapting large pretrained models to new tasks: zero catastrophic forgetting, minimal parameters, and adaptation capacity determined by rank r rather than full model depth.
• Prompt tuning keeps the entire model frozen and instead learns a small set of continuous "soft prompt" embeddings prepended to the input — no weight matrices are touched at all, so it trains even fewer parameters than LoRA. It's the right choice with very few examples (under about 50) or when the new task is close to in-distribution with pretraining, needing no real internal feature shift — and it lets you serve many different tasks from one frozen backbone, each just swapping in its own soft prompt. LoRA is the better choice once the domain shift goes deeper than surface-level (new syntax, new modality) and you have 100+ examples to support adapting internal features, which prompt tuning cannot touch.

Recap

• Linear probe ≠ fine-tuning: MAE 68% probe / 83.6% fine-tuned vs SimCLR 70% probe / ~76% — opposite ranks.
• Why: MAE preserves fine-grained local detail not linearly organized; a fine-tuned nonlinear net accesses it, a frozen linear layer can't.
• Full fine-tuning on <10K examples risks catastrophic forgetting: sparse gradients overwrite pretrained weights without correction.
• With ~1,000 labels: use LoRA/adapters, low LR (1e-5–5e-5), early stopping.
• LoRA: ΔW = AB (r ≪ d), train A,B only; ~1% params, zero forgetting, merges at inference (W+AB) with no latency.
• LLRD (d≈0.75, scale d^{L-l}): near-zero LR low layers (general), base LR upper layers (task-specific).
• LoRA vs prompt tuning: LoRA for internal feature shifts / 100+ examples; prompt tuning for in-distribution tasks, <50 examples, many tasks from one model.

Check your understanding