SSL on Non-Image and Non-Text Data

Tabular SSL (SCARF, VIME), graph SSL, audio SSL (wav2vec, HuBERT), domain applicability

A fraud detection dataset: 200 features, 5% labeled, 95% unlabeled. A supervised XGBoost model on the labeled 5% achieves baseline AUC. SCARF—self-supervised contrastive learning using random feature corruption—pretrain on all data, then fine-tune on the labeled 5%, and AUC improves. The question worth understanding is: why does this particular corruption strategy work, and when does it not?

SCARF corrupts a fraction α of feature values by replacing them with draws from that feature's marginal distribution. The original row and the corrupted row form a positive pair. NT-Xent loss runs over positive and in-batch negative pairs. The key design choice is marginal-distribution corruption: a drawn value looks plausible—it falls within the natural range of that feature—so the model cannot detect corruption by checking whether values are out-of-range. Detection requires understanding how features relate to each other. If a patient's age is 35 but their feature for "years since retirement" suddenly draws a value of 12, the model must understand that these features are correlated to notice the inconsistency. The SSL pretraining forces encoding of inter-feature relationships, which is exactly what transfers to downstream fraud detection. VIME takes a complementary route to the same problem: instead of a contrastive pull-push objective, it pretrains the encoder on two reconstruction pretext tasks—estimating which features were masked and reconstructing their original corrupted values—forcing the same inter-feature dependency encoding through denoising rather than contrastive alignment.

This logic also predicts when tabular SSL fails. If features are heavily engineered—explicitly encoding the correlations that SCARF would otherwise force the model to learn—then pretraining learns redundant structure. If the labeled dataset is large enough that supervised training can already capture inter-feature relationships directly, SSL adds noise without signal. The overhead is only justified when unlabeled data is substantially larger than labeled data and features are not already explicitly relational.

Audio SSL (wav2vec 2.0, HuBERT) is the most successful SSL transfer outside text and vision precisely because speech has natural temporal structure—phonemes, words, prosody—that maps cleanly onto masked prediction without special augmentation design. Fine-tuning on just 10 minutes of labeled speech still achieves a competitive word error rate — a separate, even more extreme low-label result. The paper's headline "100× less labeled data" figure specifically compares its 100-hour labeled fine-tune to prior state of the art, not the 10-minute case.

NOT this. "Tabular SSL always helps when you have unlabeled data" is wrong. If features are heavily engineered and correlations are already explicit, SSL learns redundant structure it cannot leverage downstream. The unlabeled data advantage only materializes when the SSL pretext task discovers non-obvious inter-feature relationships that supervised training on the small labeled set would miss.

Key points

• SCARF corrupts features by drawing from their marginal distributions, not from noise—because plausible-looking values force the model to detect corruption through inter-feature relationships rather than out-of-range detection. A Gaussian noise corruption at 3σ is detectable from the feature's own distribution alone. A marginal-distribution draw falls within the natural range; detection requires noticing that this age conflicts with that retirement year, or that this income conflicts with that transaction amount. The pretext task forces encoding of exactly the inter-feature dependencies that transfer to downstream tasks.
• Tabular SSL helps when unlabeled data is abundant, labeled data is scarce, and features have non-obvious correlations—it fails when features are heavily engineered or the labeled set is large enough for direct supervised learning. If explicit feature engineering already captures correlations (ratio features, interaction terms), SSL cannot discover structure beyond what is already visible. The break-even point is roughly when unlabeled data is 10× the labeled set and the feature space has high-dimensional correlations that domain-expert engineering has not yet captured.
• Audio SSL (wav2vec 2.0) needs remarkably little labeled data: 10 minutes gets a competitive word error rate, and separately, its 100-hour labeled subset matches prior state of the art while using 100× less labeled data — because temporal masking maps cleanly onto the same masked prediction framework that works for text, without special augmentation design. Speech has natural temporal structure (phonemes, words, prosody) analogous to linguistic structure in text. A CNN encoder produces local audio representations; a Transformer contextualizes them; a quantizer discretizes them into codebook entries; contrastive loss predicts the quantized representation of masked timesteps. The discrete codebook provides stable classification targets analogous to BERT's token vocabulary — using the continuous (unquantized) representations directly as targets instead risks collapse to a constant representation, since there is no discrete assignment step to break the symmetry that pushes different timesteps toward different targets.
• Graph SSL augmentations must respect what an edge actually means in that graph — the same edge-drop augmentation (GraphCL) is valid for social networks but breaks molecular graphs. In a social network, an edge is a partial, noisy observation of a relationship — dropping some edges is a plausible view of incomplete data, so contrastive pretraining across edge-dropped views teaches robustness to missing links. In a molecule, an edge is a bond that defines the molecule's chemical identity — dropping it doesn't create a noisy view of the same molecule, it creates a different molecule (or an invalid one). The lesson generalizes beyond graphs: any augmentation strategy has to preserve the property that makes two views "the same underlying thing," and what counts as noise in one domain can be identity-breaking in another.

Recap

• SCARF (tabular): corrupt fraction α of features with draws from each feature's *marginal* distribution; original vs corrupted = positive pair.
• Marginal draws look plausible → can't detect by out-of-range; forces encoding inter-feature relationships (age vs years-since-retirement).
• Tabular SSL fails when features are heavily engineered or labels are plentiful: break-even ≈ unlabeled 10× labeled + non-obvious correlations.
• Audio (wav2vec 2.0, HuBERT) is the big win outside text/vision: speech's temporal structure maps onto masked prediction — 10 min labeled → competitive WER; separately, the 100-hr labeled subset matches SOTA with 100× less data.
• wav2vec quantizer gives discrete symmetry-breaking targets (like BERT's vocab) to prevent collapse.
• Graph edge-drop is domain-dependent: valid for social graphs, breaks molecular identity (bonds).
• Design question always: what shortcut would a lazy model exploit, and how do you eliminate it?

Check your understanding