Self-supervised Learning

Labels are the bottleneck in machine learning.

There is a trillion words of text on the internet and billions of photos.

But only a few million examples come with a human-written correct answer attached.

So the question becomes

Self-supervised learning (SSL) does exactly that. It manufactures supervision from the structure of the input itself, so you need zero human labels.

The trick in one line

Formally you define the target as a function of the input itself: y = g(x). So the training pair is (x, g(x)), and it is unlimited — every raw example becomes its own labeled example.

The invented objective is called a pretext task. Predicting a masked word, deciding whether two crops came from the same image, or guessing the next token are all pretext tasks.

Here is the idea beginners miss

So you solve the pretext, discard the objective (or keep a lightweight adapter), and reuse the encoder. That is the entire "pretrain then fine-tune / embed then retrieve" pattern.

Yann LeCun's "dark matter of machine learning" phrase captures the asymmetry: a trillion unlabeled tokens versus a few million human-curated ones.

Why does it work? Because solving a good pretext task forces the encoder to internalize statistics that downstream tasks also need — syntax, object parts, temporal regularities, provenance.

Three properties separate a good pretext task from a useless one:

1. It is hard enough to require semantics. Predicting an image's average pixel color learns nothing; predicting a masked span requires grammar and world regularities.
2. Its solution transfers. The skills needed to solve it must overlap the target tasks.
3. It is solvable from data alone at scale, without leaking the eventual evaluation target (test-set contamination).

One sentence with a hole in it:

"The cat sat on the ___."

The real answer, "mat", is a label you got for free — it was already sitting in the data. You just hid it.

Now the model sees millions of these holes across the whole internet:

• guess "mat" → reward (correct)
• guess "car" → penalty (wrong)

To guess well, the model quietly learns grammar, meaning, and which things go together. Nobody taught it those rules; they fell out of filling blanks.

Same free-label idea, contrastive version:

• Take one photo of a dog, crop it two ways → "same dog" (a positive pair)
• Two photos of different animals → "different" (negatives)

The positive/negative labels are free again — you made them just by how you cropped. In one line

code
Copy
MASKED  (hide-and-guess)            CONTRASTIVE  (same vs different)

  "the cat sat on the [M]"             dog photo ─┐
   guess:  m_a_t  ✓  |  car ✗         two crops ───┘ view1 · view2  → pull together  ●─●
   (hidden word IS the label)          cow photo  ●                    push apart    ●   ●

• Masked modeling: black out a piece, the original piece becomes the label. BERT does this to words; MAE does it to image patches.
• Contrastive learning: take two views of the SAME thing and pull their vectors together; push vectors of DIFFERENT things apart. The "same/different" tag is the free label.

Both start from completely unlabeled data. The difference is only how you fabricate the question.

Think of self-supervised learning as building your own crossword puzzle out of the internet.

You black out random squares and quiz yourself. Nobody is grading you — the blanks come from the puzzle itself, and so do the answers.

But to fill a square, you have to learn the language, the topics, and how words fit together.

That understanding is the encoder. You reuse it for translation, for search, or for a brand-new task where someone finally gives you just a handful of real labels. The crossword (the pretext task) gets thrown away; the learning stays.

Masked modeling (BERT, MAE). Randomly mask ~15% of tokens (in 80/10/10 splits: 80% replaced with the [MASK] token, 10% random token, 10% unchanged so the model cannot assume the input is wrong) and predict them from context. Masked language modeling yields a bidirectional representation — the same reason it dominates classification/embedding work where full context matters. In vision, Masked Autoencoders (MAE) mask ~75% of image patches and reconstruct pixels with a shallow decoder; the extreme masking rate works because images are highly redundant.

Autoregressive modeling (GPT family). Predict the next token given the prefix. Causal masking means each position only sees its left context, which is exactly what makes generation and in-context learning natural. Scaling law findings (Kaplan et al., arXiv:2001.08361) show loss falls predictably with model size, data, and compute — the empirical basis for foundation-model scaling.

Contrastive learning. Learn a space where augmented views of the same sample (positives) are close and unrelated samples (negatives) are far. SimCLR (Chen et al., 2020) showed that heavy augmentation — crop, color jitter, grayscale, Gaussian blur — plus a projection head and InfoNCE is enough: on ImageNet it reached about 76.5% top-1 linear-probe accuracy, comparable to the fully supervised ResNet-50 baseline of the time, using zero labels. MoCo addressed the "need many negatives" problem with a momentum-updated queue of previous embeddings; the explicit-negative requirement disappeared entirely with BYOL and DINO, which learn from two views alone.

Multimodal alignment (CLIP). Contrast text against images across a large batch of (image, caption) pairs scraped from the web. The payoff is zero-shot classification: because both modalities share an embedding space, a new class needs only its name.

Non-text/vision uses at classical-ML scale: predicting the next value or masked window of a sensor stream, log-line masking, order-verification on time series, and co-occurrence prediction on user-event sequences.

import torch, torch.nn.functional as F

def nt_xent(zi: torch.Tensor, zj: torch.Tensor, temperature: float = 0.1):
    """zi, zj: L2-normalized embeddings of two augmented views of each sample."""
    z = torch.cat([zi, zj], dim=0)                 # (2N, d)
    sim = z @ z.t() / temperature                  # cosine similarities, scaled
    n = z.size(0)
    eye = torch.eye(n, dtype=torch.bool, device=z.device)
    sim.fill_diagonal_(-1e9)                       # exclude a view matching itself
    positives = torch.cat([sim[n // 2:, :n // 2].diagonal(),
                           sim[:n // 2, n // 2:].diagonal()])
    log_prob = positives - torch.logsumexp(sim, dim=1)
    return -log_prob.mean()                        # pull positives, push all negatives

The conventional story is "SSL is better than supervised learning." The accurate story is that SSL changes what you pay for:

• Label cost collapses. You buy compute and data instead of annotations. A supervised model for a new classification task needs thousands of labeled examples; a self-supervised encoder plus a linear probe or a few dozen demonstrations often matches it.
• One encoder serves many tasks. Pretraining is amortized across retrieval, clustering, ranking, classification, and regression downstream — the pretrain/fine-tune pattern that BERT (arXiv:1810.04825) and GPT established. BERT alone set new state-of-the-art results on eleven NLP tasks, pushing the GLUE score to 80.5% (a 7.7-point absolute gain) and SQuAD v1.1 F1 to 93.2.
• Cold start becomes tractable. New products have unlabeled events but no labels yet, so embeddings from self-supervision bootstrap search, dedup, and recommendation on day one.
• Data valuation moves. With unlimited synthetic targets, the bottleneck is corpus quality and deduplication, not annotation volume.

The real trade-offs: pretraining compute and energy are enormous; the learned space encodes whatever the corpus encodes, including biases; and pretext objectives can be solved superficially — a model may learn augmentations or template artifacts instead of semantics, a risk that shows up as brittle transfer. Finally, evaluation is harder than accuracy: you must probe representation (linear probes, k-NN, retrieval recall@k) rather than the pretext loss itself, and a low pretext loss does not guarantee good downstream structure.

Self-supervision is not confined to deep networks; the pretext-task idea composes cleanly with the classical toolkit:

1. SSL + k-means. Cluster in the embedding space rather than raw features. This is the Deezer music-style recipe and the standard approach for logs, tickets, and product catalogs — the encoder supplies a metric, clustering supplies structure.
2. SSL + logistic regression. Train a probe on top of frozen embeddings with a few hundred labels. Frozen-encoder probing is fast, regularized by construction, and hard to overfit badly.
3. SSL as the ultimate imputation engine. Masked modeling generalizes to missing values in tabular and sensor data: hide a column, predict it, and treat that prediction as a learned imputer.
4. Contrastive pretraining then gradient boosting on tabular features. Use embedding similarity as engineered features (nearest-neighbor distance, cluster ID, cosine to exemplars) — a cheap, effective way to inject unlabeled structure into an XGBoost-style production model.

The decision rule: if you have lots of unlabeled data of the same modality you will serve, pretrain. If labels are the cheap part and data is scarce, classical supervised models remain the better buy — an assumption that holds far more often than the hype suggests.