Self-Supervised Contrastive Learning

Labels are the expensive part of supervised learning, and for images they are also lossy: "dog" throws away everything about pose, lighting, and background that the pixels contain. Contrastive self-supervised learning replaces the label with a relation you can generate for free. Take one image, apply two random augmentations, and declare that the two resulting views should embed near each other and far from views of other images. Trained at scale, this recovers representations competitive with supervised pretraining on ImageNet linear evaluation (Chen et al., 2020, SimCLR, arXiv:2002.05709).

The InfoNCE objective

For an anchor view with embedding \(z_i\), its positive partner \(z_j\), and \(N-1\) negatives drawn from other images in the batch, the loss is a softmax cross-entropy over cosine similarities scaled by a temperature \(\tau\):

Read it as an \(N\)-way classification problem where the correct class is "the other view of me". The form comes from contrastive predictive coding, which framed it as a bound on mutual information between context and future (van den Oord et al., 2018, arXiv:1807.03748).

Two hyperparameters carry more weight than they look like they should. Temperature \(\tau\) controls how much the loss concentrates on the hardest negatives; values around 0.05 to 0.2 are typical and performance is genuinely sensitive to it. And the augmentation family is the inductive bias: SimCLR's ablations showed random cropping composed with colour distortion is what makes the task non-trivial, because without colour jitter the network can solve it from colour histograms alone.

Everything here is an anti-collapse device

The degenerate solution is obvious: map every image to the same vector and the positives are trivially close. Each family in this literature is a different answer to "what stops that?"

• Negatives. SimCLR uses in-batch negatives, which is why it needs batches in the thousands. MoCo decouples negative count from batch size with a queue of embeddings and a momentum-updated key encoder (He et al., 2019, arXiv:1911.05722).
• Asymmetry. BYOL removed negatives entirely, using an online network predicting the output of a slowly-updated target network, and still reached 74.3% ImageNet top-1 under linear evaluation with a ResNet-50 (Grill et al., 2020, arXiv:2006.07733). SimSiam stripped it further and showed the essential ingredient is the stop-gradient, not the momentum encoder (Chen and He, 2020, arXiv:2011.10566).
• Explicit regularisation. Barlow Twins pushes the cross-correlation matrix of two views' embeddings toward the identity (Zbontar et al., 2021, arXiv:2103.03230); VICReg names the three terms directly as variance, invariance, and covariance (Bardes et al., 2021, arXiv:2105.04906).

The mutual information story is not the reason it works

The InfoNCE bound is real, but it does not explain the empirical results. Tighter mutual-information estimators produce worse representations, and MI is invariant to any invertible transformation of the representation, so it cannot by itself prefer a linearly separable embedding over a scrambled one. The measured drivers are the architecture of the encoder and the parametrisation of the critic (Tschannen et al., 2019, On Mutual Information Maximization for Representation Learning, arXiv:1907.13625). Treat "it maximises mutual information" as a derivation of the loss form, not as an explanation of its success.

When it breaks

• Dimensional collapse. Even when representations do not collapse to a point, they can collapse to a low-dimensional subspace of the available embedding space, wasting most dimensions. It happens in contrastive methods too, not only in the negative-free ones, driven by augmentation variance exceeding data variance along some directions and by implicit regularisation across layers (Jing et al., 2021, arXiv:2110.09348). Inspect the singular-value spectrum of your embeddings; a long tail of near-zero singular values is the symptom.
• False negatives. Two different photographs of the same breed of dog are pushed apart because they are different images. At batch size 4096 on a 1000-class dataset, the expected number of same-class negatives per anchor is roughly four, and the loss is actively wrong about all of them.
• Augmentation invariance is task-specific. Colour jitter teaches colour invariance, which is exactly wrong for a bird-species classifier or a medical stain.
• Linear-probe accuracy hides the shortcomings. A frozen encoder can score well on linear evaluation and still lack the fine-grained localisation a detection head needs. Evaluate on the downstream task you actually have.