Learning Without Labels: The Collapse Problem at the Heart of Self-Supervised Learning

In June 2020, a DeepMind team published a self-supervised learning method that removed the component everyone believed was load-bearing. Contrastive learning worked by pushing apart embeddings of different images, and the standard explanation of why it worked was that without those repulsive terms the network would map every input to the same vector and the loss would go to zero. BYOL had no repulsive term at all. Its objective was simply to make one network's output predict another network's output on a different augmented view of the same image, a loss that is trivially minimised by a constant function. It reached 74.3% top-1 on ImageNet under linear evaluation with a ResNet-50, better than the contrastive methods it was compared against (Grill et al., 2020, Bootstrap Your Own Latent, arXiv:2006.07733).

The field's immediate response was to look for the hidden negatives. A widely-read reproduction argued that batch normalisation was smuggling in an implicit contrastive term by leaking statistics of other images in the batch into each embedding. That hypothesis was tested directly and failed: replacing batch norm with a batch-independent scheme of group normalisation plus weight standardisation still reached 73.9% top-1 against the original's 74.3%, with no cross-batch information anywhere in the network (Richemond et al., 2020, BYOL works even without batch statistics, arXiv:2010.10241). Whatever prevents collapse in BYOL, it is not disguised negatives.

Why this matters: Self-supervised pretraining is how every modern vision backbone, every text embedding model, and every multimodal encoder is built, and the entire family shares one degenerate solution: constant output. Understanding what actually blocks that solution is the difference between choosing a method by benchmark table and knowing which failure will show up in your embeddings six months into production.

TL;DR

• Joint-embedding self-supervised learning trains an encoder so that two augmented views of one input land close together. The objective's global minimum, in the absence of any counter-pressure, is a constant function, and every method in the field is a different counter-pressure.
• Contrastive methods block collapse with explicit negatives. InfoNCE decomposes asymptotically into an alignment term and a uniformity term on the hypersphere, and it is the uniformity term that does the blocking (Wang and Isola, 2020, arXiv:2005.10242).
• Negative-free methods block it with architectural asymmetry. SimSiam showed the essential ingredient is the stop-gradient, not the momentum encoder and not batch statistics (Chen and He, 2020, arXiv:2011.10566).
• Redundancy-reduction methods block it with an explicit penalty on the embedding covariance, which is the most transparent of the three and needs neither negatives nor asymmetry (Zbontar et al., 2021, arXiv:2103.03230; Bardes et al., 2021, arXiv:2105.04906).
• None of the three prevents dimensional collapse, where embeddings stay distinct but occupy a low-dimensional subspace of the space you paid for. It occurs in contrastive methods too, and the loss curve does not show it (Jing et al., 2021, arXiv:2110.09348).
• The mutual-information story that motivated InfoNCE does not explain its success: tighter MI estimators give worse representations, and MI is invariant to any invertible transformation of the features (Tschannen et al., 2019, arXiv:1907.13625).
• The augmentation family is the real inductive bias. The model learns to be invariant to whatever you randomise, which is why a recipe tuned for ImageNet object recognition can actively destroy the signal in a histology slide.

At a Glance

flowchart LR
  X["One input"] --> A1["Augmented view A"]
  X --> A2["Augmented view B"]
  A1 --> E1["Encoder"]
  A2 --> E2["Encoder or target"]
  E1 --> L["Pull embeddings together"]
  E2 --> L
  L --> C["Trivial solution: constant output"]
  L --> G["Anti-collapse pressure"]
  G --> R["Useful representation"]
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
  class X,A1,A2 blue
  class E1,E2,L purple
  class C rose
  class G,R emerald

The diagram is the whole subject. Everything downstream of "pull embeddings together" is a choice about what supplies the pressure on the right-hand branch, and each choice fails differently.

[IMAGE: Two-panel scatter plot of a 2D projection of embeddings during training. Left panel: healthy training, points spread over the circle with tight same-image pairs. Right panel: collapsed training, all points concentrated in one small region. Caption: "Same loss curve shape, opposite outcomes. Alignment without uniformity is collapse."]

Before Joint Embeddings: Why Labels Ran Out

Supervised pretraining on ImageNet was the field's default transfer recipe for most of the 2010s, and its ceiling was the label budget. ImageNet's 1.28 million labelled images took years of annotation effort, and the label itself is a lossy summary: "dog" discards pose, lighting, occlusion, and background, all of which the pixels contain and a downstream detection or segmentation task needs.

The first generation of alternatives invented pretext tasks whose labels came free from the data: predict the relative position of two patches, solve a jigsaw of shuffled tiles, predict the rotation applied to an image, colourise a grayscale photo. These worked, in the sense of beating random initialisation, and plateaued well below supervised transfer. The problem was that the network could often solve the pretext task with a shortcut that carried no semantic content, chromatic aberration for patch position being the famous case.

The reframing that broke the plateau came from contrastive predictive coding, which proposed a general recipe: instead of predicting a hand-designed label, predict representations of other parts of the same signal, and train with a probabilistic contrastive loss over a set of distractors drawn from elsewhere in the dataset (van den Oord et al., 2018, arXiv:1807.03748). The pretext task became "which of these candidates came from the same source as me," which has no obvious shortcut, and the loss became InfoNCE.

From there the progression was rapid and unusually legible, because each paper removed one ingredient the previous one had claimed was necessary.

timeline
    title Joint-embedding self-supervised learning
    2018 : Contrastive Predictive Coding introduces InfoNCE
         : Pretext task becomes predict-the-representation
    2019 : MoCo decouples the negative count from the batch with a queue and a momentum encoder
    2020 : SimCLR shows augmentation composition and a projection head matter more than architecture
         : BYOL removes negatives entirely and does not collapse
         : SimSiam removes the momentum encoder and shows stop-gradient is the essential part
    2021 : Barlow Twins and VICReg replace both with explicit covariance regularisation
         : DINO applies self-distillation to vision transformers, yielding emergent segmentation
    2023 : DINOv2 scales curated-data SSL into a general frozen backbone
         : I-JEPA predicts in representation space with no augmentations at all

[IMAGE: Annotated timeline strip pairing each year with a schematic of the method's architecture: CPC's autoregressive context, MoCo's queue, SimCLR's symmetric twin towers, BYOL's asymmetric online/target pair, VICReg's three loss terms, DINO's teacher-student with centering and sharpening.]

How Collapse Actually Works

The degenerate solution is not a bug in the loss, it is the loss's optimum

Write the naive objective. Given an encoder \(f_\theta\) and two augmentations \(t, t'\) of input \(x\), minimise the distance between the two embeddings:

Set \(f_\theta(\cdot) = c\) for any constant \(c\) and the loss is exactly zero, everywhere, for every input. This is not a local minimum you might stumble into; it is the global minimum and it is reachable by gradient descent from a random initialisation. Any method built on this objective is defined by what it adds to stop that from happening.

It is worth being precise about the two distinct failures, because they are routinely conflated:

Complete collapse. All embeddings converge to a single point. Trivial to detect: the variance of the embedding matrix goes to zero and downstream accuracy is chance.

Dimensional collapse. Embeddings remain distinct but span a subspace much smaller than the embedding dimension. A 2048-dimensional representation whose covariance has 30 non-negligible eigenvalues is a 30-dimensional representation carried in a 2048-dimensional container. The loss looks healthy, the variance is non-zero, and downstream accuracy is capped for reasons nothing in the training run reports.

Negatives supply uniformity

InfoNCE is the contrastive answer. For an anchor embedding \(z_i\), its positive \(z_j\), and \(N-1\) negatives:

The clean way to see why this blocks collapse is the asymptotic decomposition into two properties on the unit hypersphere: alignment, meaning positive pairs are close, and uniformity, meaning the embedding distribution spreads out over the sphere. Optimising the contrastive loss optimises both, and directly optimising the two metrics as separate terms produces representations at least as good as the contrastive loss itself (Wang and Isola, 2020, arXiv:2005.10242). Uniformity is the anti-collapse term, and under complete collapse it is maximally violated, so the loss is maximally penalised. Collapse is not merely disfavoured; it is the worst point in the objective.

Temperature \(\tau\) controls how sharply the denominator concentrates on the most similar negatives. Small \(\tau\) makes the loss almost entirely about the single hardest negative, which sharpens local structure and increases the damage done by false negatives. Large \(\tau\) spreads the penalty and blurs fine distinctions. Values between 0.05 and 0.2 dominate published recipes, and the sensitivity is real rather than cosmetic.

Asymmetry supplies it without negatives

BYOL keeps two networks. The online network \(f_\theta\) has an extra prediction head \(q_\theta\); the target network \(f_\xi\) has the same architecture with no predictor, and its weights are an exponential moving average of the online weights, \(\xi \leftarrow \tau_{\text{ema}} \xi + (1 - \tau_{\text{ema}}) \theta\). The loss asks the online network's prediction to match the target's projection, and critically, no gradient flows into the target.

Three ingredients are present: the predictor head, the stop-gradient, and the EMA. SimSiam removed the EMA entirely, setting the target network to be the same network with a stop-gradient, and still avoided collapse. Its ablations point at the stop-gradient as the necessary element (Chen and He, 2020, arXiv:2011.10566). The intuition offered there is an alternating-optimisation view: the stop-gradient makes the objective resemble an expectation-maximisation procedure in which the target acts as a slowly-moving set of cluster assignments, and the predictor's job is to approximate the expectation over augmentations. Collapse is prevented because the online network is always chasing a target it cannot instantaneously become.

This remains the least settled part of the field. The mechanism is empirically robust and theoretically contested, and the batch-statistics episode is the cautionary tale: a plausible, widely-repeated explanation was tested and eliminated within months.

Explicit regularisation makes the pressure visible

The third family gives up on implicit mechanisms and writes the anti-collapse term down. Barlow Twins computes the cross-correlation matrix \(\mathcal{C}\) between the embeddings of the two views, averaged over the batch, and pushes it toward the identity:

The diagonal term is invariance: each embedding component should agree across the two views. The off-diagonal term is redundancy reduction: different components should be decorrelated. Complete collapse makes every component identical, so all off-diagonal entries go to one and the penalty is large. It reports 73.2% ImageNet top-1 under linear evaluation with a ResNet-50.

VICReg splits the same idea into three named terms: a variance hinge that forces each embedding dimension's standard deviation across the batch above a threshold, an invariance term matching the two views, and a covariance term decorrelating dimensions. Because the variance term is per-dimension and applied within each branch, VICReg needs neither shared weights between branches nor batch-level negatives, and it also reports 73.2% top-1. The practical appeal is diagnostic: the variance term's value tells you directly how close the representation is to collapsing.

Dimensional collapse survives all of it

Here is the result that should change how you monitor these runs. Dimensional collapse was first discussed as a pathology of negative-free methods, on the reasoning that negatives enforce spread. It happens in contrastive methods too, and the analysis identifies two distinct causes: along feature directions where the augmentation variance exceeds the data variance, the weights collapse, and even when augmentation covariance is smaller than data variance in every direction, collapse still occurs through implicit regularisation from the interplay of weight matrices across layers (Jing et al., 2021, arXiv:2110.09348).

The consequence for practice is that neither the loss curve nor a linear probe reliably reveals it. The diagnostic is the singular-value spectrum of the embedding matrix, and it costs one SVD on a few thousand cached embeddings.

[IMAGE: Log-scale plot of sorted singular values of a 2048-dimensional embedding covariance for three runs: a healthy encoder with a slowly decaying spectrum, a dimensionally collapsed encoder whose spectrum falls off a cliff around index 40, and a fully collapsed encoder with one non-zero value. Caption: "One SVD on cached embeddings separates three outcomes the loss curve reports identically."]

Seeing It in Motion

The three families in one picture, organised by what supplies the pressure rather than by publication date:

flowchart TB
  subgraph Contrastive
    C1["Anchor and positive"] --> C2["InfoNCE over batch"]
    C3["Negatives from batch or queue"] --> C2
    C2 --> C4["Alignment plus uniformity"]
  end
  subgraph Distillation
    D1["Online net with predictor"] --> D2["Match target output"]
    D3["EMA target, stop-gradient"] --> D2
    D2 --> D4["Moving target blocks constant solution"]
  end
  subgraph Regularisation
    R1["Two views, same encoder"] --> R2["Covariance of embeddings"]
    R2 --> R3["Variance floor plus decorrelation"]
  end
  C4 --> OUT["Usable representation"]
  D4 --> OUT
  R3 --> OUT
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
  class C1,C3,C2,C4 blue
  class D1,D2,D3,D4 purple
  class R1,R2,R3 teal
  class OUT emerald

And the BYOL update as a sequence, because the ordering of the stop-gradient and the EMA is exactly what people get wrong when reimplementing it:

sequenceDiagram
  participant B as Batch
  participant O as Online network
  participant P as Predictor
  participant T as Target network
  participant U as Optimiser
  B->>O: view A
  B->>T: view B
  O->>P: projection
  T-->>P: target projection (stop-gradient)
  P->>U: prediction loss
  U->>O: gradient step on online weights only
  U->>T: EMA copy of online weights
  Note over T: Target never receives a gradient; it only ever lags the online net

[IMAGE: Line chart with three curves over training steps: embedding variance, off-diagonal correlation magnitude, and linear-probe accuracy, for a healthy run and a collapsing run overlaid. Caption: "Variance falls before accuracy does. Watching only the loss tells you nothing."]

Watch It Run

By the Numbers

Sources: figures are each paper's own headline number under its own protocol, from SimCLR, BYOL, Richemond et al., Barlow Twins, VICReg and DINO. They are not matched for backbone, epoch count, augmentation recipe, or projection-head width, and differences of one to two points across rows should not be read as method quality. SimCLR's 76.5% is its widened-ResNet result, which is why it sits above BYOL's ResNet-50 number; DINO's 80.1% uses a vision transformer rather than a ResNet. The comparison worth taking from the table is the third and fourth columns, not the fifth.

[IMAGE: Grouped bar chart of the table's accuracy column with backbone type encoded by bar pattern and a prominent annotation that bars are not protocol-matched. Caption: "Every number here is from a different experimental protocol. Read the mechanism column instead."]

A Concrete Example

Take a batch of two images, each with two augmented views, so four embeddings, all L2-normalised. Cosine similarities from the anchor \(z_1\) (view A of image 1):

• positive \(z_1'\) (view B of image 1): 0.85
• negative \(z_2\) (view A of image 2): 0.30
• negative \(z_2'\) (view B of image 2): 0.25

Step 1, temperature 0.1. Divide each similarity by \(\tau\) and exponentiate: \(e^{8.5} = 4914.8\), \(e^{3.0} = 20.1\), \(e^{2.5} = 12.2\). The denominator is \(4947.1\), the positive's share is \(4914.8 / 4947.1 = 0.9935\), and the loss is \(-\ln(0.9935) = 0.0065\). Nearly zero: at this temperature the model is already confident and gradients are small.

Step 2, temperature 0.5, same embeddings. Now \(e^{1.7} = 5.474\), \(e^{0.6} = 1.822\), \(e^{0.5} = 1.649\), denominator \(8.945\), positive share \(0.612\), loss \(0.491\). The same representation is scored seventy-five times worse. Temperature is not a cosmetic knob; it sets what counts as "already solved," and therefore where the remaining gradient goes.

Step 3, what collapse would score. Suppose the encoder has collapsed and all four embeddings are identical, so every similarity is 1.0. Every exponentiated term is equal, the positive's share is \(1/3\) over the three candidates, and the loss is \(\ln 3 = 1.099\) regardless of temperature. Compare with 0.0065 for the healthy case: InfoNCE assigns collapse the worst achievable score, which is exactly why negatives prevent it.

Step 4, the same collapse under BYOL's objective. BYOL minimises the squared distance between the predictor's output and the target's projection. If both networks output the same constant vector, that distance is 0, the global minimum. The identical failure that InfoNCE punishes hardest is what the negative-free objective rewards most, and only the stop-gradient plus predictor asymmetry keeps optimisation away from it.

[IMAGE: Grouped bar chart of the InfoNCE loss for four cases computed in this worked example: healthy at temperature 0.1 (0.0065), healthy at temperature 0.5 (0.491), fully collapsed (1.099), and the same collapse scored under BYOL's squared-distance objective (0.0). Caption: "The failure InfoNCE punishes hardest is the one a negative-free objective rewards most."]

Step 5, dimensional collapse in the same run. Cache 10,000 embeddings from a 2048-dimensional projection head and take the SVD of the centred matrix. Suppose the top 40 singular values carry 99% of the total variance. Nothing above changes: the InfoNCE loss is still 0.0065 on this batch, the embedding variance is non-zero, and the linear probe may still score respectably because a linear classifier only needs a few dozen useful directions. But a downstream task that needs fine-grained distinctions is being served a 40-dimensional code, and you would never learn this from the training curve.

Where It Breaks

False negatives are wrong by construction

Two distinct photographs of golden retrievers are treated as a negative pair, and the loss actively pushes them apart. With a batch of 4096 drawn uniformly from a balanced 1000-class dataset, roughly four negatives per anchor share the anchor's class, and the gradient is confidently incorrect on all of them. Larger batches improve the uniformity estimate and increase the absolute number of false negatives at the same time. Supervised contrastive variants fix this with labels, which defeats the purpose; nearest-neighbour and clustering-based variants approximate the fix without labels and introduce their own errors.

[IMAGE: Batch grid of 4096 slots showing which in-batch negatives share the anchor's class under a balanced 1000-class dataset, with the roughly four expected same-class negatives highlighted per anchor and the gradient direction annotated as incorrect for those. Caption: "Larger batches improve the uniformity estimate and add false negatives at the same rate."]

The augmentation family is the specification

The encoder becomes invariant to whatever augmentation randomises. Colour jitter is essential on ImageNet because without it the network solves the pretext task from colour histograms, and it is actively harmful for a bird-species model where plumage colour is the label, or for histopathology where stain intensity is diagnostic. Random resized cropping teaches scale and translation invariance, which a detector must then unlearn. There is no augmentation policy that is correct in general, and transplanting the ImageNet recipe to satellite imagery or medical scans without re-deriving it is the most common practical failure in this field.

Linear probing flatters the representation

Linear evaluation freezes the encoder and fits a linear classifier on top. It is cheap, standard, and measures one narrow property: whether classes are linearly separable in the embedding. It is insensitive to the spatial precision a detection head needs, to whether the representation encodes attributes beyond the probe's label set, and to dimensional collapse as long as enough directions survive for the classifier. A method that wins on linear probing can lose on fine-tuning and on k-NN retrieval, and papers report all three for exactly this reason.

Compute is the hidden entry fee

The published recipes assume hundreds of epochs over a million-image dataset with large batches; SimCLR's own results improve substantially with batch size and training length. Reproducing these results is a multi-GPU-week exercise, and a truncated schedule changes the ranking of methods. Several of the negative-free methods were specifically motivated by working at small batch sizes, which is the relevant axis if you are pretraining on your own data rather than on ImageNet.

The theory that motivated the loss does not explain it

InfoNCE was derived as a lower bound on mutual information, and that derivation is what gave the field its initial confidence. The bound is real; the explanation is not. Tighter MI estimators yield worse downstream representations, and MI is invariant under any invertible transformation, so it cannot distinguish a linearly separable embedding from an entangled one that carries the same information (Tschannen et al., 2019, arXiv:1907.13625). This matters practically: if you are choosing a method because it "maximises mutual information more tightly," you are optimising a quantity that has been measured not to predict the outcome you want.

Alternative Designs

The masked and JEPA rows deserve a note, because they sidestep the collapse problem rather than solving it. I-JEPA predicts representations of target blocks from a context block within the same image, with no view augmentations at all, and reports pretraining a ViT-H/14 in under 1200 GPU-hours (Assran et al., 2023, arXiv:2301.08243). Prediction is asymmetric by construction, so the constant solution is less accessible, and the invariances are learned from the data's own structure rather than specified by an augmentation policy.

How It Is Used in Practice

The dominant production pattern is a frozen backbone. DINOv2 trains on a large automatically curated image dataset and is used without fine-tuning across classification, depth estimation, and segmentation, with a small head trained per task (Oquab et al., 2023, arXiv:2304.07193). This is the shape self-supervised learning delivers value in: one expensive pretraining run amortised across many downstream tasks, none of which can afford their own labelled dataset.

CLIP occupies the same slot for anything that needs to be addressed in language, and its training objective is InfoNCE with the caption as the positive and other captions in the batch as negatives (Radford et al., 2021, arXiv:2103.00020). The consequences are the ones this article predicts. False negatives appear as near-duplicate captions in a batch. The augmentation policy is replaced by the caption distribution, so the model's invariances are whatever web alt-text happens to ignore. And the embedding geometry inherits the same anisotropy problems, which is why practitioners who use CLIP embeddings for retrieval routinely centre and whiten them first.

For domain-specific work the calculation is different. Self-supervised pretraining on a few hundred thousand unlabelled in-domain images and then fine-tuning on a few thousand labelled ones is frequently better than fine-tuning a general backbone, particularly when the domain's statistics are far from web photographs. The engineering work is not the loss function; it is deriving an augmentation policy that preserves the diagnostic signal, and validating that it did.

Insights Worth Remembering

1. The trivial solution is the objective's global minimum, not a local trap. Any joint-embedding method that does not name its anti-collapse mechanism has one you have not identified, and you will meet it when you change the batch size, the normalisation layer, or the predictor width.

2. Contrastive learning's negatives are a uniformity device, not a semantic one. Once the loss is understood as alignment plus uniformity, the negatives stop looking like "examples of what this is not" and start looking like a spreading force. That reframing predicts, correctly, that other spreading forces work just as well.

3. Removing components is how this field made progress. BYOL removed negatives, SimSiam removed the momentum encoder, Barlow Twins removed the asymmetry, I-JEPA removed the augmentations. Each removal falsified a claim of necessity that had been widely believed.

4. Dimensional collapse is the failure you will actually ship. Complete collapse is caught in an afternoon. A representation quietly living in 40 of 2048 dimensions passes every check you are running and caps every downstream task. Log the singular-value spectrum.

5. The augmentation policy is the model's specification of what does not matter. Choosing it is a domain-modelling decision on the same level as choosing the architecture, and it does not transfer between domains.

6. A theory that motivated a method is not evidence that the method works for that reason. The InfoNCE mutual-information story and the BYOL batch-statistics story were both plausible, widely repeated, and shown not to hold. Treat mechanism claims in this area as hypotheses with a short half-life.

7. Benchmark tables in this field are not protocol-matched. Backbone, epochs, batch size, projection width, and augmentation recipe all move ImageNet linear probe accuracy by more than the differences between methods. Compare mechanisms and compute budgets, and re-run the two candidates you care about on your own data.

Open Questions

Why exactly does the stop-gradient prevent collapse? The empirical finding is solid and reproduced. The alternating-optimisation account offered by SimSiam is suggestive rather than a proof, the batch-statistics explanation was tested and rejected, and there is still no account that predicts, ahead of time, which predictor architectures and EMA rates will collapse. This is an open theoretical problem, not a settled one with a folklore answer.

Is dimensional collapse ever benign? If a downstream task needs 40 directions, a 40-dimensional effective representation costs nothing. The measured fact is that effective rank is far below nominal dimension in many trained encoders; whether recovering the lost dimensions improves downstream tasks, and by how much, is not established across domains.

How much of self-supervised success is data curation rather than objective? DINOv2's own emphasis is on building a curated, diverse dataset rather than on the loss, which suggests the field's method-versus-method comparisons may be measuring something less important than the data pipeline. No controlled study isolates the two contributions at scale.

Do augmentation-free approaches transfer beyond images? I-JEPA's premise is attractive precisely where good augmentations are unknown, which is most non-image domains. The published evidence is largely in vision; whether the same argument holds for tabular, graph, or sensor data is untested.

What replaces linear probing? Everyone agrees the protocol is narrow and everyone still reports it, because it is cheap and comparable. A cheap evaluation that also captures spatial precision and effective dimensionality would change which methods look good, and none has been adopted.

Sources and Further Reading

1. van den Oord, A., Li, Y., & Vinyals, O. (2018). "Representation Learning with Contrastive Predictive Coding." arXiv:1807.03748
2. He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2019). "Momentum Contrast for Unsupervised Visual Representation Learning." arXiv:1911.05722
3. Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). "A Simple Framework for Contrastive Learning of Visual Representations." ICML 2020. arXiv:2002.05709
4. Grill, J.-B., et al. (2020). "Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning." NeurIPS 2020. arXiv:2006.07733
5. Richemond, P. H., Grill, J.-B., et al. (2020). "BYOL works even without batch statistics." arXiv:2010.10241
6. Chen, X., & He, K. (2020). "Exploring Simple Siamese Representation Learning." CVPR 2021. arXiv:2011.10566
7. Wang, T., & Isola, P. (2020). "Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere." ICML 2020. arXiv:2005.10242
8. Zbontar, J., Jing, L., Misra, I., LeCun, Y., & Deny, S. (2021). "Barlow Twins: Self-Supervised Learning via Redundancy Reduction." ICML 2021. arXiv:2103.03230
9. Bardes, A., Ponce, J., & LeCun, Y. (2021). "VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning." ICLR 2022. arXiv:2105.04906
10. Caron, M., et al. (2021). "Emerging Properties in Self-Supervised Vision Transformers." ICCV 2021. arXiv:2104.14294
11. Jing, L., Vincent, P., LeCun, Y., & Tian, Y. (2021). "Understanding Dimensional Collapse in Contrastive Self-supervised Learning." ICLR 2022. arXiv:2110.09348
12. Tschannen, M., Djolonga, J., Rubenstein, P. K., Gelly, S., & Lucic, M. (2019). "On Mutual Information Maximization for Representation Learning." ICLR 2020. arXiv:1907.13625
13. Assran, M., et al. (2023). "Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture." CVPR 2023. arXiv:2301.08243
14. Oquab, M., et al. (2023). "DINOv2: Learning Robust Visual Features without Supervision." arXiv:2304.07193
15. Radford, A., et al. (2021). "Learning Transferable Visual Models From Natural Language Supervision." ICML 2021. arXiv:2103.00020

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.