Self-Supervised Pretraining on Graphs

Molecular property labels come from wet-lab assays, so a typical dataset holds a few thousand labelled molecules while public repositories hold tens of millions of unlabelled structures. That ratio is exactly the one that made pretraining work for text and images, and the field spent several years porting the recipe to graphs. The outcome is more interesting than a success story: the objectives transferred, the augmentations did not, and the measured benefit on the headline application turned out to be smaller than the first papers reported.

The three objective families

Mutual-information and contrastive objectives build two views of the same graph and pull their representations together. Deep Graph Infomax contrasts node-level patch embeddings against a summary vector of the whole graph, with corrupted graphs as negatives (Veličković et al., 2019, Deep Graph Infomax, ICLR). GRACE instead builds two corrupted views by removing edges and masking features, then maximises agreement between the two copies of each node (Zhu et al., 2020, Deep Graph Contrastive Representation Learning). GraphCL systematises the view construction into four augmentation types and studies their combinations across domains (You et al., 2020, Graph Contrastive Learning with Augmentations, NeurIPS).

Bootstrapped objectives remove the negatives. Contrastive learning on graphs needs many negatives, and without a principled way to choose them the cost approaches quadratic in nodes. BGRL predicts one augmented view's representation from another through a target network with no negative sampling at all, reporting a 2 to 10 times reduction in memory and scaling to graphs with hundreds of millions of nodes (Thakoor et al., 2022, Large-Scale Representation Learning on Graphs via Bootstrapping, ICLR).

Predictive objectives mask part of the input and reconstruct it: attribute masking hides node or edge features, context prediction asks whether a subgraph belongs around a given node. The influential result here is that the level matters. Pretraining only at the node level or only at the graph level gives limited gains and can make downstream performance worse than no pretraining at all, while combining node-level and graph-level objectives gives up to 9.4 absolute points of ROC-AUC (Hu et al., 2020, Strategies for Pre-training Graph Neural Networks, ICLR).

Augmentation is where the analogy fails

Cropping a photograph of a cat leaves a photograph of a cat. Deleting an edge from a molecule produces a different molecule, often one that cannot exist. The standard graph augmentations, edge dropping, node dropping, feature masking and subgraph sampling, are all label-destroying operations on some graph types and harmless on others, which is why the effective augmentation set is domain-dependent rather than universal.

This is not a detail that tuning fixes. In image pretraining the augmentation encodes a true invariance of the data. On graphs it encodes an assumption that the task is robust to the specific perturbation, and when that assumption is wrong the objective teaches the encoder to ignore the signal the downstream task depends on. A chemistry-aware augmentation is a modelling decision, not a hyperparameter.

What the replication found

Systematic re-evaluation on molecular benchmarks reached three conclusions that should temper any plan built on this literature. The benefit of self-supervised pretraining on small molecular datasets can be negligible, and often lacks a statistically significant advantage over training from scratch. Supervised pretraining helps more, but the improvement shrinks as input features get richer and as the data splits get more balanced. And hyperparameters can affect downstream accuracy more than the choice of pretraining objective does (Sun et al., 2022, Does GNN Pretraining Help Molecular Representation?, NeurIPS).

The third finding is the one to act on. If a pretraining comparison does not tune the from-scratch baseline as hard as the pretrained model, it measures tuning effort, not transfer.

When it breaks

Transductive evaluation flatters self-supervised node embeddings. Pretraining on the full graph, including the test nodes' neighbourhoods, then fitting a linear probe on training labels, is a different problem from inductive deployment on nodes that did not exist at pretraining time. Both are legitimate; they are not comparable.

Linear-probe numbers do not predict fine-tuned numbers. Representations that separate classes linearly can still be worse starting points than random initialisation once the whole encoder is trained end to end on the target task.

Scale behaves differently than in language. More unlabelled graphs do not reliably buy more downstream accuracy, partly because graph corpora are narrower than they look: a hundred million molecules drawn from one chemical space cover less variation than their count suggests.

The baseline for a molecule is a fingerprint and a gradient-boosted tree. On many small molecular benchmarks that pair remains hard to beat, and it costs seconds to train. A pretrained graph network should be measured against it before anyone budgets for pretraining compute.