Loss Landscape Geometry and Mode Connectivity

The picture everyone carries of a non-convex loss surface is a mountain range: many isolated valleys, gradient descent falls into whichever one is nearest, and different random seeds land in genuinely different places. That picture is wrong in a specific, useful way, and correcting it changed what people believe is possible with trained weights.

The paths exist

Garipov et al. and, independently, Draxler et al. asked a direct question in 2018: take two networks trained from different random initialisations to comparable accuracy, and look for a curve in weight space connecting them along which loss stays low. They found such curves routinely, using simple parameterisations like a quadratic Bezier or a piecewise-linear path with one bend (Garipov et al., 2018, Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs, arXiv:1802.10026, NeurIPS 2018).

The finding is sharper than it sounds. The straight line between the two solutions has a high-loss barrier, which is why the mountain-range picture felt right. Allow a single bend and the barrier vanishes. The minima are not isolated points at the bottom of separate basins; they are points on a connected low-loss manifold, and the apparent isolation was an artefact of only ever looking along straight lines.

The immediate payoff was Fast Geometric Ensembling: walk along the discovered curve, collect checkpoints, and ensemble them, obtaining ensemble-quality predictions for roughly the training cost of one model.

Permutation symmetry explains most of the barrier

Why is the straight line bad if the endpoints are in one connected region? Because weight space over-counts solutions enormously. Permute the hidden units of a layer, permute the corresponding rows and columns of the neighbouring weight matrices, and the network computes exactly the same function. A layer of width \(n\) therefore has \(n!\) weight-space copies of every solution, and a deep network has the product of those factorials.

Entezari et al. conjectured that once you quotient out this symmetry, most SGD solutions are in a single basin: there exists a permutation of one network's units after which the straight line to the other has no loss barrier (Entezari et al., 2022, arXiv:2110.06296). Ainsworth, Hayase, and Srinivasa turned the conjecture into algorithms, matching units greedily by weight or activation similarity, and demonstrated linear mode connectivity between independently trained ResNets after alignment (Ainsworth et al., 2023, Git Re-Basin, arXiv:2209.04836, ICLR 2023).

The reframing matters. Independently trained networks are not different solutions in any interesting sense. They are relabellings of nearby points in one basin, and the barrier you measure between them is mostly a bookkeeping artefact.

Why this is not just geometry trivia

Three practical capabilities rest on it.

Weight averaging works. Model soups average the weights of many fine-tunes of a shared base model and beat picking the best one on held-out data. This is legitimate only because the fine-tunes stay in one basin, which they do because they share initialisation, so no permutation alignment is needed.

Model merging works, sometimes. Merging independently fine-tuned models requires either shared initialisation or explicit permutation alignment. Merges that ignore this produce garbage, and the reason is now precisely stateable rather than mysterious.

Linear mode connectivity is a training-stability diagnostic. Frankle et al. showed that networks branched from a shared checkpoint after a short "stability" phase are linearly connected to each other, and that the point at which this becomes true is also the point after which lottery-ticket pruning starts to work (Frankle et al., 2020, arXiv:1912.05671, ICML 2020). Linear connectivity became a measurable proxy for "the trajectory has committed to a basin".

When it breaks

The conjecture is stated, and mostly verified, for vision architectures of moderate depth. Transformers add symmetries beyond unit permutation, including head permutation within a layer and scale-shift redundancy between LayerNorm and the following linear map, so an alignment algorithm that only permutes units leaves barriers standing.

The alignment problem itself is hard: matching units optimally is a quadratic assignment problem, and the greedy algorithms that make it tractable find good local optima rather than the true permutation. A residual barrier after alignment therefore has two possible causes, a genuinely different basin or a failed match, and current methods cannot cleanly distinguish them.

Finally, low loss along a path does not imply the path is useful. Points midway along a connecting curve can have low training loss and noticeably worse calibration or robustness than either endpoint. Connectivity is a statement about one scalar. See optimisation theory and spectral analysis of weight matrices.