Dissection · Kingma & Ba, 2015

Anatomy of Adam

Kingma and Ba's optimiser, dissected: every line of Algorithm 1 drawn as real blocks on a 151-parameter network, checked against PyTorch, with a research desk that reruns Reddi et al.'s counterexample, finds how rare a large gradient must be to defeat it, and tests the paper's claims about bias correction, the size of the step and ε.

PaperKingma & Ba, ICLR 2015

Model151 parameters, live

Reading35 min · 25 steps

Published2026-10-18

01Overview

An optimiser, drawn as the machine it is

In December 2014 Kingma and Ba proposed "an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments" [1]. It is named for those moments: "the name Adam is derived from adaptive moment estimation." It became the default optimiser of much of deep learning, and the base of AdamW.

What you are looking at is Algorithm 1 of the paper, run on a small network. The network has 151 parameters (a 2-10-10-1 network that separates two interleaved spirals), laid out as three bands of cells: the first layer, the second layer and the output weights. Each line of the algorithm is one station of three such bands, from the gradient at the bottom, through the two moving averages on the left and right, to the step at the top. A cell sits at the same place in every station, so one cell, followed up the page, is one parameter's whole story: its gradient, its two averages, their corrected versions, and the step it takes.

The page is showing step of a training run, with . Type another step into the bar above, or add settings such as 20 b2=0.98, and every block is recomputed by the same code that the tests check against PyTorch. The visual grammar, tensors as blocks of cells, follows Brendan Bycroft's LLM Visualization [2].

How to use this page

Scroll the article, or use ← →. Drag the stage to orbit, right-drag to pan, scroll on it to zoom; O shows the whole machine, C returns to the step's view, R replays its animation. Hover any cell for its value and the arithmetic behind it. The step number takes t, a= (α), b1=, b2=, eps= and bc=0 (bias correction off).

Go deeper: Optimisers: SGD, Adam, AdamW, Lion

02Overview

Five promises in one paragraph

The introduction lists what the method offers: "the magnitudes of parameter updates are invariant to rescaling of the gradient, its stepsizes are approximately bounded by the stepsize hyperparameter, it does not require a stationary objective, it works with sparse gradients, and it naturally performs a form of step size annealing." It was "designed to combine the advantages" of AdaGrad [3], which suits sparse gradients, and RMSProp [4], which suits on-line and non-stationary settings. And it came with a theorem: "We show Adam has O(√T) regret bound".

This article takes the promises one at a time and checks them. The invariance and the bound on the step are tested on the stage (steps 12 and 13). The regret theorem, whose proof was later found to be wrong, and the sparse-gradient claim are the subject of the research desk at the end, together with the paper's bias-correction experiment redone at miniature scale and two claims it made in passing: that the step stays within α, and what happens when ε dominates.

The settings are the paper's good defaults: α = 0.001, β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸, with one change here: the miniature uses α = 0.01 because its training run is short.

Go deeper: Optimisation theory · Hyperparameter search hygiene

03Baseline

What plain gradient descent does with these gradients

Before the machinery, the baseline. Stochastic gradient descent moves every parameter against its gradient, by the same multiple of it:

θ_{t} = θ_{t - 1} - α g_{t}

The block beside the gradient on the stage is that step for α = 0.1, drawn on the same colour scale as Adam's step further up. Look at its pattern: a few cells are tall and most are nearly flat. Across the 151 parameters at this step the gradient's size varies by a factor of between its 10th and 90th percentiles, and by between the smallest and the largest. A single α cannot suit both ends: raise it until the small ones move and the large ones overshoot; lower it until the large ones are safe and the small ones stand still.

This is the problem the adaptive methods address, and it is why the paper's first claim is about scale. If the step is a ratio of the gradient to its own recent size, it no longer depends on the units in which the gradient is measured. The gradient itself comes from backpropagation, dissected in the earlier piece on that algorithm.

Go deeper: Calculus and gradients · Backpropagation and automatic differentiation

04Algorithm 1

Where the gradient comes from: twenty points and a loss

The algorithm is stated for "random subsamples (minibatches) of datapoints": at step $t$ it sees a loss $f_{t}$ computed on part of the data. Here the 200 points are shuffled afresh every epoch and cut into minibatches of 20, so ten steps make an epoch. The blocks at the bottom of the stage are the minibatch of this step: its twenty points $x$ (labelled by their index in the data), the two tanh layers, and the output probability $p$ beside the error $p - y$ .

f_{t} (θ) = \frac{1}{20} n \in B_{t} \sum [lo g (1 + e^{z_{n}}) - y_{n} z_{n}]

Here $z_{n}$ is the network's output (a logit) for point $n$ and $y_{n}$ its class. On this minibatch the loss is . Over all 200 points the network at the start of this step has a loss of ; a loss near 0.69 is chance, so a network that is still there has not yet found the spirals.

The loss on one minibatch is a noisy stand-in for the loss on everything. That noise is what the second moment will measure.

Go deeper: Cross-entropy loss

05Algorithm 1

The gradient is a noisy estimate of the one you want

Line 4 of Algorithm 1 is $g_{t} \leftarrow \nabla_{θ} f_{t} (θ_{t - 1})$ : the gradient of the minibatch loss at the parameters the previous step left. The stage draws it as three bands, one cell per parameter, blue where the loss would fall if the parameter increased and amber where it would rise.

It is an estimate. The gradient of the loss over all 200 points, which no step ever sees, points a little elsewhere: the cosine between the two at this step is . The tall cells are where the signal is strong, the flat ones where it is weak, and from step to step each cell flips and wanders. Two statistics of that wandering are what Adam keeps for every parameter, "estimates of the 1st moment (the mean) and the 2nd raw moment (the uncentered variance) of the gradient."

The largest gradient at this step: .

Go deeper: Estimators, bias, variance and consistency

06Algorithm 1

m: an average of recent gradients

The first moment is an exponential moving average of the gradient. Each step keeps a fraction $β_{1}$ of the old average and adds a fraction $1 - β_{1}$ of the new gradient:

m_{t} = β_{1} m_{t - 1} + (1 - β_{1}) g_{t} = (1 - β_{1}) i = 1 \sum t β_{1}^{t - i} g_{i}

The right-hand form says what it is: a weighted sum of every gradient so far, with weights that fall by a factor $β_{1}$ per step of age. With $β_{1} = 0.9$ a gradient's weight falls to about a third every ten steps, so $m$ is a mean over roughly the last ten minibatches. That averages away much of the minibatch noise; it is what momentum does.

It is momentum in the usual sense. PyTorch's SGD with momentum $μ$ keeps buf = μ·buf + g [5], which is $m_{t} / (1 - β_{1})$ with $β_{1} = μ$ : the same weights, scaled so that a steady gradient $g$ builds up to $g / (1 - μ)$ . test.mjs checks that limit.

On the stage, the white cell is the largest gradient; follow it from the block on the left ( $m_{t - 1}$ ), through the gradient, to the block above it ( $m_{t}$ ).

Go deeper: Exponential smoothing · Optimisers: SGD, Adam, AdamW, Lion

07Algorithm 1

v: how large the gradients have been, coordinate by coordinate

The second moment is the same average applied to the square of each gradient, element by element:

v_{t} = β_{2} v_{t - 1} + (1 - β_{2}) g_{t}^{2}

Its square root is the typical size of that parameter's gradient, whether positive or negative, over roughly the last $1/ (1 - β_{2})$ steps: a thousand steps at the default $β_{2} = 0.999$ . The stage draws $v$ , in the same units as $g$ , as bars. The Transformer, whose original recipe is the most cited use of Adam, set β₁ = 0.9, β₂ = 0.98 and ε = 10⁻⁹ [6], a memory of only fifty steps.

The paper notes that $\overset{v}{^}_{t}$ "is an approximation to the diagonal of the Fisher information matrix", so Adam's division by $v$ is a cheap, diagonal relative of natural-gradient preconditioning, with the square root making it more conservative than the full thing.

Follow the white cell: . At the first step $v_{t - 1}$ is zero, and with the default $β_{2} = 0.999$ the new $v_{t}$ is a thousandth of the squared gradient, which is the problem of the next step.

Go deeper: Optimisation theory

08Algorithm 1

Starting from zero leaves the averages short

Both averages start at zero, and a zero is not a gradient. After one step, with the defaults, $m$ is a tenth of the gradient and $v$ a thousandth of its square. The paper says so: "these moving averages are initialized as (vectors of) 0's, leading to moment estimates that are biased towards zero, especially during the initial timesteps, and especially when the decay rates are small (i.e. the βs are close to 1)." The remedy follows from writing $v_{t}$ as a sum over the gradients seen and taking expectations:

E [v_{t}] = E [g_{t}^{2}] (1 - β_{2}^{t}) + ζ

where $ζ = 0$ if the true second moment does not change. The factor $1 - β_{2}^{t}$ is the total weight of the $t$ gradients seen; the rest of the weight is on the initial zero. "In algorithm 1 we therefore divide by this term to correct the initialization bias."

The two strips on the stage show the factors for this run's $β_{1}$ and $β_{2}$ , for steps from 1 to 5,000 on a log axis, with your step marked. At step they are and . Without correction, the first step would be $(1 - β_{1}) / 1 - β_{2}$ times α: for these settings times.

Go deeper: Estimators, bias, variance and consistency

09Algorithm 1

Dividing by what is missing

The corrected averages divide each by the weight it has actually accumulated:

\overset{m}{^}_{t} = \frac{m _{t}}{1 - β _{1}^{t}}, \overset{v}{^}_{t} = \frac{v _{t}}{1 - β _{2}^{t}}

At this step . Hover the white cell on either block to see its arithmetic. The correction is exact in the one case it was designed for: if every gradient is the same number $g$ , then $\overset{m}{^}_{t} = g$ and $\overset{v}{^}_{t} = g^{2}$ at every $t$ , however small $t$ is. test.mjs checks that identity for four settings of $(β_{1}, β_{2})$ over fifty steps.

The correction fades. After a few multiples of $1/ (1 - β)$ steps the factor is within a few per cent of 1 and the corrected and uncorrected averages agree: the second factor needs about three thousand steps at $β_{2} = 0.999$ , and about 150 at 0.98. Type 20 bc=0 in the bar above to switch the correction off and see the stage without it.

Go deeper: Numerical computation gotchas

10Algorithm 1

The step is a ratio, so its size does not depend on the gradient's

The last line of the loop divides one corrected average by the square root of the other:

θ_{t} = θ_{t - 1} - α \frac{m ^ _{t}}{v ^ _{t} + ϵ}

The numerator is the direction and strength of the recent gradient; the denominator is its recent size. Their ratio is a number near $\pm 1$ when the gradient has been steady and near 0 when it has been mostly noise. The constant ε keeps the division finite where $\overset{v}{^}$ is zero.

Look at the two callouts on the step block: . What sets the size of a step is not the size of the gradient but how consistently it has pointed one way. A coordinate whose gradient keeps its sign has $∣ \overset{m}{^} ∣ \approx \overset{v}{^}$ and moves by about α; one whose gradient flips from step to step averages to a small $\overset{m}{^}$ and moves little, however large the individual gradients are. At step 1 there is no history and every cell moves by exactly α. Across all 151 cells at this step, moved by at most α, the median step is α and the largest is α. The next two steps put numbers on that.

The paper also gives a cheaper ordering of the same arithmetic: fold the two corrections into the step size, $α_{t} = α 1 - β_{2}^{t} / (1 - β_{1}^{t})$ , and update with $m_{t} / (v_{t} + \overset{ϵ}{^})$ .

Go deeper: Optimisers: SGD, Adam, AdamW, Lion

11Algorithm 1

Subtract the step

The parameters move by $Δ_{t}$ , and the loop starts again. The stage shows the parameters before, the step, and the parameters after; the white cell is the largest gradient's parameter, . Adam keeps three numbers for each parameter, $θ$ , $m$ and $v$ : for these 151 parameters that is 453 numbers, which is the paper's "little memory requirements".

One step on this minibatch changes the loss on all 200 points from to : it . A single stochastic step need not lower the full-data loss, and early on it hardly moves it; what matters is the direction of many steps, and the speed at which the step falls from its early size to the small, noisy size of late training, which the research desk measures (E3).

Go deeper: Learning-rate schedules

12Why the step looks like that

Is the step bounded by α?

Section 2.1 argues that the step has a size the user controls. With ε = 0 the step is $Δ_{t} = α \overset{m}{^}_{t} / \overset{v}{^}_{t}$ , and the paper states two bounds: "|Δt| ≤ α · (1 − β1)/√(1 − β2) in the case (1 − β1) > √(1 − β2), and |Δt| ≤ α otherwise." The first case "only happens in the most severe case of sparsity: when a gradient has been zero at all timesteps except at the current timestep." It argues that in more common cases the ratio is close to $\pm 1$ , so that $∣ Δ_{t} ∣ ≲ α$ , which "can be understood as establishing a trust region around the current parameter value". The ratio is called the signal-to-noise ratio, and "the SNR value typically becomes closer to 0 towards an optimum, leading to smaller effective steps".

The stage highlights, in white, any cell whose step exceeds α. At this step are within α; the largest is α, on the .

The stated bound is not a bound

The bound is the exact worst case only for the single-spike sequence the paper describes. Over all gradient sequences, the largest possible ratio can be written down by the Cauchy–Schwarz inequality: with weights $a_{i}$ on $g_{i}$ in $\overset{m}{^}_{t}$ and $b_{i}$ on $g_{i}^{2}$ in $\overset{v}{^}_{t}$ , the supremum squared is $\sum_{i} a_{i}^{2} / b_{i}$ , which is

g sup \frac{∣ m ^ _{t} ∣}{v ^ _{t}} = \frac{1 - β _{1}}{1 - β _{1}^{t}} \frac{1 - β _{2}^{t}}{1 - β _{2}} k = 0 \sum t - 1 (\frac{β _{1}^{2}}{β _{2}})^{k} t \to \infty \frac{1 - β _{1}}{( 1 - β _{2} ) ( 1 - β _{1}^{2} / β _{2} )}

For the defaults this is α in the limit, against the paper's α, and for the Transformer's β₁ = 0.9, β₂ = 0.98 it is 1.70 α where the paper says α. test.mjs runs Adam on the sequence that attains it (gradients growing by a factor $β_{2} / β_{1}$ per step) and gets the closed form to nine digits, and finds no random sequence above it. The supremum stays bounded as $t$ grows exactly when $β_{1}^{2} < β_{2}$ , the inequality on which Theorem 2 of Reddi et al. turns (β₁ < √β₂). When β₁ = 0 the paper's bound is exact. The practical claim, $∣ Δ_{t} ∣ ≲ α$ , is a statement about typical sequences, and E3 measures it on a real run.

Go deeper: Gradient clipping · Loss spikes and divergence

13Why the step looks like that

Multiply every gradient by c and nothing changes

The abstract claims that the method "is invariant to diagonal rescaling of the gradients", and section 2.1 gives the one-line reason: "The effective stepsize ∆t is also invariant to the scale of the gradients". Scaling the gradients by $c$ scales $\overset{m}{^}$ by $c$ and $\overset{v}{^}$ by $c^{2}$ , so

\frac{c m ^ _{t}}{c ^{2} v ^ _{t}} = \frac{m ^ _{t}}{v ^ _{t}}

The same holds for any diagonal rescaling, a different $c_{i}$ for each coordinate, because every operation in the loop is element-wise. On the stage, the gradient block swells to a thousand times its size and back while the step below it does not move. It is computed, not drawn: the engine reruns the whole history with the loss multiplied by 1,000 and by 0.001, and at this step the largest difference between the steps, with ε = 0, is α.

The invariance is exact only for ε = 0. With ε = 10⁻⁸, a gradient so small that $\overset{v}{^}$ is comparable to ε is no longer divided by its own size, and the step shrinks. The figure shows it live: over 300 steps the invariance is intact down to a loss scale of about 10⁻⁴, visibly broken by 10⁻⁵, and gone by 10⁻⁶ (research E4 measures where, and why).

Go deeper: Mixed-precision training · Numerical computation gotchas

14Theory

A convergence proof, and the quantity it needed

Section 4 analyses Adam in the online learning framework of Zinkevich [7]: "Since the nature of the sequence is unknown in advance, we evaluate our algorithm using the regret":

R (T) = t = 1 \sum T [f_{t} (θ_{t}) - f_{t} (θ^{*})]

the loss paid along the way, minus the loss of the best fixed point in hindsight. The theorem bounds it by a multiple of $T$ , comparable to the best known bound for this general convex online learning problem, so the average regret $R (T) / T$ tends to zero (Corollary 4.2). Two details of the statement matter. It is proved "when the learning rate αt is decaying at a rate of t^−1/2 and first moment running average coefficient β1,t decay exponentially", not for the constant settings of Algorithm 1; and only for convex losses, where the paper says outright that "our convergence analysis does not apply to non-convex problems".

In 2018 Reddi, Kale and Kumar showed that the proof "erroneously assumes that Γ_t is positive semi-definite and is hence, incorrect" [8]. $Γ_{t}$ is the change in the inverse learning rate:

Γ_{t + 1} = \frac{v _{t + 1}}{α _{t + 1}} - \frac{v _{t}}{α _{t}}

The proof telescopes a sum of such terms and needs each to be non-negative: "This quantity essentially measures the change in the inverse of learning rate of the adaptive method with respect to time." For SGD and AdaGrad the inverse learning rate never falls. For Adam it does, whenever a large gradient ages out of the moving average. The stage shows $Γ_{t}$ for Reddi et al.'s own counterexample ( $C = 3$ ): blue cells in the Adam row are steps where Adam's learning rate rises; the AMSGrad row, which keeps a running maximum of $v$ , has none. The rows above are the gradients and the iterates, which the research desk will examine.

Other problems in the proof

The arXiv source of Reddi et al. keeps, in a block of LaTeX comments in its appendix, a list of further problems it found. One reads "A simple counterexample for this lemma is if g_{t,i} = 1/sqrt{t}.", about Lemma 10.3 of the Adam paper, which says "Let gt = ∇ft(θt) and g1:t be defined as above and bounded, ‖gt‖2 ≤ G, ‖gt‖∞ ≤ G∞. Then," that $\sum_{t} g_{t, i}^{2} / t \leq 2 G_{\infty} ∥ g_{1 : T, i} ∥_{2}$ . With $g_{t} = 1/ t$ the left side is the harmonic number $H_{T}$ and the right side is $2 H_{T}$ , and test.mjs finds that the lemma first fails at $T = 31$ and is off by a factor of 1.9 at $T = 1 0^{6}$ . Another comment reads "The proof of Lemma 10.4 is erroneous." These comments are not in the compiled paper, which keeps only the $Γ_{t}$ argument, so they are reported here as what the source contains and not as published claims.

Go deeper: Online learning and regret bounds

15Related work

Adam's family, on one surface

Section 5 places Adam among its relatives, and two of its statements can be run. First, AdaGrad: "AdaGrad corresponds to a version of Adam with β1 = 0, infinitesimal (1 − β2) and a replacement of α by an annealed version αt = α · t^−1/2". That is because as $β_{2} \to 1$ the corrected second moment tends to the plain average of squares, $\overset{v}{^}_{t} \to t^{- 1} \sum_{i} g_{i}^{2}$ . test.mjs runs Adam with $β_{1} = 0$ , $1 - β_{2} = 2^{- 30}$ and $α / t$ beside AdaGrad on the same gradients: with the correction they agree to 10⁻¹⁰ in the parameters; without it, as the paper says, "the direct correspondence … does not hold", and after 200 steps the parameters are thousands apart.

Second, RMSProp. The paper separates it from Adam: "RMSProp with momentum generates its parameter updates using a momentum on the rescaled gradient, whereas Adam updates are directly estimated using a running average of first and second moment of the gradient." The paper adds that RMSProp lacks a bias-correction term, which matters most when β₂ is close to 1, and PyTorch's RMSprop does the momentum as described, with buf = μ·buf + g/(√v + ε) [5]. Both the AdaGrad and RMSProp algorithms are taken from the Adam paper's account of them; the lecture that introduced RMSProp [4] and AdaGrad's own paper [3] were not read for this page.

On the stage is Beale's function, a long curved valley with a flat shelf in front of it, and the paths of five optimisers from (−2, −2). The step sizes were chosen once, not tuned, so the picture shows how each moves and is not a ranking. After 1,500 steps their values of the function are: (the minimum is 0, at (3, 0.5)). The figure lets you move the start; from some starts an optimiser ends on a flat shelf near (−1.5, 1.4) or leaves the plotted square, which is where the step-size choice shows.

Go deeper: Optimisers: SGD, Adam, AdamW, Lion

16Extensions

Take the norm to infinity

Adam divides by an $L_{2}$ -style size of recent gradients. Section 7.1 generalises to $L_{p}$ and lets $p \to \infty$ , where "conveniently enough, we don't need to correct for initialization bias". The infinity norm of the decayed history is a maximum:

u_{t} = max (β_{2} u_{t - 1}, ∣ g_{t} ∣) = max (β_{2}^{t - 1} ∣ g_{1} ∣, \dots, β_{2} ∣ g_{t - 1} ∣, ∣ g_{t} ∣), Δ_{t} = \frac{α}{1 - β _{1}^{t}} \frac{m _{t}}{u _{t}}

The step is bounded exactly: "|∆t| ≤ α", with defaults α = 0.002, β₁ = 0.9, β₂ = 0.999. test.mjs checks the unrolled form against the recursion for forty steps, that the $L_{p}$ norm approaches the maximum as $p$ grows, and the bound over two hundred random gradients with an occasional large one.

The stage shows AdaMax's $u$ and the step AdaMax would take, computed from the same gradients as the Adam blocks beside them, so the two denominators can be compared cell by cell. At this step, with the white cell's $u$ = against Adam's $\overset{v}{^}$ = : .

17Extensions

Average the iterates, and correct that too

Section 7.2 adds a second use of the same idea. The last iterate is noisy, so "an exponential moving average over the parameters can be used, giving higher weight to more recent parameter values." It takes one line, and it starts from zero like the other averages, so it needs the same correction:

\overset{ˉ}{θ}_{t} = β_{2} \overset{ˉ}{θ}_{t - 1} + (1 - β_{2}) θ_{t}, \hat{θ}_{t} = \frac{θ ˉ _{t}}{1 - β _{2}^{t}}

The paper writes the averaging decay as $β_{2}$ ; nothing requires it to equal the decay of $v$ . The block on the right of the stage is $\hat{θ}_{t}$ for this run; the root-mean-square difference between it and the current parameters is . test.mjs checks the corrected average against its closed form as a weighted sum of all the iterates.

18Afterlife

What later work did to Adam

Most of what has happened to Adam since 2015 is a reaction to one of the claims above.

The proof. Reddi et al. showed that its convergence claim fails and proposed AMSGrad, which maintains the maximum of all v_t until the present time step, with a regret guarantee [8]. Défossez et al. then proved that Adam itself converges at the rate of AdaGrad when β₂ goes to 1 and α to 0, and noted that the counterexamples use a small exponential decay parameter β₂ < 1/5 [9]. Zhang et al. gave the reason: Reddi et al. pick the problem after picking the hyperparameters, while practical applications often fix the problem first and then tune them [10]. Research E1b asks how large "large enough" β₂ is.
Weight decay. Adding an L2 penalty to the loss and decaying the weights are the same for SGD but not for adaptive gradient algorithms, such as Adam, so AdamW decouples them [11].
Warm-up. Adam is usually run with a learning-rate warm-up. The Transformer recipe used 4,000 warm-up steps with $β_{2} = 0.98$ [6]. Liu et al. argue the cause is that the adaptive learning rate's variance is problematically large in the early stage, which the bias correction does not remove [12].
Generalisation. Wilson et al. found that solutions found by adaptive methods generalize worse than SGD, even when these solutions have better training performance [13]. Choi et al. showed that the comparison depends on the tuning protocol: as tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate, and they found that the popular adaptive gradient methods never underperform momentum or gradient descent when the search space is large enough [14].

The first of these is tested on the research desk below; the second to fourth are cited here and not tested, because they are about scales and objectives this page's miniature cannot reach.

Go deeper: Weight decay in pretraining · Warm-up and why it helps · Learning-rate schedules · The Transformer · Multi-seed reporting

19Experiments

What the paper tried, and what fits on this page

Section 6 compares Adam with SGD with Nesterov momentum, AdaGrad, RMSProp, AdaDelta and the sum-of-functions optimiser on L2-regularised multi-class logistic regression on MNIST, on a text classification problem with sparse features, on a network with two hidden layers of 1000 ReLU units, on a convolutional network with three conv-and-pool stages and a layer of 1000 ReLUs, and, for the bias-correction test, on a variational auto-encoder [15]. Its conclusion: "Overall, we found Adam to be robust and well-suited to a wide range of non-convex optimization problems in the field machine learning."

None of those models can run in a browser tab, and the comparisons depend on how the settings were chosen: "searched over a dense grid and the results are reported using the best hyper-parameter setting." This page does the part that fits. The figure below trains the 151-parameter network with any of seven optimisers at any step size; try SGD at its best step size against Adam at its best, and Adam at α = 0.1 against 0.001. The research desk then reports every cell of a grid, not the best.

Go deeper: Hyperparameter search hygiene · Multi-seed reporting and run-to-run variance

20Research desk

Reproducing the counterexample, and the paper's own experiment

The research desk tests claims the 2015 paper made, or that its critics made about it. Hypotheses and verdict rules were written down before any of the runs, in research/README.md, and committed before they ran; the result paragraphs on this page are generated from the result files by the same rules, so they cannot be edited to fit. Every run uses this page's engine, which test.mjs checks against PyTorch's optimisers. The first question is whether Reddi et al.'s counterexample does what it says.

Claim

"There is an online convex optimization problem where Adam has non-zero average regret": a sequence of linear functions on the interval from −1 to 1,

C x

at every third step and

- x

at the others with

C > 2

, and β₁ = 0 and β₂ = 1/(1 + C²). The mean gradient is

(C - 2) /3 > 0

, so

x = - 1

is optimal. "The large gradient C is unable to counteract this effect since it is scaled down by a factor of almost C for the given value of β2."

Hypothesis

(i) for every

C

from 2.5 to 100, Adam's iterate stays above 0, is exactly at the upper wall at every step

t \equiv 1 (mod 3)

as the proof says, and its average regret is within 5% of

2 (C - 2) /3

; (ii) the bias-corrected and ε = 10⁻⁸ variants keep at least half of that regret; (iii) AMSGrad, SGD and AdaGrad end below 5% of it. A confirmation: it is a theorem, and the pilot had already shown it.

Design

C \in {2.5, 3, 5, 10, 100}

, 30,000 steps,

α = 0.9 1 - β_{2}

with

α_{t} = α / t

, six arms, deterministic.

The second test is the paper's own experiment: a gradient of 1010 once in 101 steps and −10 otherwise, with β₁ = 0.9 and β₂ = 0.99, where "its iterates x_t converge to x = 1, which unfortunately has the largest regret amongst all points in the domain", and a stochastic version, 1010x with probability 0.01. The step size and the starting point are not stated.

Claim

Adam's iterate goes to +1 and AMSGrad's to the optimum −1, in the online and the stochastic versions.

Hypothesis

In at least 80% of cells Adam ends above 0, and AMSGrad ends below 0 in at least 3 of the 10 (α, schedule) cells for each starting point.

Design

α \in {0.1, 0.03, 0.01, 0.003, 0.001}

, constant or

α / t

x_{1} \in {0, 1}

, 300,000 steps; the stochastic version with 5 seeds per cell, judged by the median.

Details in the source

The main text states the theorem for C > 2 and the appendix proves it for C ≥ 2. At $C = 2$ the gradients sum to zero over each period, so every point is optimal and no regret is incurred; the main text's strict inequality is the right one (test.mjs checks both the zero sum and the zero regret). The analysis drops the debiasing step, noting that its arguments also apply to the debiased version; the run above includes that variant and confirms it. It also claims that for any constant ε > 0 there is a setting where Adam has non-zero average regret; the run covers one value, ε = 10⁻⁸, which changes nothing. And Reddi et al.'s AMSGrad has no bias correction: PyTorch's amsgrad=True [5] does apply it, so the two differ in the first steps, which test.mjs keeps apart as amsgrad and amsgrad-torch.

21Research desk

How far apart can the large gradients be before Adam goes the wrong way?

Reddi et al. choose the problem after the hyperparameters; a practitioner fixes the problem and then chooses β₂. Défossez et al. note that the counterexamples use a β₂ below 1/5 [9], and Zhang et al. prove that for large enough β₂ Adam converges [10]. Neither says how large is large enough for a gradient that is rare. The experiment asks it directly.

Claim

Adam converges when β₂ goes to 1; "the optimization algorithm must have long-term memory of past gradients". How much memory is enough is not stated.

Hypothesis

Take a gradient of −1 on every step except one in

P

, where it is

S = 1.1 P - 1

, so the mean is +0.1 and a good optimiser drifts left. Adam's steady-state drift is right or wrong according to

P

; the boundary

P^{*}

scales with Adam's memory

1/ (1 - β_{2})

, so

κ = (1 - β_{2}) P^{*}

varies by less than a factor of 3 over

β_{2} \in {0.9, 0.98, 0.99, 0.999, 0.9999}

Design

β_{1} = 0.9

(and 0, the theorem's setting), correction on, 51 values of

P

from 2 to 251,189, the drift measured over five periods after a burn-in of

max (10 P, 20/ (1 - β_{2}))

steps. Supported if the largest

κ

is at most 3 times the smallest; partly if at most 10. Secondary arms: AMSGrad never fails; a stochastic version fails within a factor of 3 of the periodic one.

The law has a reading in steps. A gradient spike sets $v$ , and $v$ forgets it over about $1/ (1 - β_{2})$ steps; during that forgetting the steps grow, and the many small gradients of the wrong sign add up. If the next spike arrives before they have outgrown it, Adam goes the right way; if it arrives later, it does not. For the defaults that is a few hundred steps, for the Transformer's β₂ a few tens.

22Research desk

Figure 4 of the paper, at 151 parameters

Claim

"values β2 close to 1 indeed lead to instabilities in training when no bias correction term was present, especially at first few epochs of the training." "In summary, Adam performed equal or better than RMSProp, regardless of hyper-parameter setting." The paper's test: β₁ ∈ {0, 0.9}, β₂ ∈ {0.99, 0.999, 0.9999}, log₁₀α from −5 to −1 on a VAE.

Hypothesis

H2a: at β₁ = 0.9 and step 100, correction helps by more than 0.05 in loss at 3 of 7 step sizes at β₂ = 0.9999 and makes no difference (under 0.02) at 5 of 7 at β₂ = 0.98. H2b: after 1,000 steps the corrected run is no worse than the uncorrected one by more than 0.02 in 90% of the 56 cells. H2c: with the best α for each, the two differ by at most 0.02 in 7 of 8 (β₁, β₂) pairs. H2d: with correction, β₂ = 0.98 and 0.999 reach the same best loss to within 0.02.

Design

lo g_{10} α \in {- 4, \dots, - 1}

in half steps,

β_{1} \in {0, 0.9}

β_{2} \in {0.98, 0.99, 0.999, 0.9999}

(the paper's three plus the Transformer's), correction on or off, seeds 1 to 3: 336 runs, loss over all 200 points after 100 and 1,000 steps.

23Research desk

The size of the step, over a real run

Claim

The step is "approximately bounded by the stepsize setting α" and "the SNR value typically becomes closer to 0 towards an optimum, leading to smaller effective steps". Neither is tested in the paper. The stated bound is not a bound over all sequences (step 12).

Hypothesis

H3a: at least 95% of coordinate-steps have

∣Δ∣ \leq α

. H3b: none exceeds the paper's own bound for these settings, 3.16. H3c: the median over coordinates of

∣Δ∣/ α

, averaged over steps 2,001 to 3,000, is at most a third of its average over steps 1 to 10.

Design

The paper's β₁, β₂, ε with α = 0.01, 3,000 steps, seeds 1 to 3; at every step the ratio

∣ Δ_{t, i} ∣/ α

for each of the 151 coordinates (453,000 values per seed).

One of the paper's sentences is not borne out as a description of the typical step. It says that in more common scenarios the ratio is about $\pm 1$ ; The bound is a ceiling that is rarely approached, not the size of a typical step, and only the first few steps of training move the parameters by anything near α.

The stage shows this step's 151 ratios as the step block, with any cell above α in white. At step : are within α and the largest is α.

24Research desk

When ε stops being negligible

Claim

In the CNN experiment, "the second moment estimate v̂ vanishes to zeros after a few epochs and is dominated by the ε in algorithm 1." And Adam is invariant to the scale of the gradient (step 13), which is exact only when ε = 0.

Hypothesis

H4a: Adam's final loss changes by at most 0.02 from its value at

c = 1

for every

c \geq 1 0^{- 4}

. H4b: at

c = 1 0^{- 8}

it is at least 0.1 worse. H4c: at

c = 1

, fewer than 1% of coordinates have

\overset{v}{^} < 10 ϵ

at step 1,000, so the §6.3 observation is not reproduced at this scale.

Design

Multiply the loss by

c \in {1, 1 0^{- 1}, 1 0^{- 2}, 1 0^{- 3}, 1 0^{- 4}, 1 0^{- 6}, 1 0^{- 8}}

; Adam with the paper's defaults and α = 0.01, 1,000 steps, seeds 1 to 3; SGD at α = 0.1 for contrast.

A coordinate whose $\overset{v}{^}$ is far below ε has a step of $α \overset{m}{^} / ϵ$ : that is SGD with momentum, at the step size $α c / ϵ$ for the unscaled gradient. At $c = 1 0^{- 6}$ that is $0.01 \times 1 0^{- 6} /1 0^{- 8} = 1$ ; at $c = 1 0^{- 8}$ it is 0.01, a tenth of the step size given to SGD in this experiment, where This is the sense in which ε interpolates between Adam and SGD, and the sense in which, in the inclusion argument of Choi et al., Adam can approximate Momentum for large ε [14].

The stage shows the corrected second moment at the step in the bar above and the step it produces: at step , coordinates have a $\overset{v}{^}$ below $10 ϵ$ .

25Research desk

What this adds, and what to test next

In short: the algorithm runs as the paper states, to within PyTorch's arithmetic; its stated bound on the step is exact only for β₁ = 0, and the supremum is larger; its proof had the flaw Reddi et al. say it had, and the counterexample runs as stated at every C tested. The counterexample is a statement about the gap between large gradients relative to Adam's memory, with a constant near 0.3 at the default β₂. On a small network, bias correction mattered at the largest step sizes and in the other direction at smaller ones, and ε takes over when the gradient scale falls by about a million. All of it is at 151 parameters, on one task, with a handful of seeds, which is the scale at which every claim could be checked against every cell.

How often do real gradients spike, and how far apart? Measure the gaps between large-gradient minibatches in a real training run, per parameter, and compare them to the boundary $P^{*} \approx 0.3/ (1 - β_{2})$ for the spike sizes seen. That would say whether the counterexample's regime occurs in practice, the question left open by [9] and [10].
Why does a large early step help here? In E2, uncorrected runs at β₁ = 0 reached the spirals where corrected ones stalled. Test a warm-up schedule and the rectified step of Liu et al. on the same grid, to see whether the early kick is an initial-scale effect of this network or something general about early step size [12].
An exact law for the boundary. The continuum law is derived for β₂ near 1 and matches the measured brackets at 0.9999; it is looser at short memories. Derive the discrete version and its dependence on β₁.
How long does AMSGrad need? E1c's AMSGrad cells that did not reach −1 were still moving. Re-run to 10⁷ steps and report the time to cross against α, the one thing the original figure does not show.
The ε sweep at fixed gradient scale. Sweep ε over several orders of magnitude at the gradient scales where E4 saw it take over, and compare with the tuning protocol of Choi et al., who found that ε matters [14].

The engine, the PyTorch reference and its outputs, the experiments and their results are in this page's directory, anatomy/ai-ml/2026-10-18-adam/; the JavaScript that draws the stage is the code the tests check.

References

Kingma, D. P. and Ba, J. (2015). Adam: A Method for Stochastic Optimization. ICLR 2015; arXiv:1412.6980v9 The subject of this page. The arXiv source for v9 is a PDF, not LaTeX, so quotes come from its extracted text, checked against the page images where the extraction garbled mathematics.
Bycroft, B. (2023). LLM Visualization. bbycroft.net/llm The visual grammar of tensors as blocks of cells.
Duchi, J., Hazan, E. and Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research 12, 2121–2159 AdaGrad. Not read for this page; cited only for what Kingma & Ba and Reddi et al. say about it.
Tieleman, T. and Hinton, G. (2012). Lecture 6.5, RMSProp. COURSERA: Neural Networks for Machine Learning RMSProp, an unpublished lecture. Not read for this page; cited only for what Kingma & Ba say about it.
PyTorch contributors (2026). torch.optim (Adam, Adamax, SGD, RMSprop, Adagrad). PyTorch 2.14.1, torch/optim/adam.py as installed The independent implementation test.mjs checks the engine against. Its source, not its documentation page, was read.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł. and Polosukhin, I. (2017). Attention Is All You Need. NeurIPS 2017; arXiv:1706.03762 Its optimiser settings, from the arXiv LaTeX source.
Zinkevich, M. (2003). Online convex programming and generalized infinitesimal gradient ascent. ICML 2003 The online learning framework of the convergence analysis. Not read for this page.
Reddi, S. J., Kale, S. and Kumar, S. (2018). On the Convergence of Adam and Beyond. ICLR 2018; arXiv:1904.09237 The counterexample and the AMSGrad fix. Quotes from the arXiv LaTeX source.
Défossez, A., Bottou, L., Bach, F. and Usunier, N. (2022). A Simple Convergence Proof of Adam and Adagrad. Transactions on Machine Learning Research; arXiv:2003.02395 Adam converges, at the rate of AdaGrad, when β₂ is taken to 1 as the horizon grows. Quotes from the arXiv LaTeX source.
Zhang, Y., Chen, C., Shi, N., Sun, R. and Luo, Z.-Q. (2022). Adam Can Converge Without Any Modification On Update Rules. NeurIPS 2022; arXiv:2208.09632 Reddi et al. pick the problem after the hyperparameters; fix the problem first and Adam converges for large enough β₂. Abstract read from the arXiv LaTeX source.
Loshchilov, I. and Hutter, F. (2019). Decoupled Weight Decay Regularization. ICLR 2019; arXiv:1711.05101 AdamW. Abstract read from the arXiv LaTeX source.
Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J. and Han, J. (2020). On the Variance of the Adaptive Learning Rate and Beyond. ICLR 2020; arXiv:1908.03265 Warm-up as variance reduction for the adaptive step. Abstract read from the arXiv LaTeX source.
Wilson, A. C., Roelofs, R., Stern, M., Srebro, N. and Recht, B. (2017). The Marginal Value of Adaptive Gradient Methods in Machine Learning. NeurIPS 2017; arXiv:1705.08292 Abstract read from the arXiv LaTeX source.
Choi, D., Shallue, C. J., Nado, Z., Lee, J., Maddison, C. J. and Dahl, G. E. (2019). On Empirical Comparisons of Optimizers for Deep Learning. arXiv:1910.05446 Abstract read from the arXiv LaTeX source.
Kingma, D. P. and Welling, M. (2013). Auto-Encoding Variational Bayes. ICLR 2014; arXiv:1312.6114 The VAE of the bias-correction experiment. Not read for this page.

Fact ledger

Every number from the literature on this page, with the sentence it comes from. Hover any underlined number in the text to see its entry.

Value	What	Source
adaptive estimates of lower-order moments	what Adam is	[1] Abstract We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments.
adaptive moment estimation	the name	[1] §1 the name Adam is derived from adaptive moment estimation.
five advantages	the advantages listed	[1] §1 Some of Adam’s advantages are that the magnitudes of parameter updates are invariant to rescaling of the gradient, its stepsizes are approximately bounded by the stepsize hyperparameter, it does not require a stationary objective, it works with sparse gradients, and it naturally performs a form of step size annealing.
AdaGrad and RMSProp	the two parents	[1] §1 Our method is designed to combine the advantages of two recently popular methods: AdaGrad (Duchi et al., 2011), which works well with sparse gradients, and RMSProp (Tieleman & Hinton, 2012), which works well in on-line and non-stationary settings
O(√T) regret	the regret claim	[1] §4 We show Adam has O(√T) regret bound and a proof is given in the appendix.
α = 0.001, β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸	Algorithm 1's default settings	[1] Algorithm 1 caption Good default settings for the tested machine learning problems are α = 0.001, β1 = 0.9, β2 = 0.999 and ε = 10−8.
random minibatches	where the noise comes from	[1] §2 The stochasticity might come from the evaluation at random subsamples (minibatches) of datapoints, or arise from inherent function noise.
1st moment and 2nd raw moment	what m and v estimate	[1] §2 The moving averages themselves are estimates of the 1st moment (the mean) and the 2nd raw moment (the uncentered variance) of the gradient.
β₁ = 0.9, β₂ = 0.98, ε = 10⁻⁹	the Transformer's Adam settings	[6] §5.3 We used the Adam optimizer with beta_1=0.9, beta_2=0.98 and epsilon=10^{-9}.
an approximation to its diagonal	v̂ and the Fisher information	[1] §5 Like natural gradient descent (NGD) (Amari, 1998), Adam employs a preconditioner that adapts to the geometry of the data, since v̂t is an approximation to the diagonal of the Fisher information matrix (Pascanu & Bengio, 2013)
biased towards zero	the initialisation bias	[1] §2 However, these moving averages are initialized as (vectors of) 0’s, leading to moment estimates that are biased towards zero, especially during the initial timesteps, and especially when the decay rates are small (i.e. the βs are close to 1).
1 − β₂ᵗ	what the correction divides by	[1] §3 In algorithm 1 we therefore divide by this term to correct the initialization bias.
"the exponential decay rate β₁"	the derivation's aside	[1] §3 otherwise ζ can be kept small since the exponential decay rate β1 can (and should) be chosen such that the exponential moving average assigns small weights to gradients too far in the past.
"a small value of β₂"	the sparse-gradient remark	[1] §3, last paragraph In case of sparse gradients, for a reliable estimate of the second moment one needs to average over many gradients by chosing a small value of β2; however it is exactly this case of small β2 where a lack of initialisation bias correction would lead to initial steps that are much larger.
instabilities without the correction	what Figure 4 shows	[1] §6.4 In Figure 4, values β2 close to 1 indeed lead to instabilities in training when no bias correction term was present, especially at first few epochs of the training.
αₜ = α√(1−β₂ᵗ)/(1−β₁ᵗ)	the cheaper ordering	[1] §2 Note that the efficiency of algorithm 1 can, at the expense of clarity, be improved upon by changing the order of computation
invariant to diagonal rescaling of the gradients	the properties claimed	[1] Abstract The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters.
α(1−β₁)/√(1−β₂), or α	the two bounds on the effective step	[1] §2.1 The effective stepsize has two upper bounds: \|∆t\| ≤ α · (1 − β1)/√(1 − β2) in the case (1 − β1) > √(1 − β2), and \|∆t\| ≤ α otherwise.
the most severe sparsity	when the first bound applies	[1] §2.1 The first case only happens in the most severe case of sparsity: when a gradient has been zero at all timesteps except at the current timestep.
approximately bounded by α	the trust region	[1] §2.1 This can be understood as establishing a trust region around the current parameter value, beyond which the current gradient estimate does not provide sufficient information.
SNR = m̂/√v̂	the signal-to-noise ratio	[1] §2.1 With a slight abuse of terminology, we will call the ratio m̂t/√v̂t the signal-to-noise ratio (SNR).
a form of automatic annealing	automatic annealing	[1] §2.1 For example, the SNR value typically becomes closer to 0 towards an optimum, leading to smaller effective steps in parameter space: a form of automatic annealing.
β₁ < √β₂	Theorem 2's condition	[8] §3, Theorem 2 For any constant beta_1, beta_2 in [0,1) such that beta_1 < sqrt(beta_2), there is an online convex optimization problem where Adam has non-zero average regret
invariant to the scale of the gradients	scale invariance	[1] §2.1 The effective stepsize ∆t is also invariant to the scale of the gradients
the regret	how it is evaluated	[1] §4 Since the nature of the sequence is unknown in advance, we evaluate our algorithm using the regret
comparable to the best known	how it compares	[1] §4 Our result is comparable to the best known bound for this general convex online learning problem.
αₜ ∝ t^(−1/2), β₁,ₜ = β₁λ^(t−1)	the theorem's conditions	[1] §4 Our following theorem holds when the learning rate αt is decaying at a rate of t−1/2 and first moment running average coefficient β1,t decay exponentially with λ, that is typically close to 1, e.g. 1 − 10−8.
no convergence analysis	non-convex problems	[1] §6.2 Although our convergence analysis does not apply to non-convex problems, we empirically found that Adam often outperforms other methods in such cases.
Γₜ assumed positive semi-definite	the flaw in the original proof	[8] §4 The proof in the original paper of Adam erroneously assumes that Gamma_t is positive semi-definite and is hence, incorrect
Γₜ, the change in the inverse learning rate	the quantity that breaks the proof	[8] §3 This quantity essentially measures the change in the inverse of learning rate of the adaptive method with respect to time.
Lemma 10.3 is incorrect	a note left in the source	[8] appendix.tex, in a block of LaTeX comments A simple counterexample for this lemma is if g_{t,i} = 1/sqrt{t}.
Σ √(g²ₜ/t) ≤ 2G∞‖g₁:T‖₂	Lemma 10.3	[1] Appendix 10.1 Let gt = ∇ft(θt) and g1:t be defined as above and bounded, ‖gt‖2 ≤ G, ‖gt‖∞ ≤ G∞. Then,
Lemma 10.4 is erroneous	a note left in the source	[8] appendix.tex, in a block of LaTeX comments The proof of Lemma 10.4 is erroneous.
β₁ = 0, 1−β₂ → 0, αₜ = α t^(−1/2)	AdaGrad as a limit of Adam	[1] §5 AdaGrad corresponds to a version of Adam with β1 = 0, infinitesimal (1 − β2) and a replacement of α by an annealed version αt = α · t−1/2
no correspondence without bias correction	the limit needs the correction	[1] §5 Note that this direct correspondence between Adam and Adagrad does not hold when removing the bias-correction terms; without bias correction, like in RMSProp, a β2 infinitesimally close to 1 would lead to infinitely large bias, and infinitely large parameter updates.
momentum on the rescaled gradient	RMSProp with momentum	[1] §5 RMSProp with momentum generates its parameter updates using a momentum on the rescaled gradient, whereas Adam updates are directly estimated using a running average of first and second moment of the gradient.
no bias correction	RMSProp lacks the correction	[1] §5 RMSProp also lacks a bias-correction term; this matters most in case of a value of β2 close to 1 (required in case of sparse gradients), since in that case not correcting the bias leads to very large stepsizes and often divergence, as we also empirically demonstrate in section 6.4.
"a version of RMSProp with momentum"	Adam without the correction	[1] §6.4 Discussed in section 5, removal of the bias correction terms results in a version of RMSProp (Tieleman & Hinton, 2012) with momentum.
no initialisation bias to correct	AdaMax needs no correction	[1] §7.1 Note that, conveniently enough, we don’t need to correct for initialization bias in this case.
\|Δₜ\| ≤ α	AdaMax's bound	[1] §7.1 Also note that the magnitude of parameter updates has a simpler bound with AdaMax than Adam, namely: \|∆t\| ≤ α.
α = 0.002, β₁ = 0.9, β₂ = 0.999	AdaMax's defaults	[1] Algorithm 2 caption Good default settings for the tested machine learning problems are α = 0.002, β1 = 0.9 and β2 = 0.999.
spotted by a reader	an error in the AdaMax derivation	[1] §9 Thanks to Kai Fan from Duke University for spotting an error in the original AdaMax derivation.
an exponential moving average of the parameters	temporal averaging	[1] §7.2 Alternatively, an exponential moving average over the parameters can be used, giving higher weight to more recent parameter values.
the running maximum of v	AMSGrad	[8] §4 The key difference of AMSGrad with Adam is that it maintains the maximum of all v_t until the present time step and uses this maximum value for normalizing the running average of the gradient instead of v_t in Adam.
β₂ → 1 and α → 0, at AdaGrad's rate	when Adam converges	[9] §2 When beta_2 goes to 1 and alpha to 0, our results and previous work show that Adam does converge with the same rate as Adagrad.
β₂ < 1/5	the counterexamples' β₂	[9] §2, Related work because they uses a small exponential decay parameter β2 < 1/5.
problem picked after the hyperparameters	the mismatch	[10] Abstract pick the problem after picking the hyperparameters of Adam, i.e., (β1,β2); while practical applications often fix the problem first and then tune (β1,β2).
not equivalent under Adam	weight decay and L2	[11] Abstract L$_2$ regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is not the case for adaptive gradient algorithms, such as Adam.
4,000 steps	the Transformer's warm-up	[6] §5.3 We used warmup_steps=4000.
the adaptive rate's variance is large early	why warm-up	[12] Abstract we identify a problem of the adaptive learning rate -- its variance is problematically large in the early stage, and presume warmup works as a variance reduction technique.
worse than SGD, often significantly	generalisation	[13] Abstract We observe that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance.
Adam should never be worse than momentum	inclusion relationships	[14] Abstract As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum)
adaptive methods never underperform momentum	their finding	[14] Abstract In particular, we find that the popular adaptive gradient methods never underperform momentum or gradient descent.
L2-regularised, MNIST, minibatch 128	the logistic regression experiment	[1] §6.1 We evaluate our proposed method on L2-regularized multi-class logistic regression using the MNIST dataset.
2 × 1000 ReLU, minibatch 128	the multilayer network	[1] §6.2 a neural network model with two fully connected hidden layers with 1000 hidden units each and ReLU activation are used for this experiment with minibatch size of 128.
three conv-pool stages and 1000 ReLU	the convolutional network	[1] §6.3 Our CNN architecture has three alternating stages of 5x5 convolution filters and 3x3 max pooling with stride of 2 that are followed by a fully connected layer of 1000 rectified linear hidden units (ReLU’s).
a VAE, 500 softplus units, 50-d latent	the bias-correction experiment's model	[1] §6.4 We vary the β1 and β2 when training a variational auto-encoder (VAE) with the same architecture as in (Kingma & Welling, 2013) with a single hidden layer with 500 hidden units with softplus nonlinearities and a 50-dimensional spherical Gaussian latent variable.
robust and well-suited	the conclusion	[1] §8 Overall, we found Adam to be robust and well-suited to a wide range of non-convex optimization problems in the field machine learning.
a dense grid, best setting reported	how hyperparameters were chosen	[1] §6 The hyper-parameters, such as learning rate and momentum, are searched over a dense grid and the results are reported using the best hyper-parameter setting.
non-zero average regret	Theorem 1	[8] §3, Theorem 1 There is an online convex optimization problem where Adam has non-zero average regret i.e., R_T/T ↛ 0 as T → ∞.
f_t(x) = Cx if t mod 3 = 1, −x otherwise, C > 2	the counterexample	[8] §3 Consider the following simple sequence of linear functions for F = [-1,1]
β₁ = 0, β₂ = 1/(1+C²)	its settings	[8] §3 Suppose β1 = 0 and β2 = 1/(1 + C^2). We show that Adam converges to a highly suboptimal solution of x = +1 for this setting.
scaled down by a factor of almost C	why it fails	[8] §3 The large gradient C is unable to counteract this effect since it is scaled down by a factor of almost C for the given value of β2, and hence the algorithm converges to 1 rather than -1.
1010x once in 101 steps, −10x otherwise	the synthetic experiment	[8] §5, Synthetic Experiments 1010x, for t mod 101 = 1; -10x, otherwise
β₁ = 0.9, β₂ = 0.99	its settings	[8] §5 To enable fair comparison, we set beta_1 = 0.9 and beta_2 = 0.99 for Adam and AMSGrad algorithm, which are typically the parameters settings used for Adam in practice.
Adam's iterate goes to +1	its result	[8] §5 Furthermore, its iterates x_t converge to x = 1, which unfortunately has the largest regret amongst all points in the domain.
1010x with probability 0.01	the stochastic version	[8] §5 1010x, with probability 0.01; -10x, otherwise
C > 2	C in the main text	[8] §3 where C > 2.
C ≥ 2	C in the appendix	[8] Appendix, proof of Theorem 1 where C \geq 2.
the argument applies to the debiased version too	the analysis drops the debiasing step	[8] §2, footnote However, our arguments also apply to the debiased version as well.
no, for any constant ε > 0	does ε help?	[8] §3 However, we show that for any constant epsilon > 0, there exists an online optimization setting where, again, Adam has non-zero average regret asymptotically
long-term memory of past gradients	the remedy	[8] §1 The above result indicates that in order to have guaranteed convergence the optimization algorithm must have “long-term memory” of past gradients.
equal or better, regardless	the summary of §6.4	[1] §6.4 In summary, Adam performed equal or better than RMSProp, regardless of hyper-parameter setting.
β₁ ∈ {0, 0.9}, β₂ ∈ {0.99, 0.999, 0.9999}, log₁₀α from −5 to −1	the grid of Figure 4	[1] §6.4 We iterated over a broad range of hyper-parameter choices, i.e. β1 ∈ [0, 0.9] and β2 ∈ [0.99, 0.999, 0.9999], and log10(α) ∈ [−5, ..., −1].
v̂ is dominated by ε	the CNN observation about ε	[1] §6.3 We notice the second moment estimate vbt vanishes to zeros after a few epochs and is dominated by the in algorithm 1.
Adam approximates momentum for large ε	ε and momentum	[14] Appendix, inclusion relationships Adam can approximate Momentum for large ε

How this was made

Everything on the stage is computed by engine.js in float64 for the step and settings in the input bar: the gradient of the network by backpropagation, then Adam, Adam in the paper's cheaper ordering, AdaMax, AMSGrad, SGD with and without Nesterov momentum, RMSProp with momentum and AdaGrad. test.mjs (1,293 checks) compares the gradient with PyTorch 2.14.1 autograd (to 2e-16) and finite differences, every optimiser with torch.optim over 60 minibatch steps on three seeds (to 1e-12, moments to 1e-16), and the one-parameter problems of Reddi et al. with torch.optim.Adam; it proves the paper's identities (the unrolled averages, the bias identity, the first step with and without correction, scale invariance, AdaGrad as a limit of Adam, AdaMax's maximum form), derives and simulates the exact supremum of the step, runs Theorem 1 of Reddi et al. as an execution, and recomputes every derived number in the article. Quotes are copied from the arXiv sources (the Adam paper's arXiv source is a PDF, so its quotes come from the extracted text and were checked against the page images where the extraction garbled mathematics). The research desk (pre-registered before any run, committed to the branch first) is reproducible from research/.

Built with the dissection skill. The visual grammar of tensors as blocks of cells owes much to Brendan Bycroft's LLM Visualization. The optimisers are checked against PyTorch's torch.optim. Source, PyTorch reference, tests and experiments: anatomy/ai-ml/2026-10-18-adam/ in the site's repository.

Cite this page

@misc{sigmoid-adam,
  title  = {Anatomy of Adam},
  author = {Praveen T N},
  year   = {2026},
  note   = {Sigmoid Anatomy, interactive dissection},
  url    = {https://praveentn.live/anatomy/adam}
}