A dissection of Kingma and Ba (2015), Adam: A Method for Stochastic Optimization. A small network (2-10-10-1, tanh, 151 parameters) learns two interleaved spirals, and Algorithm 1 is laid out on a 3D stage as a V of stations, bottom to top: the minibatch and its gradient, the first moment on one side and the second moment on the other, each beside its value before the step, then their bias-corrected versions, the step, and the new parameters. Every station is the same 151 parameters in three bands, so one cell followed up the page is one parameter's whole story, and the step number and hyperparameters in the input bar recompute everything with the engine that test.mjs checks against PyTorch's optimisers. A 25-step article covers the paper section by section, with its numbers hover-linked to their sentences: the mean and uncentred variance, the initialisation bias, the bound on the step (whose stated form this page shows is exact only for beta1 = 0), scale invariance, the regret theorem and the flaw Reddi, Kale and Kumar found in its proof, AdaGrad and RMSProp as relatives, AdaMax and parameter averaging. A pre-registered research desk then reproduces the counterexample, measures how far apart rare large gradients can be before Adam drifts the wrong way as a function of beta2, rebuilds the paper's Figure 4 on bias correction at miniature scale, measures the size of the step over a real run, and finds where epsilon takes over.
The machine
A V of blocks on a table, read bottom to top like the pseudocode. Each station is the 151 parameters as three bands (first layer 3x10, second layer 11x10, output weights 1x11), so a cell has the same place in every station. At the bottom the minibatch and its forward pass (left), the gradient (centre) and the parameters before the step (right); above, the first moment m on the left and the second moment v (drawn as its square root) on the right, each beside its value before the step; above those, m and v after bias correction; at the top the step and the parameters after it. Beside the gradient is SGD's step on the same colour scale as Adam's, to the right the AdaMax and parameter-average blocks of section 7. Behind and beside are Beale's function with five optimisers' paths, Reddi et al.'s one-parameter problem, the phase diagram of research E1b and Figure 4 in miniature. Colour is the sign and height the size of each value.
What is real. Everything on the stage is computed by engine.js in float64 for the step and settings in the input bar: the gradient of the network by backpropagation, then Adam, Adam in the paper's cheaper ordering, AdaMax, AMSGrad, SGD with and without Nesterov momentum, RMSProp with momentum and AdaGrad. test.mjs (1,293 checks) compares the gradient with PyTorch 2.14.1 autograd (to 2e-16) and finite differences, every optimiser with torch.optim over 60 minibatch steps on three seeds (to 1e-12, moments to 1e-16), and the one-parameter problems of Reddi et al. with torch.optim.Adam; it proves the paper's identities (the unrolled averages, the bias identity, the first step with and without correction, scale invariance, AdaGrad as a limit of Adam, AdaMax's maximum form), derives and simulates the exact supremum of the step, runs Theorem 1 of Reddi et al. as an execution, and recomputes every derived number in the article. Quotes are copied from the arXiv sources (the Adam paper's arXiv source is a PDF, so its quotes come from the extracted text and were checked against the page images where the extraction garbled mathematics). The research desk (pre-registered before any run, committed to the branch first) is reproducible from research/.
Working model · 3D
AI & ML · Neural network foundations
