When you train a small neural network on a small table of arithmetic, something strange happens. First it memorizes the training pairs and fails completely on the rest. If you keep training with weight decay, long after training loss has hit zero, test accuracy suddenly jumps from near random to 100%. Power et al. named this delayed generalization grokking [1].
The demo above trains a real one-layer transformer from scratch in a Web Worker on your CPU, solving \((a + b) \bmod 59\) from only 35% of the \(59^2 = 3{,}481\) pairs. Every gradient, weight update, Fourier transform, and frequency knockout runs live in this tab.
The clock algorithm
How does a one-layer network with no arithmetic primitives learn modular addition? Nanda et al. [2] reverse-engineered the trained weights and found a trigonometric circuit (Gromov [4] gave a closed-form analytic version). Instead of storing a lookup table, the network picks a handful of key frequencies \(k \in \{1, \dots, 29\}\) (with \(\omega_k = 2\pi k / 59\)) and maps each input residue \(a\) onto circles in its embedding matrix \(W_E\):
$$a \longmapsto \big(\cos(\omega_k a),\ \sin(\omega_k a)\big)$$Attention reads both \(a\) and \(b\) into the residual stream at the = token. Then the ReLU MLP mixes them to approximate the angle-addition identities:
Finally, the unembedding matrix \(W_U\) takes the dot product with \((\cos(\omega_k c), \sin(\omega_k c))\) for every candidate answer \(c \in \{0, \dots, 58\}\), producing a cosine wave \(\cos(\omega_k(a + b - c))\) that peaks at \(c \equiv a + b \pmod{59}\).
Because \(p = 59\) is prime, a single wave \(\cos(\omega_k(s - c))\) has a unique maximum at \(c = s\), but its closest runner-up is \(\cos(2\pi / 59) \approx 0.994\) — less than 0.6% lower — so any noise flips the prediction. Summing four or five different frequencies pushes their near-peaks onto different wrong residues, where they cancel out and open a wide margin on the true answer. You can see this in the wave panel above: before grokking the other lane carries most of the logits, and after grokking it flattens while the top five waves align on \(c^*\). Once the model has grokked, zeros out the other 24 frequencies in \(W_E\) and test accuracy holds near 100%; removes only those 5 frequencies and test accuracy drops to chance (\(1/59 \approx 1.7\%\)).
Three phases, and hidden progress
Watching only test accuracy makes grokking look sudden. Switch the top chart to to see the circuit form underneath (numbers below are for the default seed 1; click New seed to see how timing varies across runs) [2]:
- Memorization (epochs 0–150): Train accuracy hits 100% while test accuracy stays near 2.6% and test loss climbs past 13. The network fits the 1,218 training pairs with high-norm weights spread across all 29 frequencies.
- Circuit formation (epochs 150–1,000): Test accuracy climbs only slowly (2.6% → 14.5%) while test loss remains above 13, yet the restricted test loss — from projecting logits onto only the top 5 Fourier frequencies — drops steadily from 3.3 to 0.6. Weight norm \(\|W\|_2\) peaks around epoch 800 (37.7).
- Cleanup and generalization (epochs 1,000–2,000): Weight decay shrinks the high-norm memorizing weights faster than the compact trigonometric circuit. Total weight norm falls from 37.7 down to 25.3, test accuracy climbs to 100% by epoch 1,800, and excluded train loss (the loss with those 5 frequencies subtracted out) peaks above 9 around epoch 1,400 before settling near 5–6 as the lookup table is pruned away.
Drag horizontally across the training chart at any time to scrub back to an earlier checkpoint and watch the clock circle untangle from a random cloud into a 59-gon.
What makes grokking happen
Grokking comes from the competition between two solutions: a high-norm lookup table that gradient descent finds quickly, and a low-norm Fourier algorithm that takes longer to organize. Two knobs control which one wins: enough training data to rule out shortcuts, and weight decay to penalize the lookup table [2], [3].
grok.wasm. With zero weight decay, the network memorizes and stays at 3.1% test accuracy while its weight norm grows to 51.7. With only 25% of pairs, weight decay cannot force generalization (2.9%). Starting at zero weight decay and switching it on at epoch 800 still triggers full grokking (99.5% test accuracy; 99.7% with only its top 5 frequencies).Try it live: run with and let it sit flat at 100% train / ~3% test. Then drag the weight decay slider up (or click ) mid-run without resetting to trigger generalization. You can also switch to subtraction (which uses a 40% train split, wd = 2.0, seed 42): without the \((a,b) \leftrightarrow (b,a)\) symmetry of addition, test accuracy stays below 1% until ~1,500 epochs and under 10% until ~2,500 before jumping to 100% near epoch 2,900.