Grokking, Live

A one-layer transformer trains on modular arithmetic from random weights in your browser. It memorizes the training table in a second, sits flat on held-out pairs, then suddenly snaps to 100% — and you can see the trigonometric circuit forming underneath.

Task
Starting WASM trainer…
49,024 params · WASM SIMD

Training dynamics

Drag across the chart to inspect any past checkpoint without interrupting AdamW state.

Epoch
0
Train / Test
—
Weight norm
—
Top-5 Fourier
—
Restricted loss
—
Excluded loss
—

Embedding Fourier spectrum (WE)

Energy per frequency k = 1…29. Click any bar to mute or restore it.

The clock circle

2D Fourier plane of WE

Inside one prediction: Fourier wave decomposition

Top 5 Fourier components of the logits across 0…58, the other 24 frequencies (other), and the full centered logits below.

17 + 45 ≡ 3 (mod 59) Held-out test pair —
Attention at = across the 4 heads:

Published

November 20, 2026

Model

1-layer ReLU Transformer (49,024 params)

Engine

Rust WebAssembly SIMD, live in this tab

When you train a small neural network on a small table of arithmetic, something strange happens. First it memorizes the training pairs and fails completely on the rest. If you keep training with weight decay, long after training loss has hit zero, test accuracy suddenly jumps from near random to 100%. Power et al. named this delayed generalization grokking [1].

The demo above trains a real one-layer transformer from scratch in a Web Worker on your CPU, solving \((a + b) \bmod 59\) from only 35% of the \(59^2 = 3{,}481\) pairs. Every gradient, weight update, Fourier transform, and frequency knockout runs live in this tab.

The clock algorithm

How does a one-layer network with no arithmetic primitives learn modular addition? Nanda et al. [2] reverse-engineered the trained weights and found a trigonometric circuit (Gromov [4] gave a closed-form analytic version). Instead of storing a lookup table, the network picks a handful of key frequencies \(k \in \{1, \dots, 29\}\) (with \(\omega_k = 2\pi k / 59\)) and maps each input residue \(a\) onto circles in its embedding matrix \(W_E\):

$$a \longmapsto \big(\cos(\omega_k a),\ \sin(\omega_k a)\big)$$

Attention reads both \(a\) and \(b\) into the residual stream at the = token. Then the ReLU MLP mixes them to approximate the angle-addition identities:

$$\cos\!\big(\omega_k(a + b)\big) = \cos(\omega_k a)\cos(\omega_k b) - \sin(\omega_k a)\sin(\omega_k b)$$ $$\sin\!\big(\omega_k(a + b)\big) = \sin(\omega_k a)\cos(\omega_k b) + \cos(\omega_k a)\sin(\omega_k b)$$

Finally, the unembedding matrix \(W_U\) takes the dot product with \((\cos(\omega_k c), \sin(\omega_k c))\) for every candidate answer \(c \in \{0, \dots, 58\}\), producing a cosine wave \(\cos(\omega_k(a + b - c))\) that peaks at \(c \equiv a + b \pmod{59}\).

Because \(p = 59\) is prime, a single wave \(\cos(\omega_k(s - c))\) has a unique maximum at \(c = s\), but its closest runner-up is \(\cos(2\pi / 59) \approx 0.994\) — less than 0.6% lower — so any noise flips the prediction. Summing four or five different frequencies pushes their near-peaks onto different wrong residues, where they cancel out and open a wide margin on the true answer. You can see this in the wave panel above: before grokking the other lane carries most of the logits, and after grokking it flattens while the top five waves align on \(c^*\). Once the model has grokked, zeros out the other 24 frequencies in \(W_E\) and test accuracy holds near 100%; removes only those 5 frequencies and test accuracy drops to chance (\(1/59 \approx 1.7\%\)).

Three phases, and hidden progress

Watching only test accuracy makes grokking look sudden. Switch the top chart to to see the circuit form underneath (numbers below are for the default seed 1; click New seed to see how timing varies across runs) [2]:

Drag horizontally across the training chart at any time to scrub back to an earlier checkpoint and watch the clock circle untangle from a random cloud into a 59-gon.

What makes grokking happen

Grokking comes from the competition between two solutions: a high-norm lookup table that gradient descent finds quickly, and a low-norm Fourier algorithm that takes longer to organize. Two knobs control which one wins: enough training data to rule out shortcuts, and weight decay to penalize the lookup table [2], [3].

Default (35% data, wd = 2.2) Delayed decay (wd 0 → 2.2 at epoch 800) No weight decay (wd = 0) Too little data (25% data, wd = 2.2)
Figure 1. Test accuracy over 3,000 epochs across four controlled runs of seed 1 in grok.wasm. With zero weight decay, the network memorizes and stays at 3.1% test accuracy while its weight norm grows to 51.7. With only 25% of pairs, weight decay cannot force generalization (2.9%). Starting at zero weight decay and switching it on at epoch 800 still triggers full grokking (99.5% test accuracy; 99.7% with only its top 5 frequencies).

Try it live: run with and let it sit flat at 100% train / ~3% test. Then drag the weight decay slider up (or click ) mid-run without resetting to trigger generalization. You can also switch to subtraction (which uses a 40% train split, wd = 2.0, seed 42): without the \((a,b) \leftrightarrow (b,a)\) symmetry of addition, test accuracy stays below 1% until ~1,500 epochs and under 10% until ~2,500 before jumping to 100% near epoch 2,900.