# beta2 sweep at fixed norm N=88 (p=97): where does Adam's stale second moment take over from eps?

Run 2026-10-01 (22:45-23:45 UTC), CPU, single-threaded, 2 processes at a time.
Files: queue `b2sweep_queue_full.txt`, worker `b2sweep_worker.sh`, stdout `run_b2sweep.log`, per-run CSV `beta2_sweep.csv`,
analysis `b2sweep_analyze.py` (table, also saved as `beta2_sweep_table.md`), `b2sweep_vhat.py` (sqrt(v_hat)/g), figure `fig_beta2_sweep.py` -> `fig_beta2_sweep.png`.
Logs are in `logs/` under the usual clamp_gen tags.

## Method
clamp_gen.py: 2-layer MLP, (a+b) mod 97, 50% train, full batch, AdamW lr 1e-3, beta1 0.9, wd 0, total weight norm rescaled to N after every step; --steps 20000, --patience 6 (default, same as earlier runs).
T_grok = first logged step (50-step grid) with test acc >= 0.99. Censored runs (not reached by 20000) shown as ">20000".
Cells: N=88, beta2 in {0.95, 0.98, 0.985, 0.99, 0.995, 0.999} x eps in {1e-8, 1e-16} x seeds {0, 1} (0.999 also seed 2, from the previous run); N=65 at beta2 {0.95, 0.99, 0.995, 0.999}.
New this run: all 0.95, 0.985, 0.995 cells, 0.99 seed 1, and a rerun of 0.98 N=88 in clamp_gen.py (the earlier 0.98 runs were in ../temp-clamp/logs; the rerun reproduces T exactly: 5200/6400 eps1e-8, 2100/2300 eps1e-16). 0.985 was added after the first pass to bracket the predicted crossover.

Collapse rate r: least-squares slope of -ln(grad_rms) vs step (per 1000 steps) over [first log with frac_sat > 0.5, first log with grad_rms <= 1.5 x run minimum (the float32 floor, about 5e-10 to 1e-9)]. r_pre: same fit over [first grad_rms < 1e-6, frac_sat > 0.5]. N=65 runs never reach frac_sat > 0.5 or grad_rms < 1e-8, so r is undefined there.
sqrt(v_hat)/g measured directly with ../analysis/update_size_b2.py (N=88, eps 1e-16, seed 0, beta2 0.99/0.995/0.999; outputs ../analysis/update_size_b2-<b2>_N88_eps1e-16.csv).

## Table (ratio = T(eps 1e-8)/T(eps 1e-16), same seed; r/((1-beta2)/2) = 1 means the collapse rate equals the second-moment decay rate)
| N | beta2 | (1-beta2)/2 per 1000 | eps | seed | T_grok | ratio eps1e-8/eps1e-16 | r per 1000 (sat window) | r/((1-beta2)/2) | r_pre per 1000 | step frac_sat>0.5 | step g<1e-8 |
| 88 | 0.95 | 25 | 1e-08 | 0 | 3300 | 1.69 | 9.54 | 0.38 | 6.45 | 1500 | 1800 |
| 88 | 0.95 | 25 | 1e-08 | 1 | 5350 | 2.28 | 7.75 | 0.31 | 5.72 | 1650 | 2000 |
| 88 | 0.95 | 25 | 1e-16 | 0 | 1950 |  | 9.32 | 0.37 | 6.12 | 1500 | 1850 |
| 88 | 0.95 | 25 | 1e-16 | 1 | 2350 |  | 6.86 | 0.27 | 4.69 | 1750 | 2200 |
| 88 | 0.98 | 10 | 1e-08 | 0 | 5200 | 2.48 | 6.42 | 0.64 | 6.07 | 1350 | 1750 |
| 88 | 0.98 | 10 | 1e-08 | 1 | 6400 | 2.78 | 5.66 | 0.57 | 4.62 | 1500 | 2000 |
| 88 | 0.98 | 10 | 1e-16 | 0 | 2100 |  | 8.48 | 0.85 | 6.27 | 1350 | 1700 |
| 88 | 0.98 | 10 | 1e-16 | 1 | 2300 |  | 7.05 | 0.71 | 4.64 | 1500 | 1950 |
| 88 | 0.985 | 7.5 | 1e-08 | 0 | 5400 | 2.25 | 5.64 | 0.75 | 5.95 | 1400 | 1850 |
| 88 | 0.985 | 7.5 | 1e-08 | 1 | 6450 | 2.58 | 5.18 | 0.69 | 5.44 | 1500 | 2000 |
| 88 | 0.985 | 7.5 | 1e-16 | 0 | 2400 |  | 7.18 | 0.96 | 5.97 | 1400 | 1850 |
| 88 | 0.985 | 7.5 | 1e-16 | 1 | 2500 |  | 6.59 | 0.88 | 5.31 | 1500 | 1950 |
| 88 | 0.99 | 5 | 1e-08 | 0 | 6200 | 2.00 | 4.41 | 0.88 | 4.76 | 1700 | 2300 |
| 88 | 0.99 | 5 | 1e-08 | 1 | 7200 | 2.22 | 4.03 | 0.81 | 4.53 | 1750 | 2350 |
| 88 | 0.99 | 5 | 1e-16 | 0 | 3100 |  | 5.26 | 1.05 | 4.8 | 1650 | 2250 |
| 88 | 0.99 | 5 | 1e-16 | 1 | 3250 |  | 5 | 1.00 | 4.46 | 1750 | 2350 |
| 88 | 0.995 | 2.5 | 1e-08 | 0 | 8000 | 1.51 | 2.51 | 1.00 | 2.64 | 2500 | 3600 |
| 88 | 0.995 | 2.5 | 1e-08 | 1 | 8850 | 1.62 | 2.42 | 0.97 | 2.6 | 2550 | 3650 |
| 88 | 0.995 | 2.5 | 1e-16 | 0 | 5300 |  | 2.73 | 1.09 | 2.65 | 2500 | 3600 |
| 88 | 0.995 | 2.5 | 1e-16 | 1 | 5450 |  | 2.66 | 1.06 | 2.59 | 2550 | 3650 |
| 88 | 0.999 | 0.5 | 1e-08 | 0 | >20000 | >1.06 | 0.553 | 1.11 | 0.606 | 6600 | 11950 |
| 88 | 0.999 | 0.5 | 1e-08 | 1 | >20000 | >1.01 | 0.539 | 1.08 | 0.601 | 6750 | 12150 |
| 88 | 0.999 | 0.5 | 1e-08 | 2 | >20000 | >1.03 | 0.552 | 1.10 | 0.594 | 6800 | 12000 |
| 88 | 0.999 | 0.5 | 1e-16 | 0 | 18900 |  | 0.547 | 1.09 | 0.599 | 6650 | 12050 |
| 88 | 0.999 | 0.5 | 1e-16 | 1 | 19750 |  | 0.544 | 1.09 | 0.605 | 6700 | 12100 |
| 88 | 0.999 | 0.5 | 1e-16 | 2 | 19450 |  | 0.548 | 1.10 | 0.592 | 6850 | 12100 |

N=65 (below the knee):

| 65 | 0.95 | 25 | 1e-08 | 0 | 1650 | 1.06 | nan | - | nan | None | None |
| 65 | 0.95 | 25 | 1e-16 | 0 | 1550 |  | nan | - | nan | None | None |
| 65 | 0.99 | 5 | 1e-08 | 0 | 1350 | 1.00 | nan | - | nan | None | None |
| 65 | 0.99 | 5 | 1e-16 | 0 | 1350 |  | nan | - | nan | None | None |
| 65 | 0.995 | 2.5 | 1e-08 | 0 | 1600 | 0.97 | nan | - | nan | None | None |
| 65 | 0.995 | 2.5 | 1e-16 | 0 | 1650 |  | nan | - | nan | None | None |
| 65 | 0.999 | 0.5 | 1e-08 | 0 | 2400 | 1.02 | nan | - | nan | None | None |
| 65 | 0.999 | 0.5 | 1e-08 | 1 | 3350 | 0.99 | nan | - | nan | None | None |
| 65 | 0.999 | 0.5 | 1e-16 | 0 | 2350 |  | nan | - | nan | None | None |
| 65 | 0.999 | 0.5 | 1e-16 | 1 | 3400 |  | nan | - | nan | None | None |

Seed means at N=88 (eps 1e-16 / eps 1e-8; r mean over the eps 1e-16 seeds):
| beta2 | (1-beta2)/2 per 1000 | T eps1e-16 | T eps1e-8 | ratio | eps penalty T(1e-8)-T(1e-16) | r eps1e-16 |
|---|---|---|---|---|---|---|
| 0.95 | 25 | 2150 | 4325 | 2.0 | 2175 | 8.1 |
| 0.98 | 10 | 2200 | 5800 | 2.6 | 3600 | 7.8 |
| 0.985 | 7.5 | 2450 | 5925 | 2.4 | 3475 | 6.9 |
| 0.99 | 5 | 3175 | 6700 | 2.1 | 3525 | 5.1 |
| 0.995 | 2.5 | 5375 | 8425 | 1.6 | 3050 | 2.7 |
| 0.999 | 0.5 | 19367 (n=3) | >20000 (all censored) | >1.03 | >633 | 0.55 |

![beta2 sweep](fig_beta2_sweep.png)

## Direct check: sqrt(v_hat)/g (update_size_b2.py, N=88, eps 1e-16, seed 0; rates per 1000 steps over the saturation window)
| beta2 | window | r (g decay) | sqrt(v_hat) decay | (1-beta2)/2 | slope ln(sqrt(v_hat)/g) | predicted r-(1-beta2)/2 | sqrt(v_hat)/g at window start -> end | step/lr start -> end |
|---|---|---|---|---|---|---|---|---|
| 0.99 | 1650-2700 | 5.26 | 4.81 | 5 | 0.44 | 0.26 | 4.0 -> 5.9 | 0.25 -> 0.20 |
| 0.995 | 2500-4500 | 2.73 | 2.49 | 2.5 | 0.24 | 0.23 | 9.3 -> 14.1 | 0.13 -> 0.094 |
| 0.999 | 6650-13300 | 0.552 | 0.500 | 0.5 | 0.052 | 0.052 | 47 -> 67 | 0.031 -> 0.024 |
sqrt(v_hat)/g is already 3-4 (0.99), 7-8 (0.995), 11-36 (0.999) by steps 1000-4000, i.e. long before softmax saturation; it then grows only slowly.

## Conclusions
1. **The collapse rate is not an independent property of N; it locks to the second-moment decay rate.** With undamped dynamics (beta2 0.95/0.98, eps 1e-16) the gradient at N=88 collapses at r0 = 6.9-9.3 per 1000 steps (mean ~8). Whenever (1-beta2)/2 < r0 the measured r equals (1-beta2)/2 to within 0-10% (r/((1-beta2)/2) = 0.88-1.09 at 0.985-0.999 with eps 1e-16), and sqrt(v_hat) decays at exactly (1-beta2)/2 (pure memory decay, new g^2 contributions negligible). Mechanism: if g fell faster than sqrt(v_hat), the Adam step (~ lr * g/sqrt(v_hat)) would shrink and slow the collapse, so the system sits on a slow manifold with r just above (1-beta2)/2. The original hypothesis' exponential growth of sqrt(v_hat)/g at rate r-(1-beta2)/2 is confirmed quantitatively at 0.995 (0.24 vs 0.23) and 0.999 (0.052 vs 0.052), but the excess rate is small (5-10% of r) because of this feedback; the large damping factor (sqrt(v_hat)/g = 4-60) is mostly built up before saturation. The test "2r > 1-beta2" is therefore not a useful in-run diagnostic (it is marginal, ~1.0-1.1, in every damped run); the useful predictor is the comparison of the undamped r0(N) with (1-beta2)/2.
2. **(a) eps effect vs beta2:** ratio T(1e-8)/T(1e-16) is 2.0 (0.95), 2.6 (0.98), 2.4 (0.985), 2.1 (0.99), 1.6 (0.995), and ~1 (0.999: eps1e-16 groks at 18900-19750, eps1e-8 at test 0.975-0.989 at 20000). In absolute steps the eps penalty is roughly constant (3000-3600 steps) from 0.98 to 0.995, so the falling ratio is mostly a growing denominator; at 0.999 the penalty is small because sqrt(v_hat) stays 50-70x above g and is still ~2e-7 at step 14000, so eps=1e-8 is never comparable to the denominator during the run. eps matters when sqrt(v_hat), not g, approaches eps.
   **(b) eps-free time:** T(eps 1e-16) is flat at 2150-2200 for beta2 0.95-0.98, then rises: 2450 (0.985), 3175 (0.99), 5375 (0.995), 19367 (0.999). The post-saturation duration T - step(frac_sat>0.5) is 450-800 (0.95-0.98), 1450-1500 (0.99), 2800-2900 (0.995), 12250-13050 (0.999), roughly ~1/(1-beta2) once damped (ratios 1 : 2 : 8.5 vs 1 : 2 : 10).
   **(c) crossover:** predicted where (1-beta2)/2 = r0, i.e. beta2* = 1 - 2 r0 = 0.981-0.986 (r0 = 6.9-9.3/1000), centre 0.984. Observed: eps-free T and r are unchanged from 0.95 to 0.98, start to rise at 0.985 (T +11%, r 6.9 = 0.92 x (1-beta2)/2) and are clearly staleness-limited at 0.99 (T +44%, r = (1-beta2)/2). So the crossover is at beta2 ~ 0.985, matching 2 r0 = 1 - beta2. The eps-dominated regime (ratio > 2) spans beta2 <= 0.99; staleness dominates from ~0.995 (eps-free T 2.4x the undamped value, eps ratio down to 1.6) and is total at 0.999.
3. Below the knee (N=65, gradient never collapses below ~1e-6, never saturates) eps has no effect at any beta2 (ratios 0.97-1.06), and T is 1350-1650 for beta2 0.95-0.995 vs 2350-3400 at 0.999; the 0.999 slowdown at N=65 is a milder staleness effect not captured by the saturation-window r.

## Caveats
- 2 seeds per cell (3 at 0.999), one modulus, one lr, 50-step grid; seed spread in T is up to 60% for eps 1e-8 at 0.95 (3300 vs 5350), so ratios carry ~+-25% uncertainty; eps-free T is much tighter (<20%).
- r fits use 50-step logs over windows of 6-130 points; the window for eps 1e-8 is contaminated by eps damping in the last part (lower r at 0.95-0.99 with eps 1e-8), so the crossover estimate uses eps 1e-16 runs only.
- r0 (undamped rate) is taken from the beta2 0.95/0.98 runs; it is itself seed dependent (6.9-9.3).
- sqrt(v_hat)/g was measured for one seed and eps 1e-16 only. At 0.99 the measured ratio growth (0.44) exceeds the prediction (0.26) because the window ends near the gradient floor where g flattens.
- clamp_gen.py was edited by another agent at 22:46 (adds nparam to the final print and a --d_hid flag, default unchanged); the 0.98 rerun reproduces earlier T values exactly, so training is unaffected.