Factored One-sided Adam-Moment for Practical and Scalable SOAP
THE QUICK READ
The paper introduces FOAM, a lightweight variant of the second-order optimizer SOAP. According to the authors, FOAM applies input-only preconditioning, drops the output-side Kronecker factor, and replaces SOAP's dense rotated second moment with a factored Adam-moment estimator built from row-by-column statistics. The abstract reports validation loss comparable to SOAP and Muon baselines in LLM pre-training from 124M to 760M parameters, with half of SOAP's optimizer state.
KEY POINTS
- FOAM uses input-only preconditioning, removing the output-side Kronecker factor to cut memory and one eigendecomposition.
- Under a K-FAC Fisher assumption with diagonal output curvature, the authors argue the factored estimator exactly recovers SOAP's dense second moment, reporting only 15-20% K-FAC violation.
- The abstract claims SOAP-level validation loss across 124M-760M parameter LLM pre-training, half of SOAP's optimizer state, Muon-class wall-clock time, and robustness over a three-fold learning-rate range.
READING LIMITS
Reading only the title and abstract, I cannot verify experimental setup, baseline tuning fairness, how the 15-20% violation was measured, or whether the scaling claims hold; all results are author-reported.
Source abstract [EN]
Second-order optimizers (e.g., Shampoo, SOAP) have emerged as compelling alternatives to Adam/AdamW and promise faster per-step convergence than AdamW in transformer training. However, they still remain costly in practice, and even SOAP still pays substantial optimizer-state memory and eigendecomposition overhead. We ask whether the benefits of SOAP can be retained at nearly the efficiency of AdamW. In this paper, we answer yes with a lightweight SOAP variant, Factored One-sided Adam-Moment (FOAM). The path to practical impact is twofold: first, FOAM employs input-only preconditioning, dropping the output-side Kronecker factor to reduce memory and remove one eigendecomposition. The Kronecker structure of the Hessian shows that this is exact for a diagonal output-side curvature and nearly lossless for a near-diagonal one, as we observe empirically. Second, FOAM replaces SOAP's dense rotated second moment with a factored Adam-moment estimator based on row-by-column statistics. Under the K-FAC Fisher assumption with a diagonal output curvature, this estimator exactly recovers SOAP's dense second moment, and we measure only a 15-20% K-FAC violation, with negligible effect on the update. Our empirical results support the claim that FOAM matches strong SOAP and Muon baselines in validation loss across scales of large language model (LLM) pre-training, uses half of SOAP's optimizer state, runs at Muon-class wall-clock time, and retains robustness across a three-fold learning-rate range. The results hold from 124M to 760M parameters, demonstrating that SOAP's gains need not require SOAP's cost. Code: https://github.com/naver-ai/foam
AI summaries use public titles and abstracts only. Research claims belong to the authors; consult the original paper.