用於實用且可擴展 SOAP 的分解式單側 Adam 動量
Factored One-sided Adam-Moment for Practical and Scalable SOAP
研究速讀
這篇論文提出 FOAM,一種輕量化的 SOAP 變體。作者主張 FOAM 只做輸入側預條件,捨棄輸出側 Kronecker 因子,並以基於列與行統計的分解式 Adam 動量估計器取代稠密旋轉二階動量,目標是在接近 AdamW 的效率下保留 SOAP 的效益。摘要稱其在 124M 至 760M 參數的 LLM 預訓練上,驗證損失可與 SOAP 及 Muon 基線相當。
值得留意
- FOAM 採用僅輸入側的預條件,移除輸出側 Kronecker 因子,以降低記憶體並少一次特徵分解。
- 作者以 K-FAC Fisher 假設與對角輸出曲率論證:分解式估計器可精確還原 SOAP 的稠密二階動量,並稱實測 K-FAC 違反程度僅 15–20%。
- 摘要稱 FOAM 使用 SOAP 一半的優化器狀態、達到 Muon 等級的實際耗時,並在 124M 至 760M 參數規模維持與 SOAP、Muon 相當的驗證損失。
閱讀限制
僅閱讀標題與摘要,無法核實實驗設定、基線調參公平性、15–20% 違反程度的測量方式,以及跨規模結論是否穩健;這些皆為作者自述。
原始摘要 [EN]
Second-order optimizers (e.g., Shampoo, SOAP) have emerged as compelling alternatives to Adam/AdamW and promise faster per-step convergence than AdamW in transformer training. However, they still remain costly in practice, and even SOAP still pays substantial optimizer-state memory and eigendecomposition overhead. We ask whether the benefits of SOAP can be retained at nearly the efficiency of AdamW. In this paper, we answer yes with a lightweight SOAP variant, Factored One-sided Adam-Moment (FOAM). The path to practical impact is twofold: first, FOAM employs input-only preconditioning, dropping the output-side Kronecker factor to reduce memory and remove one eigendecomposition. The Kronecker structure of the Hessian shows that this is exact for a diagonal output-side curvature and nearly lossless for a near-diagonal one, as we observe empirically. Second, FOAM replaces SOAP's dense rotated second moment with a factored Adam-moment estimator based on row-by-column statistics. Under the K-FAC Fisher assumption with a diagonal output curvature, this estimator exactly recovers SOAP's dense second moment, and we measure only a 15-20% K-FAC violation, with negligible effect on the update. Our empirical results support the claim that FOAM matches strong SOAP and Muon baselines in validation loss across scales of large language model (LLM) pre-training, uses half of SOAP's optimizer state, runs at Muon-class wall-clock time, and retains robustness across a three-fold learning-rate range. The results hold from 124M to 760M parameters, demonstrating that SOAP's gains need not require SOAP's cost. Code: https://github.com/naver-ai/foam
AI 摘要僅根據公開標題與摘要整理;研究結論來自作者,請以原文為準。