WORLDWIDE_SanGuai / Research bulletin

[ worldwide.arakne.de ]

世界很大,讀一小點。

SanGuai 的三日研究佈告板。每天更新,讀完就走。

EST. 202672H上架 72 小時
人工智慧 / 2611.15777

SO-Arena:評測系統一模型的能力光譜與決策可靠性

SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models

作者: Lichen Li, Ziping Dong, Sicheng Feng, Qi Li, Xinchao Wang論文發布: 2026-11-11離架倒數: 15 小時

研究速讀

作者提出 SO-Arena 基準,涵蓋 11 個領域與五類能力,用可靠來源與人工核對標籤評估系統一模型,並在沙箱中進行互動測試。除準確率外,也衡量選項變動下的決策穩定性與無正確答案時的辨識能力。作者稱 21 個模型的結果顯示,規模與推理帶來的效益因領域而異,且整體表現最佳者可靠性僅排第七。

值得留意

  1. 涵蓋 11 個領域與五類能力,並以可靠來源及人工核對參考標籤來支撐評測。
  2. 除準確率外,另以選項集受控變動下的決策穩定性,以及無有效選項時的缺答辨識來衡量可靠性。
  3. 作者報告 21 個模型的評測結果:規模與額外推理的效益因領域而異,整體最佳模型在可靠性上僅排第七。

閱讀限制

僅依標題與摘要整理,未讀全文,無法確認基準設計細節、標籤核對流程、統計方法或結果是否經同儕審查;所有數據與結論均為作者陳述。

原始摘要 [EN]

System One models promise fast, general-purpose decision-making without lengthy token generation. However, understanding their capability spectrum and decision reliability requires more than aggregate performance on current benchmarks. Existing evaluations often provide limited insight into the breadth of model capabilities and the reliability of their decisions. We introduce SO-Arena, a benchmark characterizing the capability spectrum and decision reliability of System One models across 11 domains and five capability families. For the capability spectrum, SO-Arena offers evidence-grounded evaluation through reliable sources and manual verification of reference labels against model-visible inputs; fine-grained capability assessment through domain-specific difficulty criteria; and realistic interactive evaluation, where models iteratively execute actions in sandboxes. Reasoning-intensive tasks further enable system one-two comparison. Beyond accuracy, SO-Arena assesses decision reliability through two complementary measures: decision stability under controlled option-set changes and missing-answer recognition when no valid option is available. Evaluations across 21 models reveal that the benefits of scaling and additional reasoning vary substantially across domains. More importantly, strong overall performance does not necessarily imply reliable decision-making: the best-performing System One model ranks only seventh in reliability. Together, these findings highlight the need to characterize capability spectrum and decision reliability as complementary dimensions, providing a systematic view of the strengths and limitations of System One models.

AI 摘要僅根據公開標題與摘要整理;研究結論來自作者,請以原文為準。