SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models
THE QUICK READ
The authors introduce SO-Arena, a benchmark covering 11 domains and five capability families to assess System One models. It uses reliable sources and manually verified reference labels, domain-specific difficulty criteria, and sandboxed interactive tasks, and also measures decision stability under controlled option-set changes and missing-answer recognition. Across 21 models, the authors report that scaling and extra reasoning help unevenly by domain, and that the top-performing model ranks only seventh in reliability.
KEY POINTS
- Covers 11 domains and five capability families, with evidence-grounded evaluation using reliable sources and manually verified reference labels.
- Assesses reliability beyond accuracy through decision stability under controlled option-set changes and recognition of missing answers when no valid option exists.
- The authors report results for 21 models: benefits of scaling and additional reasoning vary by domain, and the best overall model ranks only seventh in reliability.
READING LIMITS
Based only on the title and abstract; the full paper was not read, so benchmark design details, label-verification procedures, statistical methods, and peer review status cannot be confirmed. All figures and conclusions are the authors' claims.
Source abstract [EN]
System One models promise fast, general-purpose decision-making without lengthy token generation. However, understanding their capability spectrum and decision reliability requires more than aggregate performance on current benchmarks. Existing evaluations often provide limited insight into the breadth of model capabilities and the reliability of their decisions. We introduce SO-Arena, a benchmark characterizing the capability spectrum and decision reliability of System One models across 11 domains and five capability families. For the capability spectrum, SO-Arena offers evidence-grounded evaluation through reliable sources and manual verification of reference labels against model-visible inputs; fine-grained capability assessment through domain-specific difficulty criteria; and realistic interactive evaluation, where models iteratively execute actions in sandboxes. Reasoning-intensive tasks further enable system one-two comparison. Beyond accuracy, SO-Arena assesses decision reliability through two complementary measures: decision stability under controlled option-set changes and missing-answer recognition when no valid option is available. Evaluations across 21 models reveal that the benefits of scaling and additional reasoning vary substantially across domains. More importantly, strong overall performance does not necessarily imply reliable decision-making: the best-performing System One model ranks only seventh in reliability. Together, these findings highlight the need to characterize capability spectrum and decision reliability as complementary dimensions, providing a systematic view of the strengths and limitations of System One models.
AI summaries use public titles and abstracts only. Research claims belong to the authors; consult the original paper.