DeepStress:對深度搜尋代理進行壓力測試
DeepStress: Stress-Testing Deep Search Agents
研究速讀
這篇論文提出 DeepStress,一個動態壓力測試框架,用來檢驗搜尋代理在證據品質下降時是否仍可靠。作者以合成環境控制文件的可信度、相關性與事實正確性,並提出五類答案結果分類及 RAS、CoPRAS 兩項指標。摘要指出,在 HotpotQA 與 BrowseComp-Plus 上評估 12 個系統後,部分開放權重代理在來源有明顯可信度警訊時仍自信作答,且許多 HotpotQA 問題即使所有檢索文件皆為錯誤事實,仍能靠參數知識答對。
值得留意
- 提出 DeepStress 動態壓力測試框架,以合成環境在可控比率下提供劣化證據,並分別控制文件的可信度、相關性與事實正確性。
- 以五類最終答案結果分類,並提出 RAS 與 CoPRAS 兩項考量謹慎行為與計算成本的新指標,取代單純的二元正確率。
- 摘要稱在 HotpotQA 與 BrowseComp-Plus 上評估 12 個系統,發現部分開放權重代理面對明顯可信度警訊仍自信作答,且許多 HotpotQA 問題在所有檢索文件事實錯誤時仍靠參數知識答對。
閱讀限制
僅閱讀標題與摘要,未檢視全文、實驗設計細節、指標定義或附錄,因此無法驗證結果,也無法判斷這些發現的適用範圍與限制。
原始摘要 [EN]
While search agents demonstrate impressive capabilities in multi-step question answering, their robustness to poor-quality evidence remains under-explored. Such evidence is rare in benchmark corpora, and resulting failures are hard to reproduce. We propose DeepStress, a dynamic stress-testing framework that subjects agents to degraded evidence at controlled rates. It replaces the retrieval module with a synthetic environment where documents are generated dynamically for each query, under independent control of their trustworthiness, relevance, and factuality. To characterize behavior under stress, we go beyond binary accuracy with a five-way classification of final-answer outcomes and two new reliability-aware metrics, RAS and CoPRAS, which credit cautious behavior and account for computational cost. Evaluating 12 systems on questions from HotpotQA and BrowseComp-Plus, we find that several open-weight agents answer confidently when sources contain obvious credibility red flags, and that many HotpotQA questions are answered from parametric knowledge even when all returned documents are factually wrong.
AI 摘要僅根據公開標題與摘要整理;研究結論來自作者,請以原文為準。