DeepStress: Stress-Testing Deep Search Agents
THE QUICK READ
The paper proposes DeepStress, a dynamic stress-testing framework that exposes search agents to degraded evidence at controlled rates. A synthetic environment generates documents per query while independently controlling trustworthiness, relevance, and factuality. The authors add a five-way classification of final-answer outcomes and two reliability-aware metrics, RAS and CoPRAS, which credit cautious behavior and account for computational cost. Evaluating 12 systems on HotpotQA and BrowseComp-Plus questions, the authors report that several open-weight agents answer confidently despite obvious credibility red flags, and that many HotpotQA questions are answered from parametric knowledge even when all returned documents are factually wrong.
KEY POINTS
- Introduces DeepStress, a dynamic stress-testing framework that supplies degraded evidence at controlled rates through a synthetic environment with independent control of document trustworthiness, relevance, and factuality.
- Replaces binary accuracy with a five-way classification of final-answer outcomes and proposes RAS and CoPRAS, two reliability-aware metrics that credit cautious behavior and account for computational cost.
- The abstract reports an evaluation of 12 systems on HotpotQA and BrowseComp-Plus questions, finding that several open-weight agents answer confidently despite obvious credibility red flags and that many HotpotQA questions are answered from parametric knowledge even when all returned documents are factually wrong.
READING LIMITS
This summary is based only on the title and abstract; the full paper, experimental setup, metric definitions, and appendices were not reviewed, so the authors' claims and their scope could not be independently verified.
Source abstract [EN]
While search agents demonstrate impressive capabilities in multi-step question answering, their robustness to poor-quality evidence remains under-explored. Such evidence is rare in benchmark corpora, and resulting failures are hard to reproduce. We propose DeepStress, a dynamic stress-testing framework that subjects agents to degraded evidence at controlled rates. It replaces the retrieval module with a synthetic environment where documents are generated dynamically for each query, under independent control of their trustworthiness, relevance, and factuality. To characterize behavior under stress, we go beyond binary accuracy with a five-way classification of final-answer outcomes and two new reliability-aware metrics, RAS and CoPRAS, which credit cautious behavior and account for computational cost. Evaluating 12 systems on questions from HotpotQA and BrowseComp-Plus, we find that several open-weight agents answer confidently when sources contain obvious credibility red flags, and that many HotpotQA questions are answered from parametric knowledge even when all returned documents are factually wrong.
AI summaries use public titles and abstracts only. Research claims belong to the authors; consult the original paper.