Mutation Testing of Task-Scoped State Oracles in Software-Agent Benchmarks: A Cross-Benchmark Empirical Study

<title>Abstract</title> <p>Stateful software-agent benchmarks increasingly use executable evaluators to decide whether an agent produced the requested persistent effects. Yet a high task score is informative only if the evaluator rejects harmful extra effects without rejecting equivalent representations. We introduce a deterministic mutation-testing protocol for calibrating such task-scoped state oracles. Starting from known-valid executions, the protocol injects six classes of harmful state faults and three schema-certified benign transformations, then compares official scores with three controlled reference rules. Across frozen samples from τ²-Bench, ToolSandbox, and AppWorld, 53 of 60 attempted task templates yielded 145 applicable mutations. Official evaluators rejected 100 of 110 harmful mutations (90.9%; Wilson 95% CI 84.1–95.0%) and preserved all 35 applicable JSON-key-order variants (95% CI 90.1–100%). All ten false negatives occurred in ToolSandbox: four same-record and six cross-namespace collateral changes across six templates. A post-hoc diagnostic reproduced every extra effect through its public tool; all ten still received score 1.0 with unchanged evaluation hashes. AppWorld contributed only five applicable harmful mutations, so the study does not support a broad benchmark ranking. The results motivate allowed-state-delta regression tests that verify required changes and reject unexpected persistent changes.</p>