Agent Error Simulator: Fault Injection for Testing Recovery in Agentic Workflows
Abstract
Multi-agent large language model (LLM) workflows are increasingly deployed for complex, multi-step computational tasks. Yet the fault tolerance of these systems (how individual agents respond to malformed inputs, tool failures, and subtle numerical errors) remains poorly characterized. We present the Agent Error Simulator (AES), a benchmark framework that evaluates LLM agent fault tolerance through controlled, declarative fault injection. AES intercepts agent-to-model HTTP traffic via a transparent proxy, injecting three categories of faults (format, logic, and tool call) at configurable points in a hierarchical planner-worker-aggregator workflow. We evaluate four models spanning 2B–32B parameters (Qwen2.5-Coder 32B, Gemma4 27B, Mistral 7B, and Granite4 2B) across 194 jobs organized into 10 test groups covering baseline behavior, multi-error injection, workflow scale, context pressure, compaction, and detection timing. Our results reveal a model-size-dependent logic error recovery threshold between 7B and 27B parameters; universal catastrophic failure under aggregator-level logic injection; and model-specific aggregator arithmetic deficits that interact with injected faults in non-obvious ways. We release the benchmark suite, injection tooling, and result data to support reproducible multi-agent fault tolerance research.
Finding
Across 194 jobs and four models, logic error recovery requires at least 27B parameters in the evaluated models, aggregator-level logic injection causes universal catastrophic failure, and detection timing does not change the outcome.
System
- 2023–2026
DTIO
A task-based I/O runtime built with Argonne under a DOE ASCR award. Labels became composable data tasks, online format translation with caching beat offline conversion by 49.6%, and the mechanisms were reimplemented inside CLIO Core.
More papers on DTIO
Browse all papers on DTIO- 2026
AgentRecall: Provenance and Rewind for Multi-Agent Workflows
Keith Bateman, Xian-He Sun, Anthony Kougkas. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)
- 2025
DTIO: Data Stack for AI-driven Workflows
Keith Alex Bateman, Neeraj Rajesh, Jaime Cernuda, Luke Logan, Bogdan Nicolae, … Anthony Kougkas. Proceedings of the 37th International Conference on Scalable Scientific Data Management
- 2025
A Unified Data Stack for AI Workflows
Keith Bateman, Neeraj Rajesh, Jaime Cernuda, Luke Logan, Bogdan Nicolae, … Anthony Kougkas. 37th International Conference on Scalable Scientific Data Management (SSDBM 2025)
- 2024
Viper: A High-Performance I/O Framework for Transparently Updating, Storing, and Transferring Deep Neural Network Models
Jie Ye, Jaime Cernuda, Neeraj Rajesh, Keith Bateman, Orcun Yildiz, … Anthony Kougkas et al. Proceedings of the 53rd International Conference on Parallel Processing
Citation
@inproceedings{bateman2026aes,
author = {Bateman, K. and Cernuda, J. and Logan, L. and Nicolae, B. and Cappello, F. and Sun, X.-H. and Kougkas, A.},
title = {{Agent Error Simulator: Fault Injection for Testing Recovery in Agentic Workflows}},
booktitle = {22nd IEEE International Conference on eScience (eScience 2026)},
year = {2026},
month = sep,
doi = {10.1109/eScience70237.2026.00052},
pages = {383--391},
url = {https://www.escience-conference.org/2026/papers},
publisher = {IEEE},
} Related notes
- 9 min
One destination, N tiers
Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.
- 7 min
DTIO: turning labels into schedulable data tasks
A data task is a label with a schedule. What DTIO adds to label-based I/O, what the SSDBM 2025 paper measured, and what a task runtime still has to prove.
- 4 min
DOE funds DTIO
The Department of Energy has funded DTIO, a three-year project with Argonne to turn I/O requests into schedulable data tasks. It is what LABIOS labels become when a pipeline has more than one stage.