akougkas.io

Agent Error Simulator: Fault Injection for Testing Recovery in Agentic Workflows

Keith Bateman, Jaime Cernuda, Luke Logan, Bogdan Nicolae, Franck Cappello, Xian-He Sun, Anthony Kougkas

22nd IEEE International Conference on eScience (eScience 2026)IEEEPresented

Open the DOI

Abstract

Multi-agent large language model (LLM) workflows are increasingly deployed for complex, multi-step computational tasks. Yet the fault tolerance of these systems (how individual agents respond to malformed inputs, tool failures, and subtle numerical errors) remains poorly characterized. We present the Agent Error Simulator (AES), a benchmark framework that evaluates LLM agent fault tolerance through controlled, declarative fault injection. AES intercepts agent-to-model HTTP traffic via a transparent proxy, injecting three categories of faults (format, logic, and tool call) at configurable points in a hierarchical planner-worker-aggregator workflow. We evaluate four models spanning 2B–32B parameters (Qwen2.5-Coder 32B, Gemma4 27B, Mistral 7B, and Granite4 2B) across 194 jobs organized into 10 test groups covering baseline behavior, multi-error injection, workflow scale, context pressure, compaction, and detection timing. Our results reveal a model-size-dependent logic error recovery threshold between 7B and 27B parameters; universal catastrophic failure under aggregator-level logic injection; and model-specific aggregator arithmetic deficits that interact with injected faults in non-obvious ways. We release the benchmark suite, injection tooling, and result data to support reproducible multi-agent fault tolerance research.

Finding

Across 194 jobs and four models, logic error recovery requires at least 27B parameters in the evaluated models, aggregator-level logic injection causes universal catastrophic failure, and detection timing does not change the outcome.

System

  1. 2023–2026
    complete

    DTIO

    A task-based I/O runtime built with Argonne under a DOE ASCR award. Labels became composable data tasks, online format translation with caching beat offline conversion by 49.6%, and the mechanisms were reimplemented inside CLIO Core.

Browse all papers on DTIO
  1. 2026
    poster

    AgentRecall: Provenance and Rewind for Multi-Agent Workflows

    Keith Bateman, Xian-He Sun, Anthony Kougkas. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)

  2. 2025
    conference

    DTIO: Data Stack for AI-driven Workflows

    Keith Alex Bateman, Neeraj Rajesh, Jaime Cernuda, Luke Logan, Bogdan Nicolae, … Anthony Kougkas. Proceedings of the 37th International Conference on Scalable Scientific Data Management

  3. 2025
    poster

    A Unified Data Stack for AI Workflows

    Keith Bateman, Neeraj Rajesh, Jaime Cernuda, Luke Logan, Bogdan Nicolae, … Anthony Kougkas. 37th International Conference on Scalable Scientific Data Management (SSDBM 2025)

  4. 2024
    conference

    Viper: A High-Performance I/O Framework for Transparently Updating, Storing, and Transferring Deep Neural Network Models

    Jie Ye, Jaime Cernuda, Neeraj Rajesh, Keith Bateman, Orcun Yildiz, … Anthony Kougkas et al. Proceedings of the 53rd International Conference on Parallel Processing

Citation

@inproceedings{bateman2026aes,
  author = {Bateman, K. and Cernuda, J. and Logan, L. and Nicolae, B. and Cappello, F. and Sun, X.-H. and Kougkas, A.},
  title = {{Agent Error Simulator: Fault Injection for Testing Recovery in Agentic Workflows}},
  booktitle = {22nd IEEE International Conference on eScience (eScience 2026)},
  year = {2026},
  month = sep,
  doi = {10.1109/eScience70237.2026.00052},
  pages = {383--391},
  url = {https://www.escience-conference.org/2026/papers},
  publisher = {IEEE},
}

Related notes

  1. essay

    One destination, N tiers

    Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.

    9 min
  2. essay

    DTIO: turning labels into schedulable data tasks

    A data task is a label with a schedule. What DTIO adds to label-based I/O, what the SSDBM 2025 paper measured, and what a task runtime still has to prove.

    7 min
  3. update

    DOE funds DTIO

    The Department of Energy has funded DTIO, a three-year project with Argonne to turn I/O requests into schedulable data tasks. It is what LABIOS labels become when a pipeline has more than one stage.

    4 min