akougkas.io

DaYu: Optimizing Distributed Scientific Workflows by Decoding Dataflow Semantics and Dynamics

Meng Tang, Jaime Cernuda, Jie Ye, Luanzheng Guo, Nathan R. Tallent, Anthony Kougkas, Xian-He Sun

Proceedings of the IEEE International Conference on Cluster ComputingIEEE

Abstract

The combination of ever-growing scientific datasets and distributed workflow complexity creates I/O performance bottlenecks due to data volume, velocity, and variety. Although the increasing use of descriptive data formats (e.g., HDF5, netCDF) helps organize these datasets, it also introduces obscure bottlenecks due to the need to translate high-level operations into file addresses and then into low-level I/O operations. To address this challenge, we introduce DaYu, a method and toolset for analyzing (a) semantic relationships between logical datasets and file addresses, (b) how dataset operations translate into I/O, and (c) the combination across entire workflows. DaYu’s analysis and visualization enable the identification of critical bottlenecks and the reasoning about remediation. We describe our methodology and propose optimization guidelines. Evaluation on scientific workflows demonstrates up to a 3.7x performance improvement in I/O time for obscure bottlenecks. The time and storage overhead for DaYu’s time-ordered data are typically under 0.2% of runtime and 0.25% of data volume, respectively.

Finding

DaYu’s analysis of dataset semantics and I/O identifies obscure workflow bottlenecks, improving I/O time by up to 3.7x with overheads typically under 0.2% of runtime and 0.25% of data volume.

System

  1. 2020–present
    active

    DeepIO

    A line of subcontracts with Argonne, Pacific Northwest, and Lawrence Livermore that funds students working where I/O meets AI, from Viper and KV-cache scheduling for LLM inference to WisIO's I/O bottleneck analysis.

  2. 2018–2024
    complete

    Hermes

  3. 2024–present
    active

    IOWarp

  4. experimental

    PORTUS

Browse all papers on DeepIO
  1. 2026
    poster

    Extending I/O Observability to Explainability for Targeted Optimization

    Neeraj Rajesh, S. Neuwirth, Jay Lofstead, Anthony Kougkas, Xian-He Sun, Nikoli Dryden et al. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)

  2. 2026
    conference

    PKAS: Predictive KVCache-Aware Scheduling for Faster LLM and Transformer Inferences

    Jie Ye, A. Maurya, K. Chitty-Venkata, Bogdan Nicolae, Anthony Kougkas, Xian-He Sun. 35th International Symposium on High-Performance Parallel and Distributed Computing (HPDC 2026)

  3. 2026
    poster

    Pre-RoPE versus Post-RoPE: Trading Per-Step Compute for KVCache Survivability

    Zia Uddin Chowdhury, Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)

  4. 2026
    conference

    PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference

    Krishna Teja Chitty-Venkata, Jie Ye, Siddhisanket Raskar, Xian-He Sun, Anthony Kougkas, Murali Emani et al. Findings of the Association for Computational Linguistics (EACL 2026)

Citation

@inproceedings{tang2024dayu,
  author = {Tang, Meng and Cernuda, Jaime and Ye, Jie and Guo, Luanzheng and Tallent, Nathan R. and Kougkas, Anthony and Sun, Xian-He},
  booktitle = {Proceedings of the IEEE International Conference on Cluster Computing},
  title = {DaYu: Optimizing Distributed Scientific Workflows by Decoding Dataflow Semantics and Dynamics},
  year = {2024},
  month = sep,
  publisher = {IEEE},
  pages = {357-369},
  doi = {10.1109/CLUSTER59578.2024.00038},
  url = {https://ieeexplore.ieee.org/document/10740817},
}

Related notes

  1. essay

    When is research software shipped?

    A working definition of "shipped" for research software, built around a named state someone else can run, evidence tied to that state, explicit limits, and a continuity signal, with Hermes and DLIO as the worked examples.

    7 min
  2. essay

    We need an Agent500

    Agent leaderboards score answers. Science needs a ranking that runs an agent through a real workflow with real tools and data and measures correctness, cost, I/O efficiency, and provenance together.

    7 min
  3. field note

    Reflecting on SC25

    What I noticed at SC25 in St. Louis, what the group brought, what a year of running the PDSW program committee taught me about how a workshop gets made, and a few notes for people attending their first SC.

    8 min