akougkas.io

Characterizing Dataflow for I/O-Aware Scheduling in HPC Workflows

Meng Tang, Luanzheng Guo, Anthony Kougkas, Xian-He Sun, Nathan Tallent

40th IEEE International Parallel and Distributed Processing Symposium (IPDPS 2026)IEEEPublished

Abstract

Scientific discoveries are increasingly facilitated by data-driven workflows executed in HPC environments, where complex task-to-data exchange patterns cause bottlenecks in shared storage. Existing I/O-aware scheduling techniques use application-oriented metrics that primarily focus on individual task characteristics, leaving workflow-level dataflow behaviors, such as file reuse and producer-consumer I/O patterns, underexplored despite their importance for effective orchestration. This paper presents a workflow-centric methodology to characterize and analyze I/O using workflow-specific patterns and metrics. By correlating workflow structures (e.g., task composition, execution order, and producer-consumer relationships) with I/O metrics (e.g., access type, operation count, and data size), we gain deeper insight into workflow I/O dynamics. We evaluate our approach on six diverse HPC workflows, covering 3,500 task instances and 1,387 unique files, with a total I/O volume of 1.5 TB. This analysis uncovers a spectrum of behaviors related to file reuse, data exchange, and task-level I/O heterogeneity. These insights reveal new opportunities and research directions for I/O-aware scheduling, supporting smarter task placement, data movement, and storage selection in HPC.

Finding

Across six HPC workflows (3,500 task instances, 1,387 unique files, and 1.5 TB of I/O), workflow-centric analysis shows that 1-1 producer-consumer patterns often underutilize bandwidth while n-n structures benefit from higher concurrency.

System

  1. experimental

    PORTUS

    A DOE Genesis Mission project led by PNNL that ports scientific workflows to HPC systems and tunes where their tasks and data run.

Browse all papers on PORTUS
  1. 2026
    journal

    DataFlowDrs: Automating Performance Optimization of Data Flow Within HPC Workflows

    Nathan Tallent, Meng Tang, J. Firoz, Z. Peng, Luanzheng Guo, Anthony Kougkas et al. IEEE Transactions on Parallel and Distributed Systems

  2. 2026
    poster

    The Hidden Cost of Storage Mismatches: Why Faster I/O Per Task Can Slow Down Entire Workflows

    Meng Tang, Luanzheng Guo, Nathan Tallent, Anthony Kougkas, Xian-He Sun. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)

  3. 2025
    workshop

    Quantifying AWS S3 I/O Performance Boundaries Using the Roofline Model

    Meng Tang, Zhaobin Zhu, Luanzheng Guo, James G. Bandy, Tim Carlson, … Anthony Kougkas et al. The 10th International Parallel Data Systems Workshop

  4. 2025
    poster

    Analyzing I/O Patterns and Dataflow in HPC Workflows

    Meng Tang, Luanzheng Guo, Nathan Tallent, Anthony Kougkas, Xian-He Sun. The 37h International Conference on Scalable Scientific Data Management (SSDBM 2025)

Citation

@inproceedings{tang2026dataflow,
  author = {Tang, M. and Guo, L. and Kougkas, A. and Sun, X.-H. and Tallent, N. R.},
  title = {{Characterizing Dataflow for I/O-Aware Scheduling in HPC Workflows}},
  booktitle = {40th IEEE International Parallel and Distributed Processing Symposium (IPDPS 2026)},
  year = {2026},
  month = may,
  publisher = {IEEE},
  pages = {868--884},
  doi = {10.1109/IPDPS65963.2026.00076},
  url = {https://doi.org/10.1109/IPDPS65963.2026.00076},
}