Characterizing Dataflow for I/O-Aware Scheduling in HPC Workflows
Abstract
Scientific discoveries are increasingly facilitated by data-driven workflows executed in HPC environments, where complex task-to-data exchange patterns cause bottlenecks in shared storage. Existing I/O-aware scheduling techniques use application-oriented metrics that primarily focus on individual task characteristics, leaving workflow-level dataflow behaviors, such as file reuse and producer-consumer I/O patterns, underexplored despite their importance for effective orchestration. This paper presents a workflow-centric methodology to characterize and analyze I/O using workflow-specific patterns and metrics. By correlating workflow structures (e.g., task composition, execution order, and producer-consumer relationships) with I/O metrics (e.g., access type, operation count, and data size), we gain deeper insight into workflow I/O dynamics. We evaluate our approach on six diverse HPC workflows, covering 3,500 task instances and 1,387 unique files, with a total I/O volume of 1.5 TB. This analysis uncovers a spectrum of behaviors related to file reuse, data exchange, and task-level I/O heterogeneity. These insights reveal new opportunities and research directions for I/O-aware scheduling, supporting smarter task placement, data movement, and storage selection in HPC.
Finding
Across six HPC workflows (3,500 task instances, 1,387 unique files, and 1.5 TB of I/O), workflow-centric analysis shows that 1-1 producer-consumer patterns often underutilize bandwidth while n-n structures benefit from higher concurrency.
System
-
PORTUS
A DOE Genesis Mission project led by PNNL that ports scientific workflows to HPC systems and tunes where their tasks and data run.
More papers on PORTUS
Browse all papers on PORTUS- 2026
DataFlowDrs: Automating Performance Optimization of Data Flow Within HPC Workflows
Nathan Tallent, Meng Tang, J. Firoz, Z. Peng, Luanzheng Guo, Anthony Kougkas et al. IEEE Transactions on Parallel and Distributed Systems
- 2026
The Hidden Cost of Storage Mismatches: Why Faster I/O Per Task Can Slow Down Entire Workflows
Meng Tang, Luanzheng Guo, Nathan Tallent, Anthony Kougkas, Xian-He Sun. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)
- 2025
Quantifying AWS S3 I/O Performance Boundaries Using the Roofline Model
Meng Tang, Zhaobin Zhu, Luanzheng Guo, James G. Bandy, Tim Carlson, … Anthony Kougkas et al. The 10th International Parallel Data Systems Workshop
- 2025
Analyzing I/O Patterns and Dataflow in HPC Workflows
Meng Tang, Luanzheng Guo, Nathan Tallent, Anthony Kougkas, Xian-He Sun. The 37h International Conference on Scalable Scientific Data Management (SSDBM 2025)
Citation
@inproceedings{tang2026dataflow,
author = {Tang, M. and Guo, L. and Kougkas, A. and Sun, X.-H. and Tallent, N. R.},
title = {{Characterizing Dataflow for I/O-Aware Scheduling in HPC Workflows}},
booktitle = {40th IEEE International Parallel and Distributed Processing Symposium (IPDPS 2026)},
year = {2026},
month = may,
publisher = {IEEE},
pages = {868--884},
doi = {10.1109/IPDPS65963.2026.00076},
url = {https://doi.org/10.1109/IPDPS65963.2026.00076},
}