HFlow: A Dynamic and Elastic Multi-Layered I/O Forwarder
Abstract
Modern applications are highly data-intensive, leading to the well-known I/O bottleneck problem. Scientists have proposed the placement of fast intermediate storage resources which aim to mask the I/O penalties. To manage these resources, three core software abstractions are being used in leadership-class computing facilities: IO Forwarders, Burst Buffers, and Data Stagers. Yet, with the rise of multi-tenant deployment in HPC systems, these software abstractions are: managed and maintained in isolation, leading to inefficient interactions; allocated statically, leading to load imbalance; exclusively bifurcated between the intermediate storage, leading to under-utilization of resources, and, in many cases, do not support in-situ operations. To this end, we present HFlow, a new class of data forwarding system that leverages a real-time data movement paradigm. HFlow introduces a unified data movement abstraction (the ByteFlow) providing data-independent tasks that can be executed anywhere and thus, enabling dynamic resource provisioning. Moreover, the processing elements executing the ByteFlows are designed to be ephemeral and, hence, enable elastic management of intermediate storage resources. Our results show that applications running under HFlow display an increase in performance of 3x when compared with state-of-the-art software solutions.
Finding
HFlow’s unified data movement abstraction, the ByteFlow, gives applications a 3x performance increase over state-of-the-art software solutions.
System
- 2018–2024
More papers on Hermes
Browse all papers on Hermes- 2026
Benchmarking the Performance of Semantic Search Techniques for Agent Discovery
Rajni Pawar, Luke Logan, Xian-He Sun, Anthony Kougkas. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)
- 2024
MegaMmap: Blurring the Boundary Between Memory and Storage for Data-Intensive Workloads
Luke Logan, Anthony Kougkas, Xian-He Sun. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis SC
- 2024
DaYu: Optimizing Distributed Scientific Workflows by Decoding Dataflow Semantics and Dynamics
Meng Tang, Jaime Cernuda, Jie Ye, Luanzheng Guo, Nathan R. Tallent, Anthony Kougkas et al. Proceedings of the IEEE International Conference on Cluster Computing
- 2024
HStream: A hierarchical data streaming engine for high-throughput scientific applications
Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. Proceedings of the 53rd International Conference on Parallel Processing
Citation
@inproceedings{cernuda2021hflow,
author = {Cernuda, Jaime and Devarajan, Hariharan and Logan, Luke and Bateman, Keith and Rajesh, Neeraj and Ye, Jie and Kougkas, Anthony and Sun, Xian-He},
booktitle = {Proceedings of the International Conference on Cluster Computing},
title = {{HFlow}: A Dynamic and Elastic Multi-Layered {I/O} Forwarder},
year = {2021},
month = sep,
publisher = {IEEE},
pages = {114--124},
doi = {10.1109/Cluster48925.2021.00064},
url = {https://ieeexplore.ieee.org/abstract/document/9556010},
} Related notes
- 4 min
Two PhDs, one architecture
Luke Logan and Jaime Cernuda defended in the same semester. Their dissertations are the two halves of the runtime and the agent layer we are building, and they got there by owning the problems, not by executing mine.
- 9 min
One destination, N tiers
Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.
- 3 min
CLIO Core 2.3 and 2.4: when a tier fills or a node dies
Four CLIO Core releases between October 9 and November 2, 2026 added a device-paged GPU vector and Intel GPU support, then taught the context store to recover a full tier and keep data and directories readable when a node goes down.