NIOBE: An intelligent I/O bridging engine for complex and distributed workflows
Abstract
In the age of data-driven computing, integrating High Performance Computing(HPC) and Big Data(BD) environments may be the key to increasing productivity and to driving scientific discovery forward. Scientific workflows consist of diverse applications (i.e., HPC simulations and BD analysis) each with distinct representations of data that introduce a semantic barrier between the two environments. To solve scientific problems at scale, accessing semantically different data from different storage resources is the biggest unsolved challenge. In this work, we aim to address a critical question: ”How can we exploit the existing resources and efficiently provide transparent access to data from/to both environments”. We propose iNtelligent I/O Bridging Engine(NIOBE), a new data integration framework that enables integrated data access for scientific workflows with asynchronous I/O and data aggregation. NIOBE performs the data integration using available I/O resources, in contrast to existing optimizations that ignore the I/O nodes present on the data path. In NIOBE, data access is optimized to consider both the ongoing production and the consumption of the data in the future. Experimental results show that with NIOBE, an integrated scientific workflow can be accelerated by up to 10x when compared to a no-integration baseline and by up to 133% compared to other state-of-the-art integration solutions.
Finding
NIOBE accelerates an integrated scientific workflow by up to 10x over no integration and by up to 133% over other state-of-the-art integration solutions.
System
- 2023–2026
More papers on IRIS
Browse all papers on IRIS- 2022
LuxIO: Intelligent Resource Provisioning and Auto-Configuration for Storage Services
Keith Bateman, Neeraj Rajesh, Jaime Cernuda Garcia, Luke Logan, Jie Ye, … Anthony Kougkas et al. Proceedings of the 29th International Conference on High Performance Computing, Data, and Analytics
- 2022
LabStor: A modular and extensible platform for developing high-performance, customized I/O stacks in userspace
Luke Logan, Jaime Cernuda Garcia, Jay Lofstead, Xian–He Sun, Anthony Kougkas. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
- 2022
Stimulus: Accelerate Data Management for Scientific AI applications in HPC
Hariharan Devarajan, Anthony Kougkas, Huihuo Zheng, Venkatram Vishwanath, Xian-He Sun. Proceedings of the 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing
- 2021
HFlow: A Dynamic and Elastic Multi-Layered I/O Forwarder
Jaime Cernuda, Hariharan Devarajan, Luke Logan, Keith Bateman, Neeraj Rajesh, … Anthony Kougkas et al. Proceedings of the International Conference on Cluster Computing
Citation
@inproceedings{feng2019niobe,
author = {Feng, Kun and Devarajan, Hariharan and Kougkas, Anthony and Sun, Xian-He},
booktitle = {Proceedings of the International Conference on Big Data},
title = {NIOBE: An intelligent {I/O} bridging engine for complex and distributed workflows},
year = {2019},
month = dec,
publisher = {IEEE},
pages = {493--502},
doi = {10.1109/BigData47090.2019.9006363},
url = {https://ieeexplore.ieee.org/abstract/document/9006363},
} Related notes
- 9 min
One destination, N tiers
Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.