akougkas.io

NIOBE: An intelligent I/O bridging engine for complex and distributed workflows

Kun Feng, Hariharan Devarajan, Anthony Kougkas, Xian-He Sun

Proceedings of the International Conference on Big DataIEEE

Abstract

In the age of data-driven computing, integrating High Performance Computing(HPC) and Big Data(BD) environments may be the key to increasing productivity and to driving scientific discovery forward. Scientific workflows consist of diverse applications (i.e., HPC simulations and BD analysis) each with distinct representations of data that introduce a semantic barrier between the two environments. To solve scientific problems at scale, accessing semantically different data from different storage resources is the biggest unsolved challenge. In this work, we aim to address a critical question: ”How can we exploit the existing resources and efficiently provide transparent access to data from/to both environments”. We propose iNtelligent I/O Bridging Engine(NIOBE), a new data integration framework that enables integrated data access for scientific workflows with asynchronous I/O and data aggregation. NIOBE performs the data integration using available I/O resources, in contrast to existing optimizations that ignore the I/O nodes present on the data path. In NIOBE, data access is optimized to consider both the ongoing production and the consumption of the data in the future. Experimental results show that with NIOBE, an integrated scientific workflow can be accelerated by up to 10x when compared to a no-integration baseline and by up to 133% compared to other state-of-the-art integration solutions.

Finding

NIOBE accelerates an integrated scientific workflow by up to 10x over no integration and by up to 133% over other state-of-the-art integration solutions.

System

  1. completed

    IRIS

  2. 2023–2026
    complete

    DTIO

Browse all papers on IRIS
  1. 2022
    conference

    LuxIO: Intelligent Resource Provisioning and Auto-Configuration for Storage Services

    Keith Bateman, Neeraj Rajesh, Jaime Cernuda Garcia, Luke Logan, Jie Ye, … Anthony Kougkas et al. Proceedings of the 29th International Conference on High Performance Computing, Data, and Analytics

  2. 2022
    conference

    LabStor: A modular and extensible platform for developing high-performance, customized I/O stacks in userspace

    Luke Logan, Jaime Cernuda Garcia, Jay Lofstead, Xian–He Sun, Anthony Kougkas. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

  3. 2022
    conference

    Stimulus: Accelerate Data Management for Scientific AI applications in HPC

    Hariharan Devarajan, Anthony Kougkas, Huihuo Zheng, Venkatram Vishwanath, Xian-He Sun. Proceedings of the 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing

  4. 2021
    conference

    HFlow: A Dynamic and Elastic Multi-Layered I/O Forwarder

    Jaime Cernuda, Hariharan Devarajan, Luke Logan, Keith Bateman, Neeraj Rajesh, … Anthony Kougkas et al. Proceedings of the International Conference on Cluster Computing

Citation

@inproceedings{feng2019niobe,
  author = {Feng, Kun and Devarajan, Hariharan and Kougkas, Anthony and Sun, Xian-He},
  booktitle = {Proceedings of the International Conference on Big Data},
  title = {NIOBE: An intelligent {I/O} bridging engine for complex and distributed workflows},
  year = {2019},
  month = dec,
  publisher = {IEEE},
  pages = {493--502},
  doi = {10.1109/BigData47090.2019.9006363},
  url = {https://ieeexplore.ieee.org/abstract/document/9006363},
}

Related notes

  1. essay

    One destination, N tiers

    Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.

    9 min