LABIOS

2024

1. sc24 MegaMmap: Blurring the Boundary Between Memory and Storage for Data-Intensive Workloads

   Luke Logan, Anthony Kougkas, and Xian-He Sun

   In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis SC , Nov 2024

   ABS BIB Cite

   

   Large-scale data analytics, scientific simulation, and deep learning codes in HPC perform massive computations on data greatly exceeding the bounds of main memory. These out-of-core algorithms suffer from severe data movement penalties, programming complexity, and limited code reuse. To solve this, HPC sites have steadily increased DRAM capacity. However, this is not sustainable due to financial and environmental costs. A more elegant, low-cost, and portable solution is to expand memory to distributed multitiered storage. In this work, we propose MegaMmap: a software distributed shared memory (DSM) that enlarges effective memory capacity through intelligent tiered DRAM and storage management. MegaMmap provides workload-aware data organization, eviction, and prefetching policies to reduce DRAM consumption while ensuring speedy access to critical data. A variety of memory coherence optimizations are provided through an intuitive hinting system. Evaluations show that various workloads can be executed with a fraction of the DRAM while offering competitive performance.

   @inproceedings{logan2024megammap,
     entry_type = {conference},
     author = {Logan, Luke and Kougkas, Anthony and Sun, Xian-He},
     booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis SC},
     title = {MegaMmap: Blurring the Boundary Between Memory and Storage for Data-Intensive Workloads},
     year = {2024},
     month = nov,
     publisher = {IEEE Computer Society},
     volume = {},
     number = {},
     pages = {1725-1742},
     keywords = {HPC, Systems Software, Memory Tiering, Storage Tiering},
     doi = {10.1109/SC41406.2024.00114},
     url = {https://dl.acm.org/doi/abs/10.1109/SC41406.2024.00114},
   }

2. ipdps24 TunIO: An AI-powered Framework for Optimizing HPC I/O

   Neeraj Rajesh, Keith Bateman, Jean Luca Bez, Suren Byna, Anthony Kougkas, and Xian-He Sun

   In Proceedings of the International Parallel and Distributed Processing Symposium , May 2024

   ABS BIB Cite

   

   I/O operations are a known performance bottleneck of HPC applications. To achieve good performance, users often employ an iterative multistage tuning process to find an optimal I/O stack configuration. However, an I/O stack contains multiple layers, such as high-level I/O libraries, I/O middleware, and parallel file systems, and each layer has many parameters. These parameters and layers are entangled and influenced by each other. The tuning process is time-consuming and complex. In this work, we present TunIO, an AI-powered I/O tuning framework that implements several techniques to balance the tuning cost and performance gain, including tuning the high-impact parameters first. Furthermore, TunIO analyzes the application source code to extract its I/O kernel while retaining all statements necessary to perform I/O. It utilizes a smart selection of high-impact configuration parameters of the given tuning objective. Finally, it uses a novel Reinforcement Learning (RL)-driven early stopping mechanism to balance the cost and performance gain. Experimental results show that TunIO leads to a reduction of up to ≈73% in tuning time while achieving the same performance gain when compared to H5Tuner. It achieves a significant performance gain/cost of 208.4 MBps/min (I/O bandwidth for each minute spent in tuning) over existing approaches under our testing.

   @inproceedings{rajesh2024tunio,
     entry_type = {conference},
     author = {Rajesh, Neeraj and Bateman, Keith and Bez, Jean Luca and Byna, Suren and Kougkas, Anthony and Sun, Xian-He},
     booktitle = {Proceedings of the International Parallel and Distributed Processing Symposium},
     title = {TunIO: An AI-powered Framework for Optimizing HPC I/O},
     year = {2024},
     month = may,
     publisher = {IEEE},
     volume = {},
     number = {},
     pages = {494-505},
     keywords = {I/O Performance Optimization, Storage Resource Provisioning, Task-Based I/O, High-Performance Computing},
     doi = {10.1109/IPDPS57955.2024.00050},
     url = {https://ieeexplore.ieee.org/document/10579249},
   }

2022

1. hipc22 LuxIO: Intelligent Resource Provisioning and Auto-Configuration for Storage Services

   Keith Bateman, Neeraj Rajesh, Jaime Cernuda Garcia, Luke Logan, Jie Ye, Stephen Herbein, Anthony Kougkas, and Xian-He Sun

   In Proceedings of the 29th International Conference on High Performance Computing, Data, and Analytics , Dec 2022

   ABS BIB Cite

   

   Storage in HPC is typically a single Remote and Static Storage (RSS) resource. However, applications demonstrate diverse I/O requirements that can be better served by a multi-storage approach. Current practice employs ephemeral storage systems running on either node-local or shared storage resources. Yet, the burden of provisioning and configuring intermediate storage falls solely on the users, while global job schedulers offer little to no support for custom deployments. This lack of support often leads to over- or under-provisioning of resources and poorly configured storage systems. To mitigate this, we present LuxIO, an intelligent storage resource provisioning and auto-configuration service. LuxIO constructs storage deployments configured to best match I/O requirements. LuxIO-tuned storage services show performance improvements up to 2× across common applications and benchmarks, while introducing minimal overhead of 93.40 ms on top of existing job scheduling pipelines. LuxIO improves resource utilization by up to 25% in select workflows.

   @inproceedings{bateman2022luxio,
     entry_type = {conference},
     author = {Bateman, Keith and Rajesh, Neeraj and Garcia, Jaime Cernuda and Logan, Luke and Ye, Jie and Herbein, Stephen and Kougkas, Anthony and Sun, Xian-He},
     booktitle = {Proceedings of the 29th International Conference on High Performance Computing, Data, and Analytics},
     title = {LuxIO: Intelligent Resource Provisioning and Auto-Configuration for Storage Services},
     year = {2022},
     month = dec,
     publisher = {IEEE},
     volume = {},
     number = {},
     pages = {246--255},
     keywords = {Storage Resource Provisioning, Data Management in HPC, Elastic Storage, Workflow Optimization},
     doi = {10.1109/HiPC56025.2022.00041},
     url = {https://ieeexplore.ieee.org/abstract/document/10106285},
   }

2. sc22 LabStor: A modular and extensible platform for developing high-performance, customized I/O stacks in userspace

   Luke Logan, Jaime Cernuda Garcia, Jay Lofstead, Xian–He Sun, and Anthony Kougkas

   In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , Nov 2022

   ABS BIB Cite

   

   Traditionally, I/O systems have been developed within the confines of a centralized OS kernel. This led to monolithic and rigid storage systems that are limited by low development speed, expressiveness, and performance. Various assumptions are imposed including reliance on the UNIX-file abstraction, the POSIX standard, and a narrow set of I/O policies. However, this monolithic design philosophy makes it difficult to develop and deploy new I/O approaches to satisfy the rapidly-evolving I/O requirements of modern scientific applications. To this end, we propose LabStor: a modular and extensible platform for developing high-performance, customized I/O stacks. Single-purpose I/O modules (e.g, I/O schedulers) can be developed in the comfort of userspace and released as plug-ins, while end-users can compose these modules to form workload- and hardware-specific I/O stacks. Evaluations show that by switching to a fully modular design, tailored I/O stacks can yield performance improvements of up to 60% in various applications.

   @inproceedings{logan2022labstor,
     entry_type = {conference},
     author = {Logan, Luke and Garcia, Jaime Cernuda and Lofstead, Jay and Sun, Xian--He and Kougkas, Anthony},
     booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
     title = {LabStor: A modular and extensible platform for developing high-performance, customized I/O stacks in userspace},
     year = {2022},
     month = nov,
     publisher = {ACM},
     volume = {},
     number = {},
     pages = {1--15},
     keywords = {Storage Bridging, Elastic Storage, I/O Acceleration, Task-Based I/O},
     doi = {10.1109/SC41404.2022.00028},
     url = {https://ieeexplore.ieee.org/abstract/document/10046077},
   }

2021

1. cluster21 pMEMCPY: a simple, lightweight, and portable I/O library for storing data in persistent memory

   Luke Logan, Jay Lofstead, Scott Levy, Patrick Widener, Xian-He Sun, and Anthony Kougkas

   In Proceedings of the International Conference on Cluster Computing , Sep 2021

   ABS BIB Cite

   

   Persistent memory (PMEM) devices can achieve comparable performance to DRAM while providing significantly more capacity. This has made the technology compelling as an expansion to main memory. Rethinking PMEM as storage devices can offer a high performance buffering layer for HPC applications to temporarily, but safely store data. However, modern parallel I/O libraries, such as HDF5 and pNetCDF, are complicated and introduce significant software and metadata overheads when persisting data to these storage devices, wasting much of their potential. In this work, we explore the potential of PMEM as storage through pMEMCPY: a simple, lightweight, and portable I/O library for storing data in persistent memory. We demonstrate that our approach is up to 2x faster than other popular parallel I/O libraries under real workloads.

   @inproceedings{logan2021pmemcpy,
     entry_type = {workshop},
     author = {Logan, Luke and Lofstead, Jay and Levy, Scott and Widener, Patrick and Sun, Xian-He and Kougkas, Anthony},
     booktitle = {Proceedings of the International Conference on Cluster Computing},
     title = {pMEMCPY: a simple, lightweight, and portable I/O library for storing data in persistent memory},
     year = {2021},
     month = sep,
     publisher = {IEEE},
     volume = {},
     number = {},
     pages = {664--670},
     keywords = {Computational Storage, I/O Acceleration, Storage Architectures, Data Movement Optimization},
     doi = {10.1109/Cluster48925.2021.00098},
     url = {https://ieeexplore.ieee.org/abstract/document/9555982},
   }

2. hpdc21 Apollo: An ML-assisted real-time storage resource observer

   Neeraj Rajesh, Hariharan Devarajan, Jaime Cernuda Garcia, Keith Bateman, Luke Logan, Jie Ye, Anthony Kougkas, and Xian-He Sun

   In Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing , Jun 2021

   ABS BIB Cite

   

   Applications and middleware services, such as data placement engines, I/O scheduling, and prefetching engines, require low-latency access to telemetry data in order to make optimal decisions. However, typical monitoring services store their telemetry data in a database in order to allow applications to query them, resulting in significant latency penalties. This work presents Apollo: a low-latency monitoring service that aims to provide applications and middleware libraries with direct access to relational telemetry data. Monitoring the system can create interference and overhead, slowing down raw performance of the resources for the job. However, having a current view of the system can aid middleware services in making more optimal decisions which can ultimately improve the overall performance. Apollo has been designed from the ground up to provide low latency, using Publish-Subscriber Pub-Sub semantics, and low overhead, using adaptive intervals in order to change the length of time between polling the resource for telemetry data and machine learning in order to predict changes to the telemetry data between actual resource polling. This work also provides some high level abstractions called I/O curators, which can further aid middleware libraries and applications to make optimal decisions. Evaluations showcase that Apollo can achieve sub-millisecond latency for acquiring complex insights with a memory overhead of  57 MB and CPU overhead being only 7% more than existing state-of-the-art systems.

   @inproceedings{rajesh2021apollo,
     entry_type = {conference},
     author = {Rajesh, Neeraj and Devarajan, Hariharan and Garcia, Jaime Cernuda and Bateman, Keith and Logan, Luke and Ye, Jie and Kougkas, Anthony and Sun, Xian-He},
     booktitle = {Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing},
     title = {Apollo: An ML-assisted real-time storage resource observer},
     year = {2021},
     month = jun,
     publisher = {ACM},
     volume = {},
     number = {},
     pages = {147--159},
     keywords = {Storage Resource Provisioning, I/O Profiling, Data Management in HPC, High-Performance Computing},
     doi = {10.1145/3431379.3460640},
     url = {https://dl.acm.org/doi/abs/10.1145/3431379.3460640},
   }

2020

1. tos20 Bridging Storage Semantics Using Data Labels and Asynchronous I/O

   Anthony Kougkas, Hariharan Devarajan, and Xian-He Sun

   In Proceedings of the ACM Transactions on Storage , Oct 2020

   ABS BIB Cite

   

   In the era of data-intensive computing, large-scale applications, in both scientific and the BigData communities, demonstrate unique I/O requirements leading to a proliferation of different storage devices and software stacks, many of which have conflicting requirements. Further, new hardware technologies and system designs create a hierarchical composition that may be ideal for computational storage operations. In this article, we investigate how to support a wide variety of conflicting I/O workloads under a single storage system. We introduce the idea of a Label, a new data representation, and, we present LABIOS: a new, distributed, Label- based I/O system. LABIOS boosts I/O performance by up to 17× via asynchronous I/O, supports heterogeneous storage resources, offers storage elasticity, and promotes in situ analytics and software defined storage support via data provisioning. LABIOS demonstrates the effectiveness of storage bridging to support the convergence of HPC and BigData workloads on a single platform.

   @inproceedings{kougkas2020bridging,
     entry_type = {journal},
     author = {Kougkas, Anthony and Devarajan, Hariharan and Sun, Xian-He},
     booktitle = {Proceedings of the ACM Transactions on Storage},
     title = {Bridging Storage Semantics Using Data Labels and Asynchronous I/O},
     year = {2020},
     month = oct,
     publisher = {ACM},
     volume = {16},
     number = {4},
     article = {22},
     pages = {1--34},
     keywords = {Data Labels, Storage Bridging, Task-Based I/O, Elastic Storage},
     doi = {10.1145/3415579},
     issn = {1553-3077},
     url = {https://dl.acm.org/doi/abs/10.1145/3415579},
   }

2019

1. hpdc19 Labios: A distributed label-based i/o system

   Anthony Kougkas, Hariharan Devarajan, Jay Lofstead, and Xian-He Sun

   In Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing. ║ Best Paper Award ║ , Jun 2019

   ABS BIB Cite

   

   Karsten Schwan Best Paper Award at HPDC’2019. Photo here

   In the era of data-intensive computing, large-scale applications, in both scientific and the BigData communities, demonstrate unique I/O requirements leading to a proliferation of different storage devices and software stacks, many of which have conflicting requirements. In this paper, we investigate how to support a wide variety of conflicting I/O workloads under a single storage system. We introduce the idea of a Label, a new data representation, and, we present LABIOS: a new, distributed, Label- based I/O system. LABIOS boosts I/O performance by up to 17x via asynchronous I/O, supports heterogeneous storage resources, offers storage elasticity, and promotes in-situ analytics via data provisioning. LABIOS demonstrates the effectiveness of storage bridging to support the convergence of HPC and BigData workloads on a single platform.

   @inproceedings{kougkas2019labios,
     entry_type = {conference},
     author = {Kougkas, Anthony and Devarajan, Hariharan and Lofstead, Jay and Sun, Xian-He},
     booktitle = {Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing.},
     title = {Labios: A distributed label-based i/o system},
     year = {2019},
     month = jun,
     publisher = {ACM},
     volume = {},
     number = {},
     pages = {13--24},
     keywords = {Data Labels, Task-Based I/O, Elastic Storage, Storage Bridging},
     doi = {10.1145/3307681.3325405},
     url = {https://dl.acm.org/doi/abs/10.1145/3307681.3325405},
   }

2. ccgrid19 An intelligent, adaptive, and flexible data compression framework

   Hariharan Devarajan, Anthony Kougkas, and Xian-He Sun

   In Proceedings of the 19th International Symposium on Cluster, Cloud and Grid Computing , May 2019

   ABS BIB Cite

   

   The data explosion phenomenon in modern applications causes tremendous stress on storage systems. Developers use data compression, a size-reduction technique, to address this issue. However, each compression library exhibits different strengths and weaknesses when considering the input data entry_type and format. We present Ares, an intelligent, adaptive, and flexible compression framework which can dynamically choose a compression library for a given input data based on the entry_type of the workload and provides an appropriate infrastructure to users to fine-tune the chosen library. Ares is a modular framework which unifies several compression libraries while allowing the addition of more compression libraries by the user. Ares is a unified compression engine that abstracts the complexity of using different compression libraries for each workload. Evaluation results show that under real-world applications, from both scientific and Cloud domains, Ares performed 2-6x faster than competitive solutions with a low cost of additional data analysis (i.e., overheads around 10%) and up to 10x faster against a baseline of no compression at all.

   @inproceedings{devarajan2019intelligent,
     entry_type = {conference},
     author = {Devarajan, Hariharan and Kougkas, Anthony and Sun, Xian-He},
     booktitle = {Proceedings of the 19th International Symposium on Cluster, Cloud and Grid Computing},
     title = {An intelligent, adaptive, and flexible data compression framework},
     year = {2019},
     month = may,
     publisher = {IEEE},
     volume = {},
     number = {},
     pages = {82--91},
     keywords = {Data Compression Techniques, Data Management in HPC, I/O Acceleration, Storage Resource Provisioning},
     doi = {10.1109/CCGRID.2019.00019},
     url = {https://ieeexplore.ieee.org/abstract/document/8752926},
   }

2018

1. hipc18 Vidya: Performing code-block I/O characterization for data access optimization

   Hariharan Devarajan, Anthony Kougkas, Prajwal Challa, and Xian-He Sun

   In Proceedings of the 25th International Conference on High Performance Computing , Dec 2018

   ABS BIB Cite

   

   Understanding, characterizing and tuning scientific applications’ I/O behavior is an increasingly complicated process in HPC systems. Existing tools use either offline profiling or online analysis to get insights into the applications’ I/O patterns. However, there is lack of a clear formula to characterize applications’ I/O. Moreover, these tools are application specific and do not account for multi-tenant systems. This paper presents Vidya, an I/O profiling framework which can predict application’s I/O intensity using a new formula called Code-Block I/O Characterization (CIOC). Using CIOC, developers and system architects can tune an application’s I/O behavior and better match the underlying storage system to maximize performance. Evaluation results show that Vidya can predict an application’s I/O intensity with a variance of 0.05%. Vidya can profile applications with a high accuracy of 98% while reducing profiling time by 9x. We further show how Vidya can optimize an application’s I/O time by 3.7x.

   @inproceedings{devarajan2018vidya,
     entry_type = {conference},
     author = {Devarajan, Hariharan and Kougkas, Anthony and Challa, Prajwal and Sun, Xian-He},
     booktitle = {Proceedings of the 25th International Conference on High Performance Computing},
     title = {Vidya: Performing code-block I/O characterization for data access optimization},
     year = {2018},
     month = dec,
     publisher = {IEEE},
     volume = {},
     number = {},
     pages = {255--264},
     keywords = {I/O Profiling, Data Management in HPC, I/O Performance Optimization, Task-Based I/O},
     doi = {10.1109/HiPC.2018.00036},
     url = {https://ieeexplore.ieee.org/abstract/document/8638067},
   }