Hermes

2024

1. cluster24 DaYu: Optimizing Distributed Scientific Workflows by Decoding Dataflow Semantics and Dynamics

   Meng Tang, Jaime Cernuda, Jie Ye, Luanzheng Guo, Nathan R. Tallent, Anthony Kougkas, and Xian-He Sun

   In Proceedings of the IEEE International Conference on Cluster Computing , Sep 2024

   ABS BIB Cite

   

   The combination of ever-growing scientific datasets and distributed workflow complexity creates I/O performance bottlenecks due to data volume, velocity, and variety. Although the increasing use of descriptive data formats (e.g., HDF5, netCDF) helps organize these datasets, it also introduces obscure bottlenecks due to the need to translate high-level operations into file addresses and then into low-level I/O operations. To address this challenge, we introduce DaYu, a method and toolset for analyzing (a) semantic relationships between logical datasets and file addresses, (b) how dataset operations translate into I/O, and (c) the combination across entire workflows. DaYu’s analysis and visualization enable the identification of critical bottlenecks and the reasoning about remediation. We describe our methodology and propose optimization guidelines. Evaluation on scientific workflows demonstrates up to a 3.7x performance improvement in I/O time for obscure bottlenecks. The time and storage overhead for DaYu’s time-ordered data are typically under 0.2% of runtime and 0.25% of data volume, respectively.

   @inproceedings{tang2024dayu,
     entry_type = {conference},
     author = {Tang, Meng and Cernuda, Jaime and Ye, Jie and Guo, Luanzheng and Tallent, Nathan R. and Kougkas, Anthony and Sun, Xian-He},
     booktitle = {Proceedings of the IEEE International Conference on Cluster Computing},
     title = {DaYu: Optimizing Distributed Scientific Workflows by Decoding Dataflow Semantics and Dynamics},
     year = {2024},
     month = sep,
     publisher = {IEEE},
     volume = {},
     number = {},
     pages = {357-369},
     keywords = {Workflow Optimization, Data Layout Optimization, In-Situ Analytics, Data-Intensive Applications},
     doi = {10.1109/CLUSTER59578.2024.00038},
     url = {https://ieeexplore.ieee.org/document/10740817},
   }

2. icpp24 HStream: A hierarchical data streaming engine for high-throughput scientific applications

   Jaime Cernuda, Jie Ye, Anthony Kougkas, and Xian-He Sun

   In Proceedings of the 53rd International Conference on Parallel Processing , Aug 2024

   ABS BIB Cite

   

   Data streaming is gaining traction in high-performance computing (HPC) as a mechanism for continuous data transfer, but remains underutilized as a processing paradigm due to the inadequacy of existing technologies, which are primarily designed for cloud architectures and ill-equipped to tackle HPC-specific challenges. This work introduces HStream, a novel data management design for out-of-core data streaming engines. Central to the HStream design is the separation of data and computing planes at the task level. By managing them independently, issues such as memory thrashing and back-pressure, caused by the high volume, velocity, and burstiness of I/O in HPC environments, can be effectively addressed at runtime. Specifically, HStream utilizes adaptive parallelism and hierarchical memory management, enabled by this design paradigm, to alleviate memory pressure and enhance system performance. These improvements enable HStream to match the performance of state-of-the-art HPC streaming engines and achieve up to a 1.5x reduction in latency under high data loads.

   @inproceedings{cernuda2024hstream,
     entry_type = {conference},
     author = {Cernuda, Jaime and Ye, Jie and Kougkas, Anthony and Sun, Xian-He},
     booktitle = {Proceedings of the 53rd International Conference on Parallel Processing},
     title = {HStream: A hierarchical data streaming engine for high-throughput scientific applications},
     year = {2024},
     month = aug,
     publisher = {ACM},
     volume = {},
     number = {},
     pages = {231-240},
     keywords = {Data Movement Optimization, Elastic Storage, Data Integration Frameworks, Hierarchical Buffering},
     doi = {10.1145/3673038.3673150},
     url = {https://dl.acm.org/doi/abs/10.1145/3673038.3673150},
   }

2023

1. sc23 Data Flow Lifecycles for Optimizing Workflow Coordination

   Hyungro Lee, Luanzheng Guo, Meng Tang, Jesun Firoz, Nathan Tallent, Anthony Kougkas, and Xian-He Sun

   In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , Nov 2023

   ABS BIB Cite

   

   A critical performance challenge in distributed scientific workflows is coordinating tasks and data flows on distributed resources. To guide these decisions, this paper introduces data flow lifecycle analysis. Workflows are commonly represented using directed acyclic graphs (DAGs). Data flow lifecycles (DFL) enrich task DAGs with data objects and properties that describe data flow and how tasks interact with that flow. Lifecycles enable analysis from several important perspectives: task, data, and data flow. We describe representation, measurement, analysis, visualization, and opportunity identification for DFLs. Our measurement is both distributed and scalable, using space that is constant per data file. We use lifecycles and opportunity analysis to reason about improved task placement and reduced data movement for five scientific workflows with different characteristics. Case studies show improvements of 15×, 1.9×, and 10–30×. Our work is implemented in the DataLife tool.

   @inproceedings{lee2023data,
     entry_type = {conference},
     author = {Lee, Hyungro and Guo, Luanzheng and Tang, Meng and Firoz, Jesun and Tallent, Nathan and Kougkas, Anthony and Sun, Xian-He},
     booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
     title = {Data Flow Lifecycles for Optimizing Workflow Coordination},
     year = {2023},
     month = nov,
     publisher = {ACM},
     volume = {},
     number = {},
     pages = {1--15},
     keywords = {Workflow Optimization, Data Movement Optimization, In-Situ Analytics, Data-Intensive Applications},
     doi = {10.1145/3581784.36071},
     url = {https://dl.acm.org/doi/abs/10.1145/3581784.3607104},
   }

2022

1. hipc22 LuxIO: Intelligent Resource Provisioning and Auto-Configuration for Storage Services

   Keith Bateman, Neeraj Rajesh, Jaime Cernuda Garcia, Luke Logan, Jie Ye, Stephen Herbein, Anthony Kougkas, and Xian-He Sun

   In Proceedings of the 29th International Conference on High Performance Computing, Data, and Analytics , Dec 2022

   ABS BIB Cite

   

   Storage in HPC is typically a single Remote and Static Storage (RSS) resource. However, applications demonstrate diverse I/O requirements that can be better served by a multi-storage approach. Current practice employs ephemeral storage systems running on either node-local or shared storage resources. Yet, the burden of provisioning and configuring intermediate storage falls solely on the users, while global job schedulers offer little to no support for custom deployments. This lack of support often leads to over- or under-provisioning of resources and poorly configured storage systems. To mitigate this, we present LuxIO, an intelligent storage resource provisioning and auto-configuration service. LuxIO constructs storage deployments configured to best match I/O requirements. LuxIO-tuned storage services show performance improvements up to 2× across common applications and benchmarks, while introducing minimal overhead of 93.40 ms on top of existing job scheduling pipelines. LuxIO improves resource utilization by up to 25% in select workflows.

   @inproceedings{bateman2022luxio,
     entry_type = {conference},
     author = {Bateman, Keith and Rajesh, Neeraj and Garcia, Jaime Cernuda and Logan, Luke and Ye, Jie and Herbein, Stephen and Kougkas, Anthony and Sun, Xian-He},
     booktitle = {Proceedings of the 29th International Conference on High Performance Computing, Data, and Analytics},
     title = {LuxIO: Intelligent Resource Provisioning and Auto-Configuration for Storage Services},
     year = {2022},
     month = dec,
     publisher = {IEEE},
     volume = {},
     number = {},
     pages = {246--255},
     keywords = {Storage Resource Provisioning, Data Management in HPC, Elastic Storage, Workflow Optimization},
     doi = {10.1109/HiPC56025.2022.00041},
     url = {https://ieeexplore.ieee.org/abstract/document/10106285},
   }

2. ccgrid22 Stimulus: Accelerate Data Management for Scientific AI applications in HPC

   Hariharan Devarajan, Anthony Kougkas, Huihuo Zheng, Venkatram Vishwanath, and Xian-He Sun

   In Proceedings of the 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing , May 2022

   ABS BIB Cite

   

   Modern scientific workflows couple simulations with AI-powered analytics by frequently exchanging data to accelerate time-to-science to reduce the complexity of the simulation planes. However, this data exchange is limited in performance and portability due to a lack of support for scientific data formats in AI frameworks. We need a cohesive mechanism to effectively integrate at scale complex scientific data formats such as HDF5, PnetCDF, ADIOS2, GNCF, and Silo into popular AI frameworks such as TensorFlow, PyTorch, and Caffe. To this end, we designed Stimulus, a data management library for ingesting scientific data effectively into the popular AI frameworks. We utilize the StimOps functions along with StimPack abstraction to enable the integration of scientific data formats with any AI framework. The evaluations show that Stimulus outperforms several large-scale applications with different use-cases such as Cosmic Tagger (consuming HDF5 dataset in PyTorch), Distributed FFN (consuming HDF5 dataset in TensorFlow), and CosmoFlow (converting HDF5 into TFRecord and then consuming that in TensorFlow) by 5.3 x, 2.9 x, and 1.9 x respectively with ideal I/O scalability up to 768 GPUs on the Summit supercomputer. Through Stimulus, we can portably extend existing popular AI frameworks to cohesively support any complex scientific data format and efficiently scale the applications on large-scale supercomputers.

   @inproceedings{devarajan2022stimulus,
     entry_type = {conference},
     author = {Devarajan, Hariharan and Kougkas, Anthony and Zheng, Huihuo and Vishwanath, Venkatram and Sun, Xian-He},
     booktitle = {Proceedings of the 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing},
     title = {Stimulus: Accelerate Data Management for Scientific AI applications in HPC},
     year = {2022},
     month = may,
     publisher = {IEEE},
     volume = {},
     number = {},
     pages = {109--118},
     keywords = {Deep Learning I/O, Data Integration Frameworks, Workflow Optimization, In-Situ Analytics},
     doi = {10.1109/CCGrid54584.2022.00020},
     url = {https://ieeexplore.ieee.org/abstract/document/9826104},
   }

2021

1. cluster21 Hflow: A dynamic and elastic multi-layered i/o forwarder

   Jaime Cernuda, Hariharan Devarajan, Luke Logan, Keith Bateman, Neeraj Rajesh, Jie Ye, Anthony Kougkas, and Xian-He Sun

   In Proceedings of the International Conference on Cluster Computing , Sep 2021

   ABS BIB Cite

   

   Modern applications are highly data-intensive, leading to the well-known I/O bottleneck problem. Scientists have proposed the placement of fast intermediate storage resources which aim to mask the I/O penalties. To manage these resources, three core software abstractions are being used in leadership-class computing facilities: IO Forwarders, Burst Buffers, and Data Stagers. Yet, with the rise of multi-tenant deployment in HPC systems, these software abstractions are: managed and maintained in isolation, leading to inefficient interactions; allocated statically, leading to load imbalance; exclusively bifurcated between the intermediate storage, leading to under-utilization of resources, and, in many cases, do not support in-situ operations. To this end, we present HFlow, a new class of data forwarding system that leverages a real-time data movement paradigm. HFlow introduces a unified data movement abstraction (the ByteFlow) providing data-independent tasks that can be executed anywhere and thus, enabling dynamic resource provisioning. Moreover, the processing elements executing the ByteFlows are designed to be ephemeral and, hence, enable elastic management of intermediate storage resources. Our results show that applications running under HFlow display an increase in performance of 3x when compared with state-of-the-art software solutions.

   @inproceedings{cernuda2021hflow,
     entry_type = {conference},
     author = {Cernuda, Jaime and Devarajan, Hariharan and Logan, Luke and Bateman, Keith and Rajesh, Neeraj and Ye, Jie and Kougkas, Anthony and Sun, Xian-He},
     booktitle = {Proceedings of the International Conference on Cluster Computing},
     title = {Hflow: A dynamic and elastic multi-layered i/o forwarder},
     year = {2021},
     month = sep,
     publisher = {IEEE},
     volume = {},
     number = {},
     pages = {114--124},
     kkeywords = {Data Movement Optimization, I/O Acceleration, Elastic Storage, Data Integration Frameworks},
     doi = {10.1109/Cluster48925.2021.00064},
     url = {https://ieeexplore.ieee.org/abstract/document/9556010},
   }

2. hpdc21 Apollo: An ML-assisted real-time storage resource observer

   Neeraj Rajesh, Hariharan Devarajan, Jaime Cernuda Garcia, Keith Bateman, Luke Logan, Jie Ye, Anthony Kougkas, and Xian-He Sun

   In Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing , Jun 2021

   ABS BIB Cite

   

   Applications and middleware services, such as data placement engines, I/O scheduling, and prefetching engines, require low-latency access to telemetry data in order to make optimal decisions. However, typical monitoring services store their telemetry data in a database in order to allow applications to query them, resulting in significant latency penalties. This work presents Apollo: a low-latency monitoring service that aims to provide applications and middleware libraries with direct access to relational telemetry data. Monitoring the system can create interference and overhead, slowing down raw performance of the resources for the job. However, having a current view of the system can aid middleware services in making more optimal decisions which can ultimately improve the overall performance. Apollo has been designed from the ground up to provide low latency, using Publish-Subscriber Pub-Sub semantics, and low overhead, using adaptive intervals in order to change the length of time between polling the resource for telemetry data and machine learning in order to predict changes to the telemetry data between actual resource polling. This work also provides some high level abstractions called I/O curators, which can further aid middleware libraries and applications to make optimal decisions. Evaluations showcase that Apollo can achieve sub-millisecond latency for acquiring complex insights with a memory overhead of  57 MB and CPU overhead being only 7% more than existing state-of-the-art systems.

   @inproceedings{rajesh2021apollo,
     entry_type = {conference},
     author = {Rajesh, Neeraj and Devarajan, Hariharan and Garcia, Jaime Cernuda and Bateman, Keith and Logan, Luke and Ye, Jie and Kougkas, Anthony and Sun, Xian-He},
     booktitle = {Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing},
     title = {Apollo: An ML-assisted real-time storage resource observer},
     year = {2021},
     month = jun,
     publisher = {ACM},
     volume = {},
     number = {},
     pages = {147--159},
     keywords = {Storage Resource Provisioning, I/O Profiling, Data Management in HPC, High-Performance Computing},
     doi = {10.1145/3431379.3460640},
     url = {https://dl.acm.org/doi/abs/10.1145/3431379.3460640},
   }

2020

1. bigdata20 Hreplica: a dynamic data replication engine with adaptive compression for multi-tiered storage

   Hariharan Devarajan, Anthony Kougkas, and Xian-He Sun

   In Proceedings of the International Conference on Big Data , Dec 2020

   ABS BIB Cite

   

   As the diversity of big data applications increases, their requirements diverge and often conflict with one other. Managing this diversity in any supercomputer or data center is a major challenge for system designers. Data replication is a popular approach to meet several of these requirements, such as low latency, read availability, durability, etc. This approach can be enhanced using new modern heterogeneous hardware and software techniques such as data compression. However, both these enhancements work in isolation to the detriment of both. In this work, we present HReplica: a dynamic data replication engine which harmoniously leverages data compression and hierarchical storage to increase the effectiveness of data replication. We have developed a novel dynamic selection algorithm that facilitates the optimal matching of replication schemes, compression libraries, and tiered storage. Our evaluation shows that HReplica can improve scientific and cloud application performance by 5.2x when compared to other state-of-the-art replication schemes.

   @inproceedings{devarajan2020hreplica,
     entry_type = {conference},
     author = {Devarajan, Hariharan and Kougkas, Anthony and Sun, Xian-He},
     booktitle = {Proceedings of the International Conference on Big Data},
     title = {Hreplica: a dynamic data replication engine with adaptive compression for multi-tiered storage},
     year = {2020},
     month = dec,
     publisher = {IEEE},
     volume = {},
     number = {},
     pages = {256--265},
     keywords = {Data Replication, Data Compression Techniques, Multi-Tiered Storage Hierarchy, I/O Acceleration},
     doi = {10.1109/BigData50022.2020.9378167},
     url = {https://ieeexplore.ieee.org/abstract/document/9378167},
   }

2. cluster20 Hcl: Distributing parallel data structures in extreme scales

   Hariharan Devarajan, Anthony Kougkas, Keith Bateman, and Xian-He Sun

   In Proceedings of the International Conference on Cluster Computing , Sep 2020

   ABS BIB Cite

   

   Most parallel programs use irregular control flow and data structures, which are perfect for one-sided communication paradigms such as MPI or PGAS programming languages. However, these environments lack the presence of efficient function-based application libraries that can utilize popular communication fabrics such as TCP, Infinity Band (IB), and RDMA over Converged Ethernet (RoCE). Additionally, there is a lack of high-performance data structure interfaces. We present Hermes Container Library (HCL), a high-performance distributed data structures library that offers high-level abstractions including hash-maps, sets, and queues. HCL uses a RPC over RDMA technology that implements a novel procedural programming paradigm. In this paper, we argue a RPC over RDMA technology can serve as a high-performance, flexible, and co-ordination free backend for implementing complex data structures. Evaluation results from testing real workloads shows that HCL programs are 2x to 12x faster compared to BCL, a state-of-the-art distributed data structure library.

   @inproceedings{devarajan2020hcl,
     entry_type = {conference},
     author = {Devarajan, Hariharan and Kougkas, Anthony and Bateman, Keith and Sun, Xian-He},
     booktitle = {Proceedings of the International Conference on Cluster Computing},
     title = {Hcl: Distributing parallel data structures in extreme scales},
     year = {2020},
     month = sep,
     publisher = {IEEE},
     volume = {},
     number = {},
     pages = {248--258},
     keywords = {Distributed Data Structures, Parallel I/O Optimization, Storage Architectures, High-Performance Computing},
     doi = {10.1109/CLUSTER49012.2020.00035},
     url = {https://ieeexplore.ieee.org/abstract/document/9229595},
   }

3. ipdps20.1 Hfetch: Hierarchical data prefetching for scientific workflows in multi-tiered storage environments

   Hariharan Devarajan, Anthony Kougkas, and Xian-He Sun

   In Proceedings of the International Parallel and Distributed Processing Symposium , Jul 2020

   ABS BIB Cite

   

   In the era of data-intensive computing, accessing data with a high-throughput and low-latency is more imperative than ever. Data prefetching is a well-known technique for hiding read latency. However, existing solutions do not consider the new deep memory and storage hierarchy and also suffer from under-utilization of prefetching resources and unnecessary evictions. Additionally, existing approaches implement a client-pull model where understanding the application’s I/O behavior drives prefetching decisions. Moving towards exascale, where machines run multiple applications concurrently by accessing files in a workflow, a more data-centric approach can resolve challenges such as cache pollution and redundancy. In this study, we present HFetch, a truly hierarchical data prefetcher that adopts a server-push approach to data prefetching. We demonstrate the benefits of such an approach. Results show 10-35% performance gains over existing prefetchers and over 50% when compared to systems with no prefetching.

   @inproceedings{devarajan2020hfetch,
     entry_type = {conference},
     author = {Devarajan, Hariharan and Kougkas, Anthony and Sun, Xian-He},
     booktitle = {Proceedings of the International Parallel and Distributed Processing Symposium},
     title = {Hfetch: Hierarchical data prefetching for scientific workflows in multi-tiered storage environments},
     year = {2020},
     month = jul,
     publisher = {IEEE},
     volume = {},
     number = {},
     pages = {62--72},
     kkeywords = {Hierarchical Data Prefetching, Data Prefetching, Multi-Tiered Storage Hierarchy, Data Movement Optimization},
     doi = {0.1109/IPDPS47924.2020.00017},
     url = {https://ieeexplore.ieee.org/abstract/document/9139852},
   }

4. ipdps20.2 Hcompress: Hierarchical data compression for multi-tiered storage environments

   Hariharan Devarajan, Anthony Kougkas, Luke Logan, and Xian-He Sun

   In Proceedings of the International Parallel and Distributed Processing Symposium , Jul 2020

   ABS BIB Cite

   

   Modern scientific applications read and write massive amounts of data through simulations, observations, and analysis. These applications spend the majority of their runtime in performing I/O. HPC storage solutions include fast node-local and shared storage resources to elevate applications from this bottleneck. Moreover, several middleware libraries (e.g., Hermes) are proposed to move data between these tiers transparently. Data reduction is another technique that reduces the amount of data produced and, hence, improve I/O performance. These two technologies, if used together, can benefit from each other. The effectiveness of data compression can be enhanced by selecting different compression algorithms according to the characteristics of the different tiers, and the multi-tiered hierarchy can benefit from extra capacity. In this paper, we design and implement HCompress, a hierarchical data compression library that can improve the application’s performance by harmoniously leveraging both multi-tiered storage and data compression. We have developed a novel compression selection algorithm that facilitates the optimal matching of compression libraries to the tiered storage. Our evaluation shows that HCompress can improve scientific application’s performance by 7x when compared to other state-of-the-art tiered storage solutions.

   @inproceedings{devarajan2020hcompress,
     entry_type = {conference},
     author = {Devarajan, Hariharan and Kougkas, Anthony and Logan, Luke and Sun, Xian-He},
     booktitle = {Proceedings of the International Parallel and Distributed Processing Symposium},
     title = {Hcompress: Hierarchical data compression for multi-tiered storage environments},
     year = {2020},
     month = jul,
     publisher = {IEEE},
     volume = {},
     number = {},
     pages = {557--566},
     keywords = {Data Compression Techniques, Multi-Tiered Storage Hierarchy, I/O Acceleration, Data Management in HPC},
     doi = {10.1109/IPDPS47924.2020.00064},
     url = {https://ieeexplore.ieee.org/abstract/document/9139838},
     elected = {}
   }

5. jcst20 I/O Acceleration via Multi-Tiered Data Buffering and Prefetching

   Anthony Kougkas, Hariharan Devarajan, and Xian-He Sun

   In Proceedings of the International Journal of Computer Science and Technology , Jan 2020

   ABS BIB Cite

   

   Modern High-Performance Computing (HPC) systems are adding extra layers to the memory and storage hierarchy, named deep memory and storage hierarchy (DMSH), to increase I/O performance. New hardware technologies, such as NVMe and SSD, have been introduced in burst buffer installations to reduce the pressure for external storage and boost the burstiness of modern I/O systems. The DMSH has demonstrated its strength and potential in practice. However, each layer of DMSH is an independent heterogeneous system and data movement among more layers is significantly more complex even without considering heterogeneity. How to efficiently utilize the DMSH is a subject of research facing the HPC community. Further, accessing data with a high-throughput and low-latency is more imperative than ever. Data prefetching is a well-known technique for hiding read latency by requesting data before it is needed to move it from a high-latency medium (e.g., disk) to a low-latency one (e.g., main memory). However, existing solutions do not consider the new deep memory and storage hierarchy and also suffer from under-utilization of prefetching resources and unnecessary evictions. Additionally, existing approaches implement a client-pull model where understanding the application’s I/O behavior drives prefetching decisions. Moving towards exascale, where machines run multiple applications concurrently by accessing files in a workflow, a more data-centric approach resolves challenges such as cache pollution and redundancy. In this paper, we present the design and implementation of Hermes: a new, heterogeneous-aware, multi-tiered, dynamic, and distributed I/O buffering system. Hermes enables, manages, supervises, and, in some sense, extends I/O buffering to fully integrate into the DMSH. We introduce three novel data placement policies to efficiently utilize all layers and we present three novel techniques to perform memory, metadata, and communication management in hierarchical buffering systems. Additionally, we demonstrate the benefits of a truly hierarchical data prefetcher that adopts a server-push approach to data prefetching. Our evaluation shows that, in addition to automatic data movement through the hierarchy, Hermes can significantly accelerate I/O and outperforms by more than 2x state-of-the-art buffering platforms. Lastly, results show 10% to 35% performance gains over existing prefetchers and over 50% when compared to systems with no prefetching.

   @inproceedings{kougkas2020acceleration,
     entry_type = {journal},
     author = {Kougkas, Anthony and Devarajan, Hariharan and Sun, Xian-He},
     booktitle = {Proceedings of the International Journal of Computer Science and Technology},
     title = {I/O Acceleration via Multi-Tiered Data Buffering and Prefetching},
     year = {2020},
     month = jan,
     publisher = {Springer},
     volume = {35},
     number = {1},
     pages = {92--120},
     keywords = {I/O Acceleration, Hierarchical Data Prefetching, Multi-Tiered Storage Hierarchy, Hierarchical Buffering},
     doi = {10.1007/s11390-020-9781-1},
     url = {https://link.springer.com/article/10.1007/s11390-020-9781-1},
   }

2019

1. ccgrid19 An intelligent, adaptive, and flexible data compression framework

   Hariharan Devarajan, Anthony Kougkas, and Xian-He Sun

   In Proceedings of the 19th International Symposium on Cluster, Cloud and Grid Computing , May 2019

   ABS BIB Cite

   

   The data explosion phenomenon in modern applications causes tremendous stress on storage systems. Developers use data compression, a size-reduction technique, to address this issue. However, each compression library exhibits different strengths and weaknesses when considering the input data entry_type and format. We present Ares, an intelligent, adaptive, and flexible compression framework which can dynamically choose a compression library for a given input data based on the entry_type of the workload and provides an appropriate infrastructure to users to fine-tune the chosen library. Ares is a modular framework which unifies several compression libraries while allowing the addition of more compression libraries by the user. Ares is a unified compression engine that abstracts the complexity of using different compression libraries for each workload. Evaluation results show that under real-world applications, from both scientific and Cloud domains, Ares performed 2-6x faster than competitive solutions with a low cost of additional data analysis (i.e., overheads around 10%) and up to 10x faster against a baseline of no compression at all.

   @inproceedings{devarajan2019intelligent,
     entry_type = {conference},
     author = {Devarajan, Hariharan and Kougkas, Anthony and Sun, Xian-He},
     booktitle = {Proceedings of the 19th International Symposium on Cluster, Cloud and Grid Computing},
     title = {An intelligent, adaptive, and flexible data compression framework},
     year = {2019},
     month = may,
     publisher = {IEEE},
     volume = {},
     number = {},
     pages = {82--91},
     keywords = {Data Compression Techniques, Data Management in HPC, I/O Acceleration, Storage Resource Provisioning},
     doi = {10.1109/CCGRID.2019.00019},
     url = {https://ieeexplore.ieee.org/abstract/document/8752926},
   }

2018

1. hipc18 Vidya: Performing code-block I/O characterization for data access optimization

   Hariharan Devarajan, Anthony Kougkas, Prajwal Challa, and Xian-He Sun

   In Proceedings of the 25th International Conference on High Performance Computing , Dec 2018

   ABS BIB Cite

   

   Understanding, characterizing and tuning scientific applications’ I/O behavior is an increasingly complicated process in HPC systems. Existing tools use either offline profiling or online analysis to get insights into the applications’ I/O patterns. However, there is lack of a clear formula to characterize applications’ I/O. Moreover, these tools are application specific and do not account for multi-tenant systems. This paper presents Vidya, an I/O profiling framework which can predict application’s I/O intensity using a new formula called Code-Block I/O Characterization (CIOC). Using CIOC, developers and system architects can tune an application’s I/O behavior and better match the underlying storage system to maximize performance. Evaluation results show that Vidya can predict an application’s I/O intensity with a variance of 0.05%. Vidya can profile applications with a high accuracy of 98% while reducing profiling time by 9x. We further show how Vidya can optimize an application’s I/O time by 3.7x.

   @inproceedings{devarajan2018vidya,
     entry_type = {conference},
     author = {Devarajan, Hariharan and Kougkas, Anthony and Challa, Prajwal and Sun, Xian-He},
     booktitle = {Proceedings of the 25th International Conference on High Performance Computing},
     title = {Vidya: Performing code-block I/O characterization for data access optimization},
     year = {2018},
     month = dec,
     publisher = {IEEE},
     volume = {},
     number = {},
     pages = {255--264},
     keywords = {I/O Profiling, Data Management in HPC, I/O Performance Optimization, Task-Based I/O},
     doi = {10.1109/HiPC.2018.00036},
     url = {https://ieeexplore.ieee.org/abstract/document/8638067},
   }

2. cluster18 Harmonia: An interference-aware dynamic I/O scheduler for shared non-volatile burst buffers

   Anthony Kougkas, Hariharan Devarajan, Xian-He Sun, and Jay Lofstead

   In Proceedings of the International Conference on Cluster Computing , Sep 2018

   ABS BIB Cite

   

   Modern HPC systems employ burst buffer installations to reduce the peak I/O requirements for external storage and deal with the burstiness of I/O in modern scientific applications. These I/O buffering resources are shared between multiple applications that run concurrently. This leads to severe performance degradation due to contention, a phenomenon called cross-application I/O interference. In this paper, we first explore the negative effects of interference at the burst buffer layer and we present two new metrics that can quantitatively describe the slowdown applications experience due to interference. We introduce Harmonia, a new dynamic I/O scheduler that is aware of interference, adapts to the underlying system, implements a new 2-way decision-making process and employs several scheduling policies to maximize the system efficiency and applications’ performance. Our evaluation shows that Harmonia, through better I/O scheduling, can outperform by 3x existing state-of-the-art buffering management solutions and can lead to better resource utilization.

   @inproceedings{kougkas2018harmonia,
     entry_type = {conference},
     author = {Kougkas, Anthony and Devarajan, Hariharan and Sun, Xian-He and Lofstead, Jay},
     booktitle = {Proceedings of the International Conference on Cluster Computing},
     title = {Harmonia: An interference-aware dynamic I/O scheduler for shared non-volatile burst buffers},
     year = {2018},
     month = sep,
     publisher = {IEEE},
     volume = {},
     number = {},
     pages = {290--301},
     keywords = {Burst Buffers, I/O Scheduling Policies, I/O Interference, Heterogeneous Storage Systems},
     doi = {10.1109/CLUSTER.2018.00046},
     url = {https://ieeexplore.ieee.org/abstract/document/8514889},
   }

3. hpdc18 Hermes: a heterogeneous-aware multi-tiered distributed I/O buffering system

   Anthony Kougkas, Hariharan Devarajan, and Xian-He Sun

   In Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing , Jun 2018

   ABS BIB Cite

   

   Modern High-Performance Computing (HPC) systems are adding extra layers to the memory and storage hierarchy named deep memory and storage hierarchy (DMSH), to increase I/O performance. New hardware technologies, such as NVMe and SSD, have been introduced in burst buffer installations to reduce the pressure for external storage and boost the burstiness of modern I/O systems. The DMSH has demonstrated its strength and potential in practice. However, each layer of DMSH is an independent heterogeneous system and data movement among more layers is significantly more complex even without considering heterogeneity. How to efficiently utilize the DMSH is a subject of research facing the HPC community. In this paper, we present the design and implementation of Hermes: a new, heterogeneous-aware, multi-tiered, dynamic, and distributed I/O buffering system. Hermes enables, manages, supervises, and, in some sense, extends I/O buffering to fully integrate into the DMSH. We introduce three novel data placement policies to efficiently utilize all layers and we present three novel techniques to perform memory, metadata, and communication management in hierarchical buffering systems. Our evaluation shows that, in addition to automatic data movement through the hierarchy, Hermes can significantly accelerate I/O and outperforms by more than 2x state-of-the-art buffering platforms.

   @inproceedings{kougkas2018hermes,
     entry_type = {conference},
     author = {Kougkas, Anthony and Devarajan, Hariharan and Sun, Xian-He},
     booktitle = {Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing},
     title = {Hermes: a heterogeneous-aware multi-tiered distributed I/O buffering system},
     year = {2018},
     month = jun,
     publisher = {ACM},
     volume = {},
     number = {},
     pages = {219--230},
     keywords = {Hierarchical Buffering, Multi-Tiered Storage Hierarchy, Heterogeneity-Aware Buffering, I/O Acceleration},
     doi = {10.1145/3208040.3208059},
     url = {https://dl.acm.org/doi/abs/10.1145/3208040.3208059},
   }

2016

1. escience16 Leveraging burst buffer coordination to prevent I/O interference

   Anthony Kougkas, Matthieu Dorier, Rob Latham, Rob Ross, and Xian-He Sun

   In Proceedings of the 12th International Conference on e-Science , Jun 2016

   ABS BIB Cite

   

   Concurrent accesses to the shared storage resources in current HPC machines lead to severe performance degradation caused by I/O contention. In this study, we identify some key challenges to efficiently handling interleaved data accesses, and we propose a system-wide solution to optimize global performance. We implemented and tested several I/O scheduling policies, including prioritizing specific applications by leveraging burst buffers to defer the conflicting accesses from another application and/or directing the requests to different storage servers inside the parallel file system infrastructure. The results show that we mitigate the negative effects of interference and optimize the performance up to 2x depending on the selected I/O policy.

   @inproceedings{bbio,
     entry_type = {conference},
     author = {Kougkas, Anthony and Dorier, Matthieu and Latham, Rob and Ross, Rob and Sun, Xian-He},
     booktitle = {Proceedings of the 12th International Conference on e-Science},
     title = {Leveraging burst buffer coordination to prevent I/O interference},
     year = {2016},
     month = jun,
     publisher = {IEEE},
     volume = {},
     number = {},
     pages = {371-380},
     keywords = {Burst Buffers, I/O Interference, I/O Scheduling Policies, Data Movement Optimization},
     doi = {10.1109/eScience.2016.7870922},
     url = {https://ieeexplore.ieee.org/document/7870922},
   }