LuxIO: Intelligent Resource Provisioning and Auto-Configuration for Storage Services
Proceedings of the 29th International Conference on High Performance Computing, Data, and Analytics · IEEE
Abstract
Storage in HPC is typically a single Remote and Static Storage (RSS) resource. However, applications demonstrate diverse I/O requirements that can be better served by a multi-storage approach. Current practice employs ephemeral storage systems running on either node-local or shared storage resources. Yet, the burden of provisioning and configuring intermediate storage falls solely on the users, while global job schedulers offer little to no support for custom deployments. This lack of support often leads to over- or under-provisioning of resources and poorly configured storage systems. To mitigate this, we present LuxIO, an intelligent storage resource provisioning and auto-configuration service. LuxIO constructs storage deployments configured to best match I/O requirements. LuxIO-tuned storage services show performance improvements up to 2× across common applications and benchmarks, while introducing minimal overhead of 93.40 ms on top of existing job scheduling pipelines. LuxIO improves resource utilization by up to 25% in select workflows.
Connected work
Citation
@inproceedings{bateman2022luxio,
author = {Bateman, Keith and Rajesh, Neeraj and Garcia, Jaime Cernuda and Logan, Luke and Ye, Jie and Herbein, Stephen and Kougkas, Anthony and Sun, Xian-He},
booktitle = {Proceedings of the 29th International Conference on High Performance Computing, Data, and Analytics},
title = {LuxIO: Intelligent Resource Provisioning and Auto-Configuration for Storage Services},
year = {2022},
month = dec,
publisher = {IEEE},
volume = {},
number = {},
pages = {246--255},
keywords = {Storage Resource Provisioning, Data Management in HPC, Elastic Storage, Workflow Optimization},
doi = {10.1109/HiPC56025.2022.00041},
url = {https://ieeexplore.ieee.org/abstract/document/10106285},
}