LuxIO: Intelligent Resource Provisioning and Auto-Configuration for Storage Services
Abstract
Storage in HPC is typically a single Remote and Static Storage (RSS) resource. However, applications demonstrate diverse I/O requirements that can be better served by a multi-storage approach. Current practice employs ephemeral storage systems running on either node-local or shared storage resources. Yet, the burden of provisioning and configuring intermediate storage falls solely on the users, while global job schedulers offer little to no support for custom deployments. This lack of support often leads to over- or under-provisioning of resources and poorly configured storage systems. To mitigate this, we present LuxIO, an intelligent storage resource provisioning and auto-configuration service. LuxIO constructs storage deployments configured to best match I/O requirements. LuxIO-tuned storage services show performance improvements up to 2× across common applications and benchmarks, while introducing minimal overhead of 93.40 ms on top of existing job scheduling pipelines. LuxIO improves resource utilization by up to 25% in select workflows.
Finding
LuxIO-tuned storage services improve performance by up to 2x with 93.40 ms of added overhead, and improve resource utilization by up to 25% in select workflows.
System
- 2018–2024
- 2017–present
- 2020–present
More papers on Hermes
Browse all papers on Hermes- 2026
Benchmarking the Performance of Semantic Search Techniques for Agent Discovery
Rajni Pawar, Luke Logan, Xian-He Sun, Anthony Kougkas. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)
- 2024
MegaMmap: Blurring the Boundary Between Memory and Storage for Data-Intensive Workloads
Luke Logan, Anthony Kougkas, Xian-He Sun. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis SC
- 2024
DaYu: Optimizing Distributed Scientific Workflows by Decoding Dataflow Semantics and Dynamics
Meng Tang, Jaime Cernuda, Jie Ye, Luanzheng Guo, Nathan R. Tallent, Anthony Kougkas et al. Proceedings of the IEEE International Conference on Cluster Computing
- 2024
HStream: A hierarchical data streaming engine for high-throughput scientific applications
Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. Proceedings of the 53rd International Conference on Parallel Processing
Citation
@inproceedings{bateman2022luxio,
author = {Bateman, Keith and Rajesh, Neeraj and Garcia, Jaime Cernuda and Logan, Luke and Ye, Jie and Herbein, Stephen and Kougkas, Anthony and Sun, Xian-He},
booktitle = {Proceedings of the 29th International Conference on High Performance Computing, Data, and Analytics},
title = {LuxIO: Intelligent Resource Provisioning and Auto-Configuration for Storage Services},
year = {2022},
month = dec,
publisher = {IEEE},
pages = {246--255},
doi = {10.1109/HiPC56025.2022.00041},
url = {https://ieeexplore.ieee.org/abstract/document/10106285},
} Related notes
- 9 min
One destination, N tiers
Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.
- 3 min
CLIO Core 2.3 and 2.4: when a tier fills or a node dies
Four CLIO Core releases between October 9 and November 2, 2026 added a device-paged GPU vector and Intel GPU support, then taught the context store to recover a full tier and keep data and directories readable when a node goes down.
- 4 min
Two years of IOWarp
Two years after the NSF award, what IOWarp promised, what shipped as public releases, what changed course, and what is still a plan.