akougkas.io

Apollo: An ML-assisted real-time storage resource observer

Neeraj Rajesh, Hariharan Devarajan, Jaime Cernuda Garcia, Keith Bateman, Luke Logan, Jie Ye, Anthony Kougkas, Xian-He Sun

Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed ComputingACM

Abstract

Applications and middleware services, such as data placement engines, I/O scheduling, and prefetching engines, require low-latency access to telemetry data in order to make optimal decisions. However, typical monitoring services store their telemetry data in a database in order to allow applications to query them, resulting in significant latency penalties. This work presents Apollo: a low-latency monitoring service that aims to provide applications and middleware libraries with direct access to relational telemetry data. Monitoring the system can create interference and overhead, slowing down raw performance of the resources for the job. However, having a current view of the system can aid middleware services in making more optimal decisions which can ultimately improve the overall performance. Apollo has been designed from the ground up to provide low latency, using Publish-Subscriber Pub-Sub semantics, and low overhead, using adaptive intervals in order to change the length of time between polling the resource for telemetry data and machine learning in order to predict changes to the telemetry data between actual resource polling. This work also provides some high level abstractions called I/O curators, which can further aid middleware libraries and applications to make optimal decisions. Evaluations showcase that Apollo can achieve sub-millisecond latency for acquiring complex insights with a memory overhead of  57 MB and CPU overhead being only 7% more than existing state-of-the-art systems.

Finding

Apollo delivers sub-millisecond latency for complex telemetry insights with about 57 MB of memory overhead and 7% more CPU than existing state-of-the-art systems.

System

  1. 2018–2024
    complete

    Hermes

  2. 2017–present
    evolving

    LABIOS

  3. 2020–present
    active

    ChronoLog

  4. completed

    IRIS

Browse all papers on Hermes
  1. 2026
    poster

    Benchmarking the Performance of Semantic Search Techniques for Agent Discovery

    Rajni Pawar, Luke Logan, Xian-He Sun, Anthony Kougkas. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)

  2. 2024
    conference

    MegaMmap: Blurring the Boundary Between Memory and Storage for Data-Intensive Workloads

    Luke Logan, Anthony Kougkas, Xian-He Sun. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis SC

  3. 2024
    conference

    DaYu: Optimizing Distributed Scientific Workflows by Decoding Dataflow Semantics and Dynamics

    Meng Tang, Jaime Cernuda, Jie Ye, Luanzheng Guo, Nathan R. Tallent, Anthony Kougkas et al. Proceedings of the IEEE International Conference on Cluster Computing

  4. 2024
    conference

    HStream: A hierarchical data streaming engine for high-throughput scientific applications

    Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. Proceedings of the 53rd International Conference on Parallel Processing

Citation

@inproceedings{rajesh2021apollo,
  author = {Rajesh, Neeraj and Devarajan, Hariharan and Garcia, Jaime Cernuda and Bateman, Keith and Logan, Luke and Ye, Jie and Kougkas, Anthony and Sun, Xian-He},
  booktitle = {Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing},
  title = {Apollo: An ML-assisted real-time storage resource observer},
  year = {2021},
  month = jun,
  publisher = {ACM},
  pages = {147--159},
  doi = {10.1145/3431379.3460640},
  url = {https://dl.acm.org/doi/abs/10.1145/3431379.3460640},
}

Related notes

  1. essay

    One destination, N tiers

    Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.

    9 min
  2. update

    CLIO Core 2.3 and 2.4: when a tier fills or a node dies

    Four CLIO Core releases between October 9 and November 2, 2026 added a device-paged GPU vector and Intel GPU support, then taught the context store to recover a full tier and keep data and directories readable when a node goes down.

    3 min
  3. update

    Two years of IOWarp

    Two years after the NSF award, what IOWarp promised, what shipped as public releases, what changed course, and what is still a plan.

    4 min