conferencecluster202020

Hcl: Distributing parallel data structures in extreme scales

Hariharan Devarajan, Anthony Kougkas, Keith Bateman, Xian-He Sun

Proceedings of the International Conference on Cluster Computing · IEEE

Read PDF DOI ↗

Abstract

Most parallel programs use irregular control flow and data structures, which are perfect for one-sided communication paradigms such as MPI or PGAS programming languages. However, these environments lack the presence of efficient function-based application libraries that can utilize popular communication fabrics such as TCP, Infinity Band (IB), and RDMA over Converged Ethernet (RoCE). Additionally, there is a lack of high-performance data structure interfaces. We present Hermes Container Library (HCL), a high-performance distributed data structures library that offers high-level abstractions including hash-maps, sets, and queues. HCL uses a RPC over RDMA technology that implements a novel procedural programming paradigm. In this paper, we argue a RPC over RDMA technology can serve as a high-performance, flexible, and co-ordination free backend for implementing complex data structures. Evaluation results from testing real workloads shows that HCL programs are 2x to 12x faster compared to BCL, a state-of-the-art distributed data structure library.

Connected work

HermesChronologDtio

Citation

@inproceedings{devarajan2020hcl,
  author = {Devarajan, Hariharan and Kougkas, Anthony and Bateman, Keith and Sun, Xian-He},
  booktitle = {Proceedings of the International Conference on Cluster Computing},
  title = {Hcl: Distributing parallel data structures in extreme scales},
  year = {2020},
  month = sep,
  publisher = {IEEE},
  volume = {},
  number = {},
  pages = {248--258},
  keywords = {Distributed Data Structures, Parallel I/O Optimization, Storage Architectures, High-Performance Computing},
  doi = {10.1109/CLUSTER49012.2020.00035},
  url = {https://ieeexplore.ieee.org/abstract/document/9229595},
}