akougkas.io

Syndesis: Mapping Objects to Files for a Unified Data Access System

Anthony Kougkas, Hariharan Devarajan, Xian-He Sun

The ACM SIGHPC 8th International Workshop on Many-Task Computing on Clouds, Grids, and Supercomputers (MTAGS 2017), in conjunction with SC’17, Denver, CO, USAACM

Abstract

The predominant data model in cloud storage is the object-based storage. Object stores follow a simpler API with get() and put() operations to interact with the data. A wide variety of data analysis software has been developed around objects using their APIs. In fact, the evolution of Big Data analytics is a major driver for highly optimized data-centric quality software. However, organizations maintain file-based storage clusters and a high volume of existing data are stored in files. This is specifically true for the scientific communities. In this paper, we present the key characteristics of object-based and file-based storage APIs, we explore several object-to-file mappings aiming to bridge the semantic gap between these data models. The evaluation of our mapping algorithms exposes various strengths and weaknesses of each strategy and frames the extended potential of a unified data access system. Results show that our solution can offer more than 3x higher performance for specific workloads while keeping minimal overhead of our library.

Finding

Syndesis’s object-to-file mapping strategies offer more than 3x higher performance for specific workloads while keeping the library’s overhead minimal.

System

  1. completed

    IRIS

    A unified I/O layer over POSIX files and object stores, from my doctoral work.

Browse all papers on IRIS
  1. 2022
    conference

    LuxIO: Intelligent Resource Provisioning and Auto-Configuration for Storage Services

    Keith Bateman, Neeraj Rajesh, Jaime Cernuda Garcia, Luke Logan, Jie Ye, … Anthony Kougkas et al. Proceedings of the 29th International Conference on High Performance Computing, Data, and Analytics

  2. 2022
    conference

    LabStor: A modular and extensible platform for developing high-performance, customized I/O stacks in userspace

    Luke Logan, Jaime Cernuda Garcia, Jay Lofstead, Xian–He Sun, Anthony Kougkas. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

  3. 2022
    conference

    Stimulus: Accelerate Data Management for Scientific AI applications in HPC

    Hariharan Devarajan, Anthony Kougkas, Huihuo Zheng, Venkatram Vishwanath, Xian-He Sun. Proceedings of the 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing

  4. 2021
    conference

    HFlow: A Dynamic and Elastic Multi-Layered I/O Forwarder

    Jaime Cernuda, Hariharan Devarajan, Luke Logan, Keith Bateman, Neeraj Rajesh, … Anthony Kougkas et al. Proceedings of the International Conference on Cluster Computing

Citation

@inproceedings{kougkas2017syndesis,
  author = {Kougkas, A. and Devarajan, H. and Sun, X.-H.},
  title = {{Syndesis: Mapping Objects to Files for a Unified Data Access System}},
  booktitle = {The ACM SIGHPC 8th International Workshop on Many-Task Computing on Clouds, Grids, and Supercomputers (MTAGS 2017), in conjunction with SC'17, Denver, CO, USA},
  year = {2017},
  month = nov,
  publisher = {ACM},
  url = {http://cs.iit.edu/%7Escs/assets/files/Syndesis.pdf},
}