DTIO

dtio (datatask i/o) is a groundbreaking framework that introduces a new i/o paradigm to support a wide variety of conflicting i/o workloads under a single scalable platform ready for the convergence of high-performance computing (hpc), ai, and big data. as scientific applications become increasingly data-intensive with diverse, often conflicting i/o requirements, dtio provides a unified solution through an innovative task-based i/o approach.

what makes dtio special? 💡

dtio acts as an intelligent mediator between different i/o workloads. the core innovation is the concept of datatasks (dts) - a tuple of an operation and a pointer to data that allows applications to treat data as active objects capable of performing operations on themselves or other dts. this enables seamless integration of compute-centric and data-centric environments while providing robust i/o optimization capabilities.

behind the innovation

dtio emerged from the recognition that modern scientific workflows require a diverse and often conflicting set of storage features and semantics. our research shows that a unified approach is essential to address the challenges of data-intensive computing. the project, funded by the department of energy’s advanced scientific computing research (doe ascr) program, aims to create a novel data movement infrastructure that can efficiently handle the convergence of hpc, big data, and ai workloads.

key innovations

• datatask abstraction: novel concept enabling atomic, distributed and composable data transformations
• qos-aware scheduling: intelligent scheduling with asynchronous i/o capabilities
• unified i/o solution: seamless support for both legacy and modern i/o interfaces
• high performance: achieves significant speedup on real scientific applications
• resilience: built-in fault tolerance and data lineage tracking
• active storage: programmable locality-aware storage capabilities

real-world impact 🌍

dtio is making significant contributions across various scientific domains:

• climate modeling: supporting applications like cm1 with efficient data analysis integration
• molecular dynamics: enabling seamless i/o for applications like lammps
• astronomical data processing: accelerating workflows like montage
• weather forecasting: enhancing i/o performance for wrf simulations
• high-performance data analytics: bridging the gap between computing and data processing

technical architecture

dtio consists of several key components:

• datatask manager: handles dt specification and composition
• qos scheduler: manages dt scheduling and resource allocation
• asynchronous i/o engine: enables efficient overlapping of i/o operations
• resilience manager: provides fault tolerance and data lineage
• active storage layer: supports programmable storage capabilities

looking forward

dtio continues to evolve with exciting developments in:

• extended support for various i/o interfaces and patterns
• enhanced qos-aware scheduling techniques
• advanced resilience and fault tolerance capabilities
• deeper integration with active storage technologies

join the dtio community 🤝

dtio is an open-source project welcoming contributions from both academic and industrial researchers:

• repository: github - dtio project
• documentation: coming soon
• research papers: coming soon

acknowledgements 🙏

this material is based upon work supported by the u.s. department of energy, office of science, under award number de-sc0023386. this project is a collaborative effort between illinois institute of technology and argonne national laboratory. i am grateful to my partners at anl and the broader doe community whose expertise has been instrumental in advancing this project.

Interested in learning more about DTIO or discussing potential collaborations? Feel free to reach out!