Hermes

The question

High-performance machines do not have one kind of storage. They have a hierarchy: memory, local devices, burst buffers, parallel file systems, and increasingly object services. Applications knew the hierarchy existed, but they were often left to manage it with brittle, machine-specific logic.

Hermes asked whether middleware could present those tiers as one coordinated buffering system while still making intelligent placement decisions.

The architectural bet

The core bet was policy-driven placement. Data should move through the hierarchy according to workload behavior, device capability, capacity, and intent, not because one path was compiled into an application years earlier.

That required more than a fast data path. Hermes needed distributed metadata, adapters for existing I/O interfaces, a runtime that could coordinate placement, and deployment practices that worked outside the machine where a paper was written.

What shipped

Hermes became public, versioned middleware under The HDF Group organization. It acquired adapters, packaging, documentation, and a release history. That translation from architecture to maintained software is the part I return to most often.

A paper can isolate a good idea. Software forces the idea to negotiate with compilers, dependencies, failures, other people’s workloads, and the cost of being understandable. Those negotiations changed the architecture for the better and exposed which abstractions were durable.

My part, and the collective work

I served as project lead and architect. I worked on the placement model, system structure, and the long path from research prototype to a project others could build on. Xian-He Sun, The HDF Group, students, and collaborators across laboratories shaped the implementation and validation. The artifact trail is intentionally plural because that is how the system was made.

What carried forward

Hermes is complete as a project, but its ideas did not stop. The separation between interface, runtime, and heterogeneous storage; the attention to data placement; and the demand for deployable integration all carried into IOWarp and related systems.

That is the outcome I value most: not a frozen success story, but an architecture that became useful material for the next thing.

What I would do differently

I would make observability and evidence contracts first-class earlier. Placement decisions are easier to trust when a user can see why they happened. I would also design the packaging and upgrade path alongside the runtime from the beginning. Research software does not become real after the algorithm works; installation, diagnosis, and change are part of the system.