When is research software shipped?

A paper acceptance is not a software release. I have blurred that line myself, more than once.

In research we say a system “exists” when what we mean is that our team ran a particular commit, on a particular machine, with enough local knowledge to produce the evaluation in a paper. That is a real achievement. It is also a long way from giving a stranger a fair chance to use the software. I needed a more useful word for the far side of that boundary, and the word I settled on is shipped.

Research software is shipped when someone outside the room where it was built can identify a named state, reach it, run at least one intended behavior, understand the evidence and its boundary, and report what happened without reconstructing the project from our private memory.

That definition is deliberately smaller than “production ready.” It is also more demanding than “the code is on GitHub.”

The states are different

I find it useful to separate five states that usually get collapsed into one announcement:

Experiment. The code answers a research question for the people running it.
Prototype. The important mechanism is implemented well enough to inspect and evaluate.
Paper artifact. The software, inputs, and method support a published argument within a stated environment.
Release. A named version has an installation path, an intended behavior, and public notes about what changed.
Maintained software. Ownership, issue handling, compatibility decisions, and another release are all plausible after the paper deadline has passed.

Each state is valuable. The trouble starts when I describe one as another. A prototype is not diminished by calling it a prototype. A paper artifact is not a failed product. The label tells a reader what kind of promise we are making, and I would rather make a small promise I can keep.

“Shipped” usually begins at the release boundary, but a tag alone is not enough. The tag has to point at a usable contract.

The smallest release I trust

Before I call one of our systems shipped, I look for a short list of receipts.

A named, recoverable state. A version, release, image digest, or immutable commit that two people can identify without asking which branch was current that afternoon. “Use main” is a moving target, not a release coordinate.

A path from zero. Installation instructions that begin before the maintainer’s shell history: dependencies, supported environments, build or package commands, and the first expected result. A container helps, but a container with no description of its assumptions only freezes the mystery.

One behavior another person can exercise. A minimal example persuades more than a broad capability list. The question is not whether every scientific workflow has been packaged. It is whether a newcomer can make one intended path work and recognize the output.

Evidence connected to the version. A benchmark number needs the version, configuration, input, machine boundary, baseline, and measurement method that make it interpretable. The paper carries the argument; the release should say which part of that argument the current code reproduces.

Explicit limits. Supported platforms, incomplete components, scaling boundaries, known failures, data restrictions, and security assumptions belong next to the instructions. Hiding them does not make the software stronger. It makes the first external user rediscover the same boundary, expensively.

A continuity signal. Where issues go, who decides whether a change belongs, and whether the project is active, complete, or archived. That is not a promise of support forever. It stops a public repository from impersonating a maintained service.

Two systems I hold to this

Hermes is the cleanest example I have. It started as the center of my PhD, became five years of NSF-funded work with The HDF Group, and closed in August 2024 with v1.2.1: a named state, under an organization that outlives the grant, with adapters, packaging, and documentation that other people used on Theta and Summit. It is complete, and I say so. Its ideas kept moving into IOWarp, but the software itself has a last release, and that is a legitimate ending.

DLIO shipped in a way I did not plan. We built it with Argonne to reproduce the I/O of scientific deep-learning jobs; the CCGrid 2021 paper carried the argument. What made it shipped was everything after: MLPerf Storage adopted it as the workload generator, the Argonne team and the community kept releasing it, and v2.0.1 is a state that storage vendors install without ever talking to us. Integration sharpened the interfaces, because assumptions that felt harmless in one lab became bugs across submitters’ platforms.

Neither release proves portability, performance, or adoption by itself. What each one does is give those questions a concrete object.

Where I am being careful

IOWarp is at a different point, and I want the language to say so. The public organization makes active implementation visible, but the visibility of a repository must not turn every design target into a shipped claim. Inside CLIO, Core and Kit are released and maintained, Agent is beta, and Clio Coder is experimental and ships quickly. Each component earns “shipped” through its own releases, instructions, tests, and stated limits. ChronoLog 3.0 earned it in May with a changelog; the LABIOS 2.0 branch has not yet, and its story says exactly that.

Shipping changes the research

Release work is sometimes treated as cleanup after the interesting systems work. In my experience it exposes systems questions that the paper path lets you avoid. What is the actual interface? Which configuration is part of the scientific method? Which behavior is stable, and which was incidental to one machine? Which dependency is essential? Which failure should stop a run instead of silently changing it? Who owns the next decision?

Those questions improve the architecture. They also change the relationship with readers. A paper asks someone to evaluate an argument. Shipped software lets them inspect the mechanism, try it in a different setting, and return evidence we did not control. The most useful feedback I have received on any of these systems arrived after the software left the environment that made all of its assumptions feel obvious.

Software is collective work

The release is never just the person who tells the story about it. Hermes was built by a team spanning the Gnosis Research Center, The HDF Group, and research partners. DLIO grew through the Argonne team and collaborators who designed, implemented, evaluated, documented, and maintained it. Students and research software engineers do most of the work that turns a paper prototype into something another person can run.

Credit should follow those contributions into the release notes, the paper, the repository history, and the project story. “I shipped it” is only honest when “I” names my role and “we” stays visible.

What shipped does not mean

It does not mean the software works on every system, is secure in every setting, or will be supported indefinitely. Some artifacts require scarce hardware or licensed data. Some cannot be released immediately because of an embargo, a security concern, a partner agreement, or a privacy obligation. Some research code should be preserved as an artifact and then deliberately archived rather than maintained badly.

It also does not mean finished. Shipped software can be narrow, early, and unfinished. The promise is smaller: here is the state, here is what it does, here is how we know, here is what we do not know, and here is where the next result belongs.

That is enough to move research software from a claim about our past into an object someone else can question, use, and carry forward. It is the standard I am trying to hold IOWarp and CLIO to, one component at a time.