<?xml version="1.0" encoding="utf-8"?> <feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"> <title>Updates · Anthony Kougkas</title> <subtitle>Updates by Anthony Kougkas.</subtitle> <id>https://akougkas.io/feed/update.xml</id> <link rel="self" type="application/atom+xml" href="https://akougkas.io/feed/update.xml"/> <link rel="alternate" type="text/html" href="https://akougkas.io/journal/?kind=update"/> <updated>2026-11-03T00:00:00-06:00</updated> <author> <name>Anthony Kougkas</name> <uri>https://akougkas.io/about/</uri> </author> <entry> <title type="html">CLIO Core 2.3 and 2.4: when a tier fills or a node dies</title> <link rel="alternate" type="text/html" href="https://akougkas.io/journal/2026/clio-core-2-3-and-2-4-when-a-tier-fills-or-a-node-dies/"/> <id>https://akougkas.io/journal/2026/clio-core-2-3-and-2-4-when-a-tier-fills-or-a-node-dies/</id> <published>2026-11-03T00:00:00-06:00</published> <updated>2026-11-03T00:00:00-06:00</updated> <summary type="html">Four CLIO Core releases between October 9 and November 2, 2026 added a device-paged GPU vector and Intel GPU support, then taught the context store to recover a full tier and keep data and directories readable when a node goes down.</summary> <content type="html" xml:base="https://akougkas.io/journal/2026/clio-core-2-3-and-2-4-when-a-tier-fills-or-a-node-dies/">&lt;p&gt;&lt;a href=&quot;https://github.com/iowarp/clio-core&quot;&gt;CLIO Core&lt;/a&gt;, the tiered context and data runtime under &lt;a href=&quot;/work/clio/&quot;&gt;CLIO&lt;/a&gt;, shipped four releases between October 9 and November 2. The thread through them is what a context store owes its callers when a tier fills or a node dies. &lt;a href=&quot;/work/hermes/&quot;&gt;Hermes&lt;/a&gt;, the multi-tier I/O buffer I designed in 2018, asked how data should move through a storage hierarchy; the Context Transfer Engine in Core is its direct descendant, and these releases ask what happens when that hierarchy breaks.&lt;/p&gt; &lt;h2 id=&quot;230-gpus-hdf5-and-full-tiers&quot;&gt;2.3.0: GPUs, HDF5, and full tiers&lt;/h2&gt; &lt;p&gt;&lt;a href=&quot;https://github.com/iowarp/clio-core/releases/tag/v2.3.0&quot;&gt;2.3.0&lt;/a&gt; is 85 merged pull requests since 2.2.1. It adds a device-paged &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;gpu_vector&lt;/code&gt;, and the GPU runtime now starts and links on Intel GPUs, which took Aurora from 104 to 254 passing tests. HDF5 VFD reads are now served from the CLIO cache tier.&lt;/p&gt; &lt;p&gt;The storage-correctness section is the part I care about most. A full tier now evicts to make room, the configured tier score decides placement instead of a bandwidth guess, a WAL-replay deadlock is gone, and safe-profile recovery took 7 fixes across 5 fault scenarios. The runtime also serves its own dashboard, and a cross-node collective bug was fixed with 3.1x lower latency.&lt;/p&gt; &lt;p&gt;The 2.3.0 wheels lacked the indexer module that the default configuration composes, so they were never published. &lt;a href=&quot;https://github.com/iowarp/clio-core/releases/tag/v2.3.1&quot;&gt;2.3.1&lt;/a&gt; fixed that the same day, and the wheel tests now check for every composed module.&lt;/p&gt; &lt;h2 id=&quot;232-recovering-a-full-tier-from-python&quot;&gt;2.3.2: recovering a full tier from Python&lt;/h2&gt; &lt;p&gt;Eviction covers data that can be dropped. Data that cannot be dropped needs somewhere to go. &lt;a href=&quot;https://github.com/iowarp/clio-core/releases/tag/v2.3.2&quot;&gt;2.3.2&lt;/a&gt; lets Python callers query capacity, targets, and statistics, and register a new file target with a stable identity while the runtime keeps running. Its regression test fills a tier, observes the refusal, registers another target, retries the refused record, and verifies nine byte-identical records across a restart. That is what I want from storage: refuse clearly, then let the caller add capacity without losing what is there.&lt;/p&gt; &lt;h2 id=&quot;240-when-a-node-dies&quot;&gt;2.4.0: when a node dies&lt;/h2&gt; &lt;p&gt;&lt;a href=&quot;https://github.com/iowarp/clio-core/releases/tag/v2.4.0&quot;&gt;2.4.0&lt;/a&gt; has no written notes, so this account comes from its &lt;a href=&quot;https://github.com/iowarp/clio-core/compare/v2.3.2...v2.4.0&quot;&gt;merged changes&lt;/a&gt;. Most of them came out of a six-node reliability and tiering suite on Ares, the &lt;a href=&quot;https://www.iit.edu/&quot;&gt;Illinois Tech&lt;/a&gt; cluster, and they share one rule: a store must not report data as missing when it is only unreachable.&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;strong&gt;Failover reads.&lt;/strong&gt; A stand-in that misses a failover read answers “owner down”, not “no blob”, and a non-owner never reports a dead owner’s blob as absent.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;Replication hand-back.&lt;/strong&gt; A hand-back push never rolls back a newer write by the owner, and a restart’s hand-back pull waits for a bounded time.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;clio-fs directories.&lt;/strong&gt; A directory journal replays before a home block can load, and directory blocks survive their home node going down.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;FUSE and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;safe_bdev&lt;/code&gt;.&lt;/strong&gt; Hard-linked files publish their size at close, a closer never waits on itself, and a stuck-operation watchdog names callbacks in flight. &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;safe_bdev&lt;/code&gt; starts degraded when members are dead at start, within its configured failure budget, and keeps a mid-write member loss recoverable.&lt;/li&gt; &lt;/ul&gt; &lt;h2 id=&quot;boundary-and-next-step&quot;&gt;Boundary and next step&lt;/h2&gt; &lt;p&gt;The Aurora test counts and the 3.1x latency figure come from the 2.3.0 notes, not from a measurement of mine. The Ares suite’s pull request was merged with a work-in-progress title, and 2.4.0’s behavior is described here by its change titles until the release has notes of its own.&lt;/p&gt; &lt;h2 id=&quot;who-did-the-work&quot;&gt;Who did the work&lt;/h2&gt; &lt;p&gt;Day-to-day Core engineering is led by &lt;a href=&quot;https://grc.iit.edu/center/team/luke-logan/&quot;&gt;Luke Logan&lt;/a&gt;, who built the runtime under Hermes as a student and carried most of the reliability work in these four releases. Engineers at GRC and &lt;a href=&quot;https://www.hdfgroup.org/&quot;&gt;The HDF Group&lt;/a&gt; contributed across the GPU, adapter, and storage changes, and each change’s pull request records its author.&lt;/p&gt; </content> </entry> <entry> <title type="html">Two years of IOWarp</title> <link rel="alternate" type="text/html" href="https://akougkas.io/journal/2026/two-years-of-iowarp/"/> <id>https://akougkas.io/journal/2026/two-years-of-iowarp/</id> <published>2026-09-06T00:00:00-05:00</published> <updated>2026-09-06T00:00:00-05:00</updated> <summary type="html">Two years after the NSF award, what IOWarp promised, what shipped as public releases, what changed course, and what is still a plan.</summary> <content type="html" xml:base="https://akougkas.io/journal/2026/two-years-of-iowarp/">&lt;p&gt;The &lt;a href=&quot;/work/iowarp/&quot;&gt;IOWarp&lt;/a&gt; award started on August 1, 2024. When I &lt;a href=&quot;/journal/2024/iowarp-nsf-award/&quot;&gt;announced it&lt;/a&gt; I wrote that an award is runway and the proof is in releases. Two years in, here is the ledger. I keep it in public on purpose; the version in my head is always kinder.&lt;/p&gt; &lt;aside class=&quot;entry-glance&quot; aria-label=&quot;At a glance&quot;&gt; &lt;p class=&quot;entry-glance__kicker&quot;&gt;At a glance&lt;/p&gt; &lt;dl&gt; &lt;div&gt;&lt;dt&gt;Anchor&lt;/dt&gt;&lt;dd&gt;NSF award OAC-2411318, started August 1, 2024, two years in&lt;/dd&gt;&lt;/div&gt; &lt;div&gt;&lt;dt&gt;Systems&lt;/dt&gt;&lt;dd&gt;&lt;a href=&quot;/work/iowarp/&quot;&gt;IOWarp&lt;/a&gt;, &lt;a href=&quot;/work/clio/&quot;&gt;CLIO&lt;/a&gt;, &lt;a href=&quot;/work/clio-coder/&quot;&gt;CLIO Coder&lt;/a&gt;&lt;/dd&gt;&lt;/div&gt; &lt;div&gt;&lt;dt&gt;The claim&lt;/dt&gt;&lt;dd&gt;The proof of an award is in public releases, and after two years the ledger is tags, not plans.&lt;/dd&gt;&lt;/div&gt; &lt;div&gt;&lt;dt&gt;Code&lt;/dt&gt;&lt;dd&gt;&lt;a href=&quot;https://github.com/iowarp&quot;&gt;github.com/iowarp&lt;/a&gt;&lt;/dd&gt;&lt;/div&gt; &lt;/dl&gt; &lt;/aside&gt; &lt;h2 id=&quot;what-we-promised&quot;&gt;What we promised&lt;/h2&gt; &lt;p&gt;The proposal I wrote in 2023 had four parts: a self-describing unit called Content, a transfer engine that was &lt;a href=&quot;/work/hermes/&quot;&gt;Hermes&lt;/a&gt; grown up, an assimilation engine for reshaping data across formats, and a natural-language layer that would answer questions about scientific data by calling operators as tools. The claim under all of it was that scientific workflows need a context layer, not only a faster byte pipe.&lt;/p&gt; &lt;h2 id=&quot;what-shipped&quot;&gt;What shipped&lt;/h2&gt; &lt;p&gt;Public tags, not plans.&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;strong&gt;&lt;a href=&quot;https://github.com/iowarp/clio-kit&quot;&gt;CLIO Kit&lt;/a&gt;.&lt;/strong&gt; The MCP servers for HDF5, Slurm, ParaView, NetCDF, Zarr, and the rest, releasing since August 2025 and at &lt;a href=&quot;https://github.com/iowarp/clio-kit/releases/tag/v2.10.6&quot;&gt;v2.10.6&lt;/a&gt;.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;&lt;a href=&quot;https://github.com/iowarp/clio-core&quot;&gt;CLIO Core&lt;/a&gt;.&lt;/strong&gt; The runtime and tiered context store, which reached &lt;a href=&quot;https://github.com/iowarp/clio-core/releases/tag/v2.0.0&quot;&gt;2.0.0&lt;/a&gt; on May 27, 2026 with the Transfer and Assimilation Engines, 2.1.0 in June with semantic and temporal search, and 2.2.1 on August 18.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;&lt;a href=&quot;https://github.com/iowarp/clio-agent&quot;&gt;CLIO Agent&lt;/a&gt;.&lt;/strong&gt; Tagged releases since April 2026, at &lt;a href=&quot;https://github.com/iowarp/clio-agent/releases/tag/v0.9.1.1&quot;&gt;v0.9.1.1&lt;/a&gt;, in beta.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;&lt;a href=&quot;https://github.com/iowarp/clio-relay&quot;&gt;CLIO Relay&lt;/a&gt;.&lt;/strong&gt; At v1.6.8.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;&lt;a href=&quot;/work/clio-coder/&quot;&gt;CLIO Coder&lt;/a&gt;.&lt;/strong&gt; From an experimental tag in April to &lt;a href=&quot;https://www.npmjs.com/package/@iowarp/clio-coder&quot;&gt;npm&lt;/a&gt; in August, at &lt;a href=&quot;https://github.com/iowarp/clio-coder/releases&quot;&gt;v0.4.x&lt;/a&gt; this week.&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;The team coordinated all of it into one release wave between August 14 and 24.&lt;/p&gt; &lt;figure class=&quot;figure--diagram&quot;&gt; &lt;svg viewBox=&quot;0 0 640 240&quot; role=&quot;img&quot; aria-labelledby=&quot;fig-iowarp-two-years&quot;&gt; &lt;title id=&quot;fig-iowarp-two-years&quot;&gt;Public IOWarp releases from August 2025 to August 2026: CLIO Kit first, then CLIO Agent and CLIO Coder tags in April 2026, CLIO Core 2.0.0 in May 2026, and the release wave in August 2026.&lt;/title&gt; &lt;path class=&quot;d-rule&quot; d=&quot;M40 130 H596&quot; /&gt; &lt;polygon class=&quot;d-accent-fill&quot; points=&quot;596,124 610,130 596,136&quot; /&gt; &lt;circle class=&quot;d-accent-fill&quot; cx=&quot;100&quot; cy=&quot;130&quot; r=&quot;8&quot; /&gt; &lt;circle class=&quot;d-accent-fill&quot; cx=&quot;260&quot; cy=&quot;130&quot; r=&quot;8&quot; /&gt; &lt;circle class=&quot;d-accent-fill&quot; cx=&quot;420&quot; cy=&quot;130&quot; r=&quot;8&quot; /&gt; &lt;circle class=&quot;d-cyan-fill&quot; cx=&quot;540&quot; cy=&quot;130&quot; r=&quot;8&quot; /&gt; &lt;text class=&quot;d-text&quot; x=&quot;100&quot; y=&quot;96&quot; text-anchor=&quot;middle&quot;&gt;Kit releases&lt;/text&gt; &lt;text class=&quot;d-label&quot; x=&quot;100&quot; y=&quot;170&quot; text-anchor=&quot;middle&quot;&gt;Aug 2025&lt;/text&gt; &lt;path class=&quot;d-rule&quot; d=&quot;M260 74 V118&quot; /&gt; &lt;text class=&quot;d-text&quot; x=&quot;260&quot; y=&quot;62&quot; text-anchor=&quot;middle&quot;&gt;Agent and Coder&lt;/text&gt; &lt;text class=&quot;d-label&quot; x=&quot;260&quot; y=&quot;170&quot; text-anchor=&quot;middle&quot;&gt;Apr 2026&lt;/text&gt; &lt;text class=&quot;d-text&quot; x=&quot;420&quot; y=&quot;96&quot; text-anchor=&quot;middle&quot;&gt;Core 2.0.0&lt;/text&gt; &lt;text class=&quot;d-label&quot; x=&quot;420&quot; y=&quot;170&quot; text-anchor=&quot;middle&quot;&gt;May 2026&lt;/text&gt; &lt;path class=&quot;d-rule&quot; d=&quot;M540 74 V118&quot; /&gt; &lt;text class=&quot;d-text&quot; x=&quot;540&quot; y=&quot;62&quot; text-anchor=&quot;middle&quot;&gt;Release wave&lt;/text&gt; &lt;text class=&quot;d-label&quot; x=&quot;540&quot; y=&quot;170&quot; text-anchor=&quot;middle&quot;&gt;Aug 2026&lt;/text&gt; &lt;/svg&gt; &lt;figcaption&gt;One year of public tags: CLIO Kit from August 2025, the first CLIO Agent and CLIO Coder tags in April 2026, CLIO Core 2.0.0 in May 2026, and the coordinated release wave in August 2026.&lt;/figcaption&gt; &lt;/figure&gt; &lt;p&gt;Core has been built on Ares, DeltaAI, Polaris, and Aurora. Eighteen papers in this site’s bibliography carry the IOWarp tag. And in August GRC joined two &lt;a href=&quot;https://www.energy.gov/undersecretaryforscience/genesis-mission/genesis-mission&quot;&gt;DOE Genesis&lt;/a&gt; projects that will put Core, Kit, Agent, and Coder in front of teams at &lt;a href=&quot;https://www.anl.gov/&quot;&gt;Argonne&lt;/a&gt; and &lt;a href=&quot;https://www.pnnl.gov/&quot;&gt;Pacific Northwest&lt;/a&gt;.&lt;/p&gt; &lt;h2 id=&quot;what-changed-course&quot;&gt;What changed course&lt;/h2&gt; &lt;p&gt;The natural-language layer in the proposal predated the &lt;a href=&quot;https://modelcontextprotocol.io/&quot;&gt;Model Context Protocol&lt;/a&gt;. When MCP arrived, a bespoke model made less sense than a kit of tools any capable agent could mount, and that became &lt;a href=&quot;https://toolkit.iowarp.ai/&quot;&gt;CLIO Kit&lt;/a&gt; and &lt;a href=&quot;https://clio.iowarp.ai/&quot;&gt;CLIO Agent&lt;/a&gt;. Along the way I forked Gemini CLI as &lt;a href=&quot;https://github.com/akougkas/warpio-cli&quot;&gt;Warpio&lt;/a&gt;; its last public commit is from September 2025 and I retired it, because inheriting someone else’s harness cost more than owning one. What I learned from it went through two private prototypes, AWOC and then PanCode, before it became CLIO Coder. The &lt;a href=&quot;https://github.com/akougkas/claude-code-4-science&quot;&gt;“4 Science” extensions&lt;/a&gt; for commercial agents are public but Kit is the maintained path.&lt;/p&gt; &lt;aside class=&quot;entry-pull&quot;&gt; &lt;p&gt;Agent is beta and Coder is experimental, and both words are load-bearing.&lt;/p&gt; &lt;/aside&gt; &lt;h2 id=&quot;what-is-still-a-plan&quot;&gt;What is still a plan&lt;/h2&gt; &lt;p&gt;Agent is beta and Coder is experimental, and both words are load-bearing. I have overstated before, and I would rather not again. The &lt;a href=&quot;/work/labios/&quot;&gt;LABIOS 2.0&lt;/a&gt; branch is not a release.&lt;/p&gt; &lt;p&gt;AgentLog, the typed activity log on top of &lt;a href=&quot;/work/chronolog/&quot;&gt;ChronoLog&lt;/a&gt;, is design-stage. The shared record schemas that should carry the contracts between components still live partly in prose. And I do not have a universal schema for context; I doubt one exists across instruments, simulations, and institutions, which is why the smallest useful contract is the research question.&lt;/p&gt; &lt;h2 id=&quot;year-three&quot;&gt;Year three&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;&lt;strong&gt;Make the receipt a first-class object&lt;/strong&gt; across every component, not only inside CLIO Coder.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;Keep local models the design target&lt;/strong&gt; so a lab with one GPU is a real user.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;Build the regression corpus&lt;/strong&gt; that turns “the agent got better” into a measurable claim.&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;My rule for the year: no claim on this site that a tag cannot back. The award record says 2029. The releases will say whether we earned it.&lt;/p&gt; </content> </entry> <entry> <title type="html">CLIO in two Genesis Mission projects</title> <link rel="alternate" type="text/html" href="https://akougkas.io/journal/2026/clio-in-two-genesis-mission-projects/"/> <id>https://akougkas.io/journal/2026/clio-in-two-genesis-mission-projects/</id> <published>2026-08-27T00:00:00-05:00</published> <updated>2026-11-08T00:00:00-06:00</updated> <summary type="html">Two first-round DOE Genesis Mission projects, PORTUS with Pacific Northwest and SPOTTER-AI with Argonne, build on CLIO. What each asks of it, and why context and provenance matter when agents act on shared infrastructure.</summary> <content type="html" xml:base="https://akougkas.io/journal/2026/clio-in-two-genesis-mission-projects/">&lt;p&gt;In July the Department of Energy &lt;a href=&quot;https://science.osti.gov/-/media/funding/pdf/Awards-Lists/2026/GM-RFA-Awards-List.pdf&quot;&gt;selected&lt;/a&gt; two first-round &lt;a href=&quot;https://www.energy.gov/undersecretaryforscience/genesis-mission/genesis-mission&quot;&gt;Genesis Mission&lt;/a&gt; projects that build on &lt;a href=&quot;/work/clio/&quot;&gt;CLIO&lt;/a&gt;, the context architecture for scientific agents that I created and lead across &lt;a href=&quot;/work/iowarp/&quot;&gt;IOWarp&lt;/a&gt;. GRC started work on both in August. The selection list is careful to say that selection for award negotiations is not a commitment, and I will be just as careful here about what each project asks of CLIO.&lt;/p&gt; &lt;h2 id=&quot;portus&quot;&gt;PORTUS&lt;/h2&gt; &lt;p&gt;&lt;a href=&quot;https://grc.iit.edu/research/projects/portus/&quot;&gt;PORTUS&lt;/a&gt;, Automatic Performance Portability for Distributed Scientific Workflows on the Genesis Mission Platform, is led by &lt;a href=&quot;https://www.pnnl.gov/people/nathan-tallent&quot;&gt;Nathan Tallent&lt;/a&gt; at &lt;a href=&quot;https://www.pnnl.gov/&quot;&gt;Pacific Northwest&lt;/a&gt;. The project builds workflow performance models that guide coding agents, using surrogate models that reason about where a workflow bottlenecks on the platform’s HPC and cloud resources. CLIO supplies the context and data services those agents use. &lt;a href=&quot;https://www.iit.edu/directory/people/xian-he-sun&quot;&gt;Xian-He Sun&lt;/a&gt; and I lead GRC’s part, and &lt;a href=&quot;https://grc.iit.edu/center/team/xiaoyang-lu/&quot;&gt;Xiaoyang Lu&lt;/a&gt; and &lt;a href=&quot;https://grc.iit.edu/center/team/luke-logan/&quot;&gt;Luke Logan&lt;/a&gt; carry the research.&lt;/p&gt; &lt;h2 id=&quot;spotter-ai&quot;&gt;SPOTTER-AI&lt;/h2&gt; &lt;p&gt;&lt;a href=&quot;https://grc.iit.edu/research/projects/spotter-ai/&quot;&gt;SPOTTER-AI&lt;/a&gt;, Scientific Provenance-Oriented Threat Tracing and Attribution for Genesis Workflows, is led by &lt;a href=&quot;https://www.bnicolae.net/&quot;&gt;Bogdan Nicolae&lt;/a&gt; at &lt;a href=&quot;https://www.anl.gov/&quot;&gt;Argonne&lt;/a&gt;, together with &lt;a href=&quot;https://www.ornl.gov/&quot;&gt;Oak Ridge&lt;/a&gt;, HPE, and &lt;a href=&quot;https://www.iit.edu/&quot;&gt;Illinois Tech&lt;/a&gt;. Xian-He and I lead GRC’s work, and &lt;a href=&quot;https://grc.iit.edu/center/team/jaime-cernuda/&quot;&gt;Jaime Cernuda&lt;/a&gt; carries the research. GRC contributes four CLIO components: Core, Kit, Agent, and Coder. It is a Phase I project.&lt;/p&gt; &lt;h2 id=&quot;why-context-and-provenance&quot;&gt;Why context and provenance&lt;/h2&gt; &lt;p&gt;Both projects put agents to work on infrastructure that many teams share. On a laptop, an agent that misreads a dataset or reruns the wrong job wastes an afternoon. On a shared facility, the same mistake spends allocation that belongs to other people, and the question afterwards is who did what, to which data, and on whose authority.&lt;/p&gt; &lt;p&gt;CLIO was built around that question. It gives agents what databases give applications: structured, queryable state about data, jobs, and tools, instead of whatever fits in a prompt. Its contracts decide what an agent can discover, what moves without copying, what persists across sessions and machines, what needs a person to approve, and what leaves a receipt. For PORTUS the first of those matters most, because an agent cannot tune a workflow it cannot see. For SPOTTER-AI the last one does, because tracing a threat to data or results needs a record kept while the work ran, not one reconstructed after it.&lt;/p&gt; &lt;h2 id=&quot;boundary-and-next-step&quot;&gt;Boundary and next step&lt;/h2&gt; &lt;p&gt;&lt;a href=&quot;https://core.iowarp.ai/&quot;&gt;CLIO Core&lt;/a&gt; has public builds on Ares at Illinois Tech, NSF DeltaAI at NCSA, and &lt;a href=&quot;https://www.alcf.anl.gov/&quot;&gt;ALCF&lt;/a&gt; Polaris and Aurora. Neither project has results yet, and I will write the next entry when one does.&lt;/p&gt; </content> </entry> <entry> <title type="html">ChronoLog 3.0: SQL, pub/sub, and an MCP server</title> <link rel="alternate" type="text/html" href="https://akougkas.io/journal/2026/chronolog-3-0-sql-pub-sub-and-an-mcp-server/"/> <id>https://akougkas.io/journal/2026/chronolog-3-0-sql-pub-sub-and-an-mcp-server/</id> <published>2026-05-29T00:00:00-05:00</published> <updated>2026-05-29T00:00:00-05:00</updated> <summary type="html">ChronoLog 3.0.0 adds ChronoSQL, a pub/sub plugin, and an in-tree MCP server, so an agent can record and replay time-ordered activity through the same log a scientific workflow writes to.</summary> <content type="html" xml:base="https://akougkas.io/journal/2026/chronolog-3-0-sql-pub-sub-and-an-mcp-server/">&lt;p&gt;&lt;a href=&quot;/work/chronolog/&quot;&gt;ChronoLog&lt;/a&gt; &lt;a href=&quot;https://github.com/grc-iit/chronolog/releases/tag/v3.0.0&quot;&gt;3.0.0&lt;/a&gt; shipped today. It is a non-prerelease tag with a changelog, which is the bar I hold our systems to before I call something a release. I have been lenient about that bar in the past and regretted it.&lt;/p&gt; &lt;aside class=&quot;entry-glance&quot; aria-label=&quot;At a glance&quot;&gt; &lt;p class=&quot;entry-glance__kicker&quot;&gt;At a glance&lt;/p&gt; &lt;dl&gt; &lt;div&gt;&lt;dt&gt;Anchor&lt;/dt&gt;&lt;dd&gt;ChronoLog v3.0.0, released May 29, 2026&lt;/dd&gt;&lt;/div&gt; &lt;div&gt;&lt;dt&gt;Systems&lt;/dt&gt;&lt;dd&gt;&lt;a href=&quot;/work/chronolog/&quot;&gt;ChronoLog&lt;/a&gt;, &lt;a href=&quot;/work/clio/&quot;&gt;CLIO&lt;/a&gt;&lt;/dd&gt;&lt;/div&gt; &lt;div&gt;&lt;dt&gt;The claim&lt;/dt&gt;&lt;dd&gt;An agent can record and replay time-ordered activity through the same log a scientific workflow writes to.&lt;/dd&gt;&lt;/div&gt; &lt;div&gt;&lt;dt&gt;Code&lt;/dt&gt;&lt;dd&gt;&lt;a href=&quot;https://github.com/grc-iit/chronolog&quot;&gt;grc-iit/chronolog&lt;/a&gt;, &lt;a href=&quot;https://github.com/grc-iit/chronolog/releases/tag/v3.0.0&quot;&gt;release notes&lt;/a&gt;&lt;/dd&gt;&lt;/div&gt; &lt;/dl&gt; &lt;/aside&gt; &lt;h2 id=&quot;what-changed&quot;&gt;What changed&lt;/h2&gt; &lt;p&gt;Three things matter in this one. The rest is API hygiene, which also matters, but I will come back to it.&lt;/p&gt; &lt;p&gt;&lt;strong&gt;ChronoSQL.&lt;/strong&gt; A plugin that answers SQL-style queries over the log. ChronoLog orders records by physical time and stores them as stories inside chronicles; ChronoSQL lets you query them with a statement instead of a replay loop you write yourself.&lt;/p&gt; &lt;p&gt;&lt;strong&gt;ChronoPubSub.&lt;/strong&gt; A pub/sub plugin. The log was always readable at the tail; now a subscriber can be pushed new records rather than poll for them.&lt;/p&gt; &lt;p&gt;&lt;strong&gt;The MCP server, in-tree.&lt;/strong&gt; Until now ChronoLog’s &lt;a href=&quot;https://modelcontextprotocol.io/&quot;&gt;Model Context Protocol&lt;/a&gt; server lived outside the main repository. It has been &lt;a href=&quot;https://github.com/grc-iit/chronolog/tree/v3.0.0/plugins/chrono-mcp&quot;&gt;migrated in&lt;/a&gt;, so it versions with the log it serves. An MCP-capable agent can create a chronicle, record events, replay a story, and run time-range queries through the same interface any other tool in &lt;a href=&quot;https://github.com/iowarp/clio-kit&quot;&gt;CLIO Kit&lt;/a&gt; exposes.&lt;/p&gt; &lt;p&gt;The API cleanup is what a returning user will notice first.&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;CreateChronicle&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AcquireStory&lt;/code&gt; lost unused parameters.&lt;/li&gt; &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ShowChronicles&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ShowStories&lt;/code&gt; return status codes.&lt;/li&gt; &lt;li&gt;Destructive client calls are now actually destructive.&lt;/li&gt; &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ReplayStory&lt;/code&gt; gained a callback variant.&lt;/li&gt; &lt;li&gt;Acquired stories are released on disconnect.&lt;/li&gt; &lt;li&gt;A client identifier is a meaningful service endpoint, and the event sequence is exposed.&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Each was a small paper cut; together they were why the 2.x client felt provisional.&lt;/p&gt; &lt;h2 id=&quot;why-a-log-wants-an-mcp-server&quot;&gt;Why a log wants an MCP server&lt;/h2&gt; &lt;p&gt;An agent working on science produces activity, not only answers: which tool it called, with what arguments, what came back, what it decided next. If that activity goes into a shared log ordered by physical time, another process can replay the session, a person can audit it, and a second agent can pick up where the first one stopped. That is the receipt I keep describing in the &lt;a href=&quot;/work/clio/&quot;&gt;CLIO&lt;/a&gt; work, and ChronoLog is the durable, high-rate place to keep it. &lt;a href=&quot;https://github.com/iowarp/clio-core/releases/tag/v2.0.0&quot;&gt;CLIO Core&lt;/a&gt; reached 2.0.0 two days ago; the two are meant to be used together. I designed the log years before I had an agent to feed it, and the fit is better than I had any right to expect.&lt;/p&gt; &lt;aside class=&quot;entry-pull&quot;&gt; &lt;p&gt;I designed the log years before I had an agent to feed it, and the fit is better than I had any right to expect.&lt;/p&gt; &lt;/aside&gt; &lt;h2 id=&quot;who-did-the-work&quot;&gt;Who did the work&lt;/h2&gt; &lt;p&gt;&lt;a href=&quot;https://grc.iit.edu/center/team/eneko-gonzalez/&quot;&gt;Eneko Gonzalez&lt;/a&gt; authored the release and most of its pull requests: both plugins, the MCP migration, the API cleanup, the README, and the new website and docs. Inna Brodkin at the &lt;a href=&quot;https://www.uchicago.edu/&quot;&gt;University of Chicago&lt;/a&gt; contributed the ChronoPlayer recording service, the pipeline change that decoupled story processing from the extraction queue, and the response transfer agent. The full list is in the &lt;a href=&quot;https://github.com/grc-iit/chronolog/releases/tag/v3.0.0&quot;&gt;release notes&lt;/a&gt;, with a pull request behind every line. My contribution to this release is mostly having stayed out of the way. The &lt;a href=&quot;https://grc.iit.edu/research/projects/chronolog/&quot;&gt;project record at GRC&lt;/a&gt; carries the longer history.&lt;/p&gt; &lt;h2 id=&quot;what-to-try-first&quot;&gt;What to try first&lt;/h2&gt; &lt;p&gt;Pull the &lt;a href=&quot;https://github.com/grc-iit/chronolog/pkgs/container/chronolog&quot;&gt;container&lt;/a&gt; and run the single-node tutorial:&lt;/p&gt; &lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;docker pull ghcr.io/grc-iit/chronolog:latest docker run &lt;span class=&quot;nt&quot;&gt;-it&lt;/span&gt; &lt;span class=&quot;nt&quot;&gt;--rm&lt;/span&gt; ghcr.io/grc-iit/chronolog:latest bash &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt; &lt;p&gt;The &lt;a href=&quot;https://chronolog.dev/docs/&quot;&gt;documentation&lt;/a&gt; has the walkthrough from there, plus &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.deb&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.rpm&lt;/code&gt; packages and a Spack environment for a cluster build. If you have an MCP client already, point it at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;plugins/chrono-mcp&lt;/code&gt; and ask it to replay something.&lt;/p&gt; &lt;p&gt;The commands above are from the v3.0.0 README; I have not yet run the new release on a machine outside the lab. The next release is where I want to see the agent side exercised by someone who is not us.&lt;/p&gt; </content> </entry> <entry> <title type="html">DLIO joins MLPerf Storage</title> <link rel="alternate" type="text/html" href="https://akougkas.io/journal/2024/dlio-joins-mlperf-storage/"/> <id>https://akougkas.io/journal/2024/dlio-joins-mlperf-storage/</id> <published>2024-11-15T11:00:00-06:00</published> <updated>2026-11-12T00:00:00-06:00</updated> <summary type="html">MLPerf Storage now uses DLIO, Hariharan Devarajan&apos;s deep-learning I/O benchmark maintained at Argonne, to generate its workloads. A research tool found a second life.</summary> <content type="html" xml:base="https://akougkas.io/journal/2024/dlio-joins-mlperf-storage/">&lt;p&gt;&lt;a href=&quot;https://github.com/mlcommons/storage&quot;&gt;MLPerf Storage&lt;/a&gt; now uses the &lt;a href=&quot;https://github.com/argonne-lcf/dlio_benchmark&quot;&gt;DLIO Benchmark&lt;/a&gt; to generate the I/O behavior of deep-learning training workloads. That is the whole announcement, and I am delighted by it.&lt;/p&gt; &lt;h2 id=&quot;what-dlio-does&quot;&gt;What DLIO does&lt;/h2&gt; &lt;p&gt;Deep-learning training does not stress storage in one universal way. Datasets, worker counts, epochs, checkpoints, metadata operations, and framework-specific access patterns all change the picture. Running an entire training stack to study each storage design is expensive and hard to reproduce.&lt;/p&gt; &lt;p&gt;DLIO makes that behavior configurable. You describe the pattern you want to exercise and it generates that pattern, repeatably, without the model. The &lt;a href=&quot;https://doi.org/10.1109/CCGrid51090.2021.00018&quot;&gt;CCGrid 2021 paper&lt;/a&gt; explains the design and the experiments behind it; it won the best paper award, which was a good day.&lt;/p&gt; &lt;h2 id=&quot;why-the-mlperf-integration-matters-to-me&quot;&gt;Why the MLPerf integration matters to me&lt;/h2&gt; &lt;p&gt;A benchmark becomes real when a second community uses it to compare systems they built. MLPerf Storage has to exercise storage under defined workload configurations, and its public implementation names DLIO as the generator for those paths. Anyone can inspect the dependency, the configurations, and the rules.&lt;/p&gt; &lt;p&gt;Three things I take from this. DLIO left the environment that produced the paper; other people install it, configure it, and change it without its authors’ private context. Integration sharpened the interfaces, because assumptions that felt harmless in one lab became bugs across submitters’ platforms. And the benchmark now sits in a longer evidence chain: MLPerf defines the process, DLIO produces the workload, a submitter supplies the system.&lt;/p&gt; &lt;h2 id=&quot;credit&quot;&gt;Credit&lt;/h2&gt; &lt;p&gt;DLIO is &lt;a href=&quot;https://grc.iit.edu/center/team/hariharan-devarajan/&quot;&gt;Hariharan Devarajan&lt;/a&gt;’s work, built with &lt;a href=&quot;https://www.alcf.anl.gov/about/people/huihuo-zheng&quot;&gt;Huihuo Zheng&lt;/a&gt; and &lt;a href=&quot;https://www.alcf.anl.gov/about/people/venkatram-vishwanath&quot;&gt;Venkatram Vishwanath&lt;/a&gt; at &lt;a href=&quot;https://www.anl.gov/&quot;&gt;Argonne&lt;/a&gt;; I was part of the &lt;a href=&quot;https://www.iit.edu/&quot;&gt;Illinois Tech&lt;/a&gt; team that co-advised Hari. Its growth into shared infrastructure, including its place in MLPerf Storage, is the work of Hari, the Argonne team, contributors, and the &lt;a href=&quot;https://mlcommons.org/&quot;&gt;MLCommons&lt;/a&gt; Storage community. Being part of MLPerf does not make DLIO a standard by declaration; it means the code was useful enough for another community to build on. That is the outcome I want from research software.&lt;/p&gt; </content> </entry> <entry> <title type="html">NSF funds the next five years of IOWarp</title> <link rel="alternate" type="text/html" href="https://akougkas.io/journal/2024/iowarp-nsf-award/"/> <id>https://akougkas.io/journal/2024/iowarp-nsf-award/</id> <published>2024-09-01T12:00:00-05:00</published> <updated>2026-08-27T00:00:00-05:00</updated> <summary type="html">A five-year NSF award for IOWarp, the collaboration with The HDF Group and the University of Utah that grew out of Hermes.</summary> <content type="html" xml:base="https://akougkas.io/journal/2024/iowarp-nsf-award/">&lt;p&gt;Thrilled to share that the National Science Foundation has funded &lt;a href=&quot;/work/iowarp/&quot;&gt;IOWarp&lt;/a&gt; for five years, 2024 through 2029. It is a collaborative award among &lt;a href=&quot;https://www.iit.edu/&quot;&gt;Illinois Tech&lt;/a&gt;’s &lt;a href=&quot;https://grc.iit.edu/&quot;&gt;Gnosis Research Center&lt;/a&gt;, &lt;a href=&quot;https://www.hdfgroup.org/&quot;&gt;The HDF Group&lt;/a&gt;, and the &lt;a href=&quot;https://www.utah.edu/&quot;&gt;University of Utah&lt;/a&gt;, and it is the largest single research investment my group has been part of. The details are in the &lt;a href=&quot;https://www.nsf.gov/awardsearch/show-award/?AWD_ID=2411318&quot;&gt;NSF record for OAC-2411318&lt;/a&gt;. I wrote the proposal, serve as a co-PI, and lead the architecture.&lt;/p&gt; &lt;h2 id=&quot;what-we-are-building&quot;&gt;What we are building&lt;/h2&gt; &lt;p&gt;IOWarp bridges HPC, AI, and data analytics with one idea: scientific workflows need a context layer, not only a faster byte pipe. The runtime moves data across storage tiers, reshapes it across formats, and exposes it to agents and people through tools with explicit contracts. It builds directly on &lt;a href=&quot;/work/hermes/&quot;&gt;Hermes&lt;/a&gt;, our multi-tier buffering system, and on years of work on &lt;a href=&quot;/work/labios/&quot;&gt;labels&lt;/a&gt;, tracing, and &lt;a href=&quot;/work/dtio/&quot;&gt;task-based I/O&lt;/a&gt;.&lt;/p&gt; &lt;p&gt;The original announcement mentioned a specialized language model for natural-language data exploration. The plan evolved. What shipped instead is the CLIO ecosystem: a context store, a tool kit of MCP servers, a science agent, and a coding agent, which you can read about in the &lt;a href=&quot;/work/clio/&quot;&gt;CLIO story&lt;/a&gt;.&lt;/p&gt; &lt;h2 id=&quot;what-the-award-changes&quot;&gt;What the award changes&lt;/h2&gt; &lt;p&gt;Five years is enough time to do systems work properly: architecture, interfaces, implementation, integration, evaluation, documentation, releases, and time for people outside the lab to find the assumptions we missed. It also means students can own substantial pieces of the platform rather than isolated experiments.&lt;/p&gt; &lt;p&gt;An award is runway. The proof is in releases, and that is what the &lt;a href=&quot;https://github.com/iowarp&quot;&gt;IOWarp organization&lt;/a&gt; has been accumulating since.&lt;/p&gt; </content> </entry> <entry> <title type="html">Hermes is complete</title> <link rel="alternate" type="text/html" href="https://akougkas.io/journal/2024/hermes-is-complete/"/> <id>https://akougkas.io/journal/2024/hermes-is-complete/</id> <published>2024-08-16T00:00:00-05:00</published> <updated>2024-08-16T00:00:00-05:00</updated> <summary type="html">Hermes v1.2.1 is the final release of a project that started as the center of my PhD and ran for five years with The HDF Group. What the multi-tier buffering bet proved, what it did not, and what moves into IOWarp.</summary> <content type="html" xml:base="https://akougkas.io/journal/2024/hermes-is-complete/">&lt;p&gt;&lt;a href=&quot;https://github.com/HDFGroup/hermes/releases/tag/v1.2.1&quot;&gt;Hermes v1.2.1&lt;/a&gt; is out, and it is the last release of the project. The NSF award that funded it, &lt;a href=&quot;https://www.nsf.gov/awardsearch/show-award/?AWD_ID=1835764&quot;&gt;OAC-1835764&lt;/a&gt;, ran from 2018 through 2023, and the project record closes at the end of this month. &lt;a href=&quot;/work/hermes/&quot;&gt;Hermes&lt;/a&gt; started as the center of my PhD, became a collaborative Frameworks project with &lt;a href=&quot;https://www.hdfgroup.org/&quot;&gt;The HDF Group&lt;/a&gt; in September 2018, and I have led its technical work from the first design to this tag. I want to mark the ending properly rather than let it fade. Most research software dies by neglect, and I would rather this one had a date.&lt;/p&gt; &lt;aside class=&quot;entry-glance&quot; aria-label=&quot;At a glance&quot;&gt; &lt;p class=&quot;entry-glance__kicker&quot;&gt;At a glance&lt;/p&gt; &lt;dl&gt; &lt;div&gt;&lt;dt&gt;Anchor&lt;/dt&gt;&lt;dd&gt;Hermes v1.2.1, tagged August 16, 2024, closing NSF award OAC-1835764&lt;/dd&gt;&lt;/div&gt; &lt;div&gt;&lt;dt&gt;Systems&lt;/dt&gt;&lt;dd&gt;&lt;a href=&quot;/work/hermes/&quot;&gt;Hermes&lt;/a&gt;, &lt;a href=&quot;/work/iowarp/&quot;&gt;IOWarp&lt;/a&gt;&lt;/dd&gt;&lt;/div&gt; &lt;div&gt;&lt;dt&gt;The claim&lt;/dt&gt;&lt;dd&gt;Middleware can present memory, NVMe, burst buffers, and the parallel file system as one hierarchy and place data on the application&apos;s behalf.&lt;/dd&gt;&lt;/div&gt; &lt;div&gt;&lt;dt&gt;Code&lt;/dt&gt;&lt;dd&gt;&lt;a href=&quot;https://github.com/HDFGroup/hermes&quot;&gt;HDFGroup/hermes&lt;/a&gt;&lt;/dd&gt;&lt;/div&gt; &lt;/dl&gt; &lt;/aside&gt; &lt;h2 id=&quot;what-changed&quot;&gt;What changed&lt;/h2&gt; &lt;p&gt;The bet in the &lt;a href=&quot;/publications/kougkas2018hermes/&quot;&gt;HPDC 2018 paper&lt;/a&gt; was that middleware could present a machine’s storage tiers, memory, local NVMe, burst buffers, and the parallel file system, as one buffering hierarchy, and make placement decisions on the application’s behalf from workload behavior, device capability, capacity, and intent. The application would keep writing POSIX, &lt;a href=&quot;https://www.hdfgroup.org/solutions/hdf5/&quot;&gt;HDF5&lt;/a&gt;, or MPI-IO. Hermes would intercept the calls and decide where the bytes went.&lt;/p&gt; &lt;figure class=&quot;figure--diagram&quot;&gt; &lt;svg viewBox=&quot;0 0 640 360&quot; role=&quot;img&quot; aria-labelledby=&quot;fig-hermes-tiers&quot;&gt; &lt;title id=&quot;fig-hermes-tiers&quot;&gt;The Hermes placement engine sits above four storage tiers, memory, NVMe, burst buffer, and parallel file system, and decides where each request lands.&lt;/title&gt; &lt;rect class=&quot;d-box&quot; x=&quot;120&quot; y=&quot;40&quot; width=&quot;400&quot; height=&quot;64&quot; rx=&quot;4&quot; /&gt; &lt;text class=&quot;d-text&quot; x=&quot;320&quot; y=&quot;78&quot; text-anchor=&quot;middle&quot;&gt;Hermes placement engine&lt;/text&gt; &lt;path class=&quot;d-accent&quot; d=&quot;M320 104 V150&quot; /&gt; &lt;path class=&quot;d-accent&quot; d=&quot;M96 150 H544&quot; /&gt; &lt;path class=&quot;d-accent&quot; d=&quot;M96 150 V190&quot; /&gt; &lt;path class=&quot;d-accent&quot; d=&quot;M245 150 V190&quot; /&gt; &lt;path class=&quot;d-accent&quot; d=&quot;M395 150 V190&quot; /&gt; &lt;path class=&quot;d-accent&quot; d=&quot;M544 150 V190&quot; /&gt; &lt;polygon class=&quot;d-accent-fill&quot; points=&quot;96,196 89,184 103,184&quot; /&gt; &lt;polygon class=&quot;d-accent-fill&quot; points=&quot;245,196 238,184 252,184&quot; /&gt; &lt;polygon class=&quot;d-accent-fill&quot; points=&quot;395,196 388,184 402,184&quot; /&gt; &lt;polygon class=&quot;d-accent-fill&quot; points=&quot;544,196 537,184 551,184&quot; /&gt; &lt;rect class=&quot;d-soft&quot; x=&quot;24&quot; y=&quot;200&quot; width=&quot;144&quot; height=&quot;60&quot; rx=&quot;4&quot; /&gt; &lt;text class=&quot;d-text&quot; x=&quot;96&quot; y=&quot;236&quot; text-anchor=&quot;middle&quot;&gt;memory&lt;/text&gt; &lt;rect class=&quot;d-soft&quot; x=&quot;173&quot; y=&quot;200&quot; width=&quot;144&quot; height=&quot;60&quot; rx=&quot;4&quot; /&gt; &lt;text class=&quot;d-text&quot; x=&quot;245&quot; y=&quot;236&quot; text-anchor=&quot;middle&quot;&gt;NVMe&lt;/text&gt; &lt;rect class=&quot;d-soft&quot; x=&quot;323&quot; y=&quot;200&quot; width=&quot;144&quot; height=&quot;60&quot; rx=&quot;4&quot; /&gt; &lt;text class=&quot;d-text&quot; x=&quot;395&quot; y=&quot;236&quot; text-anchor=&quot;middle&quot;&gt;burst buffer&lt;/text&gt; &lt;rect class=&quot;d-soft&quot; x=&quot;472&quot; y=&quot;200&quot; width=&quot;144&quot; height=&quot;60&quot; rx=&quot;4&quot; /&gt; &lt;text class=&quot;d-text&quot; x=&quot;544&quot; y=&quot;236&quot; text-anchor=&quot;middle&quot;&gt;parallel FS&lt;/text&gt; &lt;text class=&quot;d-kicker&quot; x=&quot;24&quot; y=&quot;316&quot;&gt;one hierarchy, policy-driven placement&lt;/text&gt; &lt;/svg&gt; &lt;figcaption&gt;Hermes intercepts POSIX, HDF5, and MPI-IO calls and places each request across memory, NVMe, burst buffers, and the parallel file system from workload behavior, device capability, capacity, and intent, so the application sees one buffering hierarchy.&lt;/figcaption&gt; &lt;/figure&gt; &lt;p&gt;That bet held, which I say with some relief, since I made it as a student. Policy-driven placement works, the adapters intercept real applications without code changes, and the distributed metadata and RDMA-capable runtime that coordinate placement across nodes ran on &lt;a href=&quot;https://www.anl.gov/&quot;&gt;Argonne&lt;/a&gt;’s &lt;a href=&quot;https://www.alcf.anl.gov/alcf-resources/theta&quot;&gt;Theta&lt;/a&gt; and &lt;a href=&quot;https://www.ornl.gov/&quot;&gt;Oak Ridge&lt;/a&gt;’s &lt;a href=&quot;https://www.olcf.ornl.gov/summit/&quot;&gt;Summit&lt;/a&gt; as well as our own cluster. In this site’s bibliography, nineteen papers name Hermes as a project, which is a fair measure of how much research the architecture carried, and the &lt;a href=&quot;https://grc.iit.edu/research/projects/hermes/&quot;&gt;project record at GRC&lt;/a&gt; lists them with the team.&lt;/p&gt; &lt;p&gt;&lt;a href=&quot;https://grc.iit.edu/center/team/hariharan-devarajan/&quot;&gt;Hariharan Devarajan&lt;/a&gt;’s &lt;a href=&quot;/publications/devarajan2020hfetch/&quot;&gt;HFetch&lt;/a&gt;, &lt;a href=&quot;/publications/devarajan2020hcompress/&quot;&gt;HCompress&lt;/a&gt;, and &lt;a href=&quot;/publications/devarajan2020hreplica/&quot;&gt;HReplica&lt;/a&gt; each had to fit inside the same placement model, and the fact that they did is the strongest evidence that the model was right.&lt;/p&gt; &lt;p&gt;What did not hold was my sense of how much of the work was the algorithm. I was wrong in the direction every systems student is wrong. The placement engine was a small fraction of the final system.&lt;/p&gt; &lt;p&gt;Most of the five years went into the things a paper lets you skip: distributed metadata that survives node failure, adapters that behave under every I/O pattern a real application produces, packaging and documentation that let a stranger install it, and the runtime &lt;a href=&quot;https://grc.iit.edu/center/team/luke-logan/&quot;&gt;Luke Logan&lt;/a&gt; built to keep all of it coordinated. That runtime outgrew Hermes and is now a thing of its own.&lt;/p&gt; &lt;p&gt;The partnership mattered. &lt;a href=&quot;https://www.hdfgroup.org/team/gerd-heber/&quot;&gt;Gerd Heber&lt;/a&gt; was the HDF Group’s co-PI, and the software lives under &lt;a href=&quot;https://github.com/HDFGroup/hermes&quot;&gt;their organization&lt;/a&gt;, which is why it has a home that outlasts the grant. A research group’s GitHub is a poor archive. An organization whose business is keeping a format alive for decades is a better one.&lt;/p&gt; &lt;aside class=&quot;entry-pull&quot;&gt; &lt;p&gt;Most research software dies by neglect, and I would rather this one had a date.&lt;/p&gt; &lt;/aside&gt; &lt;h2 id=&quot;boundary-and-next-step&quot;&gt;Boundary and next step&lt;/h2&gt; &lt;p&gt;Complete means complete. There will not be a v1.3. I have been asked, and a clear no is a kindness to anyone deciding whether to build on it. Issues will be answered when someone has time, which is an honest description of most research software after its funding ends. The v1.2.1 tag is the state I would point anyone at, and I have written elsewhere about why a named final state is worth more than a quietly abandoned main branch.&lt;/p&gt; &lt;p&gt;Two things I would do differently, and I would tell any student starting a five-year system the same. I would make observability first-class from the start. Placement decisions are easy to trust when a user can see why they happened and hard to trust when they cannot, and we added the seeing late. And I would design the packaging and upgrade path alongside the runtime from day one instead of treating it as the work after the interesting work.&lt;/p&gt; &lt;p&gt;The ideas do not stop. Earlier this month NSF funded &lt;a href=&quot;/work/iowarp/&quot;&gt;IOWarp&lt;/a&gt;, and its transfer engine is Hermes’s direct descendant: the same separation between interface, runtime, and heterogeneous storage, the same attention to placement, with a much larger question attached about what moves through the hierarchy when the consumer is an AI workflow rather than a simulation. Hermes has a last release. The architecture is just getting started.&lt;/p&gt; </content> </entry> <entry> <title type="html">DOE funds DTIO</title> <link rel="alternate" type="text/html" href="https://akougkas.io/journal/2023/doe-funds-dtio/"/> <id>https://akougkas.io/journal/2023/doe-funds-dtio/</id> <published>2023-08-07T00:00:00-05:00</published> <updated>2026-09-06T00:00:00-05:00</updated> <summary type="html">The Department of Energy has funded DTIO, a three-year project with Argonne to turn I/O requests into schedulable data tasks. It is what LABIOS labels become when a pipeline has more than one stage.</summary> <content type="html" xml:base="https://akougkas.io/journal/2023/doe-funds-dtio/">&lt;p&gt;The Department of Energy’s Office of Advanced Scientific Computing Research has funded DTIO for three years, from August 2023 through June 2026. The award is DE-SC0024593, collaborative between &lt;a href=&quot;https://www.iit.edu/&quot;&gt;Illinois Tech&lt;/a&gt;’s &lt;a href=&quot;https://grc.iit.edu/&quot;&gt;Gnosis Research Center&lt;/a&gt; and &lt;a href=&quot;https://www.anl.gov/&quot;&gt;Argonne National Laboratory&lt;/a&gt;. I am a co-PI, and DTIO is the project I proposed, so I will be carrying the architecture. I have wanted to write this proposal since the LABIOS paper, and I am relieved someone agreed the question was worth asking. The &lt;a href=&quot;https://grc.iit.edu/research/projects/dtio/&quot;&gt;project record at GRC&lt;/a&gt; carries the award number and the team.&lt;/p&gt; &lt;aside class=&quot;entry-glance&quot; aria-label=&quot;At a glance&quot;&gt; &lt;p class=&quot;entry-glance__kicker&quot;&gt;At a glance&lt;/p&gt; &lt;dl&gt; &lt;div&gt;&lt;dt&gt;Anchor&lt;/dt&gt;&lt;dd&gt;DOE ASCR award DE-SC0024593, August 2023 through June 2026&lt;/dd&gt;&lt;/div&gt; &lt;div&gt;&lt;dt&gt;Systems&lt;/dt&gt;&lt;dd&gt;&lt;a href=&quot;/work/dtio/&quot;&gt;DTIO&lt;/a&gt;, &lt;a href=&quot;/work/labios/&quot;&gt;LABIOS&lt;/a&gt;&lt;/dd&gt;&lt;/div&gt; &lt;div&gt;&lt;dt&gt;The claim&lt;/dt&gt;&lt;dd&gt;Treating every I/O request as a schedulable data task lets format translation happen online, while the producer is still writing.&lt;/dd&gt;&lt;/div&gt; &lt;div&gt;&lt;dt&gt;Code&lt;/dt&gt;&lt;dd&gt;&lt;a href=&quot;https://github.com/grc-iit/DTIO&quot;&gt;grc-iit/DTIO&lt;/a&gt;&lt;/dd&gt;&lt;/div&gt; &lt;/dl&gt; &lt;/aside&gt; &lt;h2 id=&quot;what-changed&quot;&gt;What changed&lt;/h2&gt; &lt;p&gt;&lt;a href=&quot;/work/dtio/&quot;&gt;DTIO&lt;/a&gt; stands for Data Task I/O. The idea is short to state. Today an HPC simulation writes &lt;a href=&quot;https://www.hdfgroup.org/solutions/hdf5/&quot;&gt;HDF5&lt;/a&gt;, a Python analysis reads &lt;a href=&quot;https://numpy.org/&quot;&gt;NumPy&lt;/a&gt;, and a training job wants &lt;a href=&quot;https://pytorch.org/&quot;&gt;PyTorch&lt;/a&gt; tensors. Each stage speaks a different I/O dialect, and the usual bridge is an offline conversion that has to finish before the next stage can start. The conversion is pure overhead, and it serializes a pipeline that ought to overlap. I think that wait is a design choice, not a law of nature.&lt;/p&gt; &lt;p&gt;DTIO treats every I/O request as a task instead of a file. A data task carries an operation, its input and output formats, ordering constraints, and dependencies, which means a runtime can compose it with other tasks, schedule it near the data, and execute it asynchronously on the right storage tier.&lt;/p&gt; &lt;p&gt;Format translation happens online, while the producer is still writing. Provenance is recorded per task so a failed partition can be replayed rather than recomputed. Because tasks can run inside the storage layer, the same machinery gives us computational storage without asking applications to change.&lt;/p&gt; &lt;p&gt;If that sounds like &lt;a href=&quot;/work/labios/&quot;&gt;LABIOS&lt;/a&gt;, it should. A LABIOS label binds an operation to a data reference and routing metadata so the system can inspect and schedule it. A data task is a label with formats, dependencies, and ordering attached. &lt;a href=&quot;https://grc.iit.edu/center/team/keith-bateman/&quot;&gt;Keith Bateman&lt;/a&gt; has been developing that abstraction in his doctoral research, and DTIO is where it gets a full runtime around it.&lt;/p&gt; &lt;p&gt;The Argonne side brings &lt;a href=&quot;https://www.bnicolae.net/&quot;&gt;Bogdan Nicolae&lt;/a&gt; and &lt;a href=&quot;https://franckcappello.github.io/&quot;&gt;Franck Cappello&lt;/a&gt;, whose work on resilience and fault tolerance is exactly what a task-based runtime needs once it starts replaying partial failures. DTIO will be built with &lt;a href=&quot;https://www.alcf.anl.gov/aurora&quot;&gt;Aurora&lt;/a&gt; in mind from the first commit. &lt;a href=&quot;https://grc.iit.edu/center/team/neeraj-rajesh/&quot;&gt;Neeraj Rajesh&lt;/a&gt;, &lt;a href=&quot;https://grc.iit.edu/center/team/jaime-cernuda/&quot;&gt;Jaime Cernuda&lt;/a&gt;, and &lt;a href=&quot;https://grc.iit.edu/center/team/luke-logan/&quot;&gt;Luke Logan&lt;/a&gt; round out the team at GRC.&lt;/p&gt; &lt;aside class=&quot;entry-pull&quot;&gt; &lt;p&gt;A proposal is a bet made in public, and the honest thing is to name the result that would lose it.&lt;/p&gt; &lt;/aside&gt; &lt;h2 id=&quot;boundary-and-next-step&quot;&gt;Boundary and next step&lt;/h2&gt; &lt;p&gt;Nothing has shipped. The &lt;a href=&quot;https://github.com/grc-iit/DTIO&quot;&gt;repository&lt;/a&gt; exists, the design is on paper, and the first target is a six-stage pipeline: intercept, compose, decompose, schedule, execute, record. The claim I most want to test is that online translation with task-level caching beats offline conversion on a real producer-consumer workflow, not only on a microbenchmark. If it does not, the abstraction is wrong and I would rather learn that in the first year. A proposal is a bet made in public, and the honest thing is to name the result that would lose it.&lt;/p&gt; &lt;p&gt;The bigger question is whether a task is the right unit for the whole data path, from parallel file system to training loop, or only for the seams between stages. &lt;a href=&quot;/work/hermes/&quot;&gt;Hermes&lt;/a&gt; taught me that placement decisions are only trustworthy when you can see why they happened. A task that carries its own provenance is my current answer to how you make that visible. I expect to revise that answer.&lt;/p&gt; &lt;h2 id=&quot;update-september-2026&quot;&gt;Update, September 2026&lt;/h2&gt; &lt;p&gt;The project closed in June 2026. Keith’s &lt;a href=&quot;/publications/bateman2025dtio/&quot;&gt;SSDBM 2025 paper&lt;/a&gt; measured the online translation bet on the PtychoNN workflow, and the mechanisms were reimplemented inside &lt;a href=&quot;https://core.iowarp.ai/&quot;&gt;CLIO Core&lt;/a&gt;, the runtime that &lt;a href=&quot;/work/iowarp/&quot;&gt;IOWarp&lt;/a&gt; ships. The &lt;a href=&quot;/work/dtio/&quot;&gt;DTIO story&lt;/a&gt; has the results and where the code went.&lt;/p&gt; </content> </entry> </feed>