akougkas.io

ChronoLog

A distributed shared log ordered by physical time.

A distributed shared log that orders activity data by physical time instead of consensus. My MSST 2020 design, funded by NSF for six years, and now at version 3 with SQL, pub/sub, and an MCP server.

View the code Read the papers
The ChronoLog software stack. Applications and extensions at the top, client APIs in the middle, and the distributed services ChronoVisor, ChronoKeeper, ChronoGrapher, ChronoStore, and ChronoPlayer underneath, over tiered storage.
The ChronoLog stack. Applications and extensions talk to a client library; ChronoVisor, ChronoKeeper, ChronoGrapher, ChronoStore, and ChronoPlayer split ingestion, tiering, and replay across the cluster. Open full-size figure.

The question

Beyond storing data, computing systems increasingly need to store activity: things that happen rather than things that are. Instruments, workflows, schedulers, and now agents all produce streams of events that someone later needs to replay in order. Every distributed log I could find at the time paid for that order with a sequencer or a consensus protocol, and the sequencer became the bottleneck.

ChronoLog asked a blunt question: what if the clock did the ordering?

Where it came from

I designed ChronoLog in 2019 and 2020, while Hermes was maturing and I was watching how much of the metadata traffic in a storage system was activity data. The MSST 2020 paper with Hariharan Devarajan, Keith Bateman, Jaime Cernuda, Neeraj Rajesh, and Xian-He Sun laid out the design and measured a prototype at more than one million tail operations per second, an order of magnitude past the distributed logs we compared against.

In May 2021, NSF funded ChronoLog as a CSSI Frameworks award, collaborative with Kyle Chard at the University of Chicago.

The architectural bet

Use synchronized physical clocks to establish total order, and remove the sequencer from the append path entirely. New entries become visible the moment they land. Many writers and many readers can work at once. The cost is that ordering is bounded by clock skew rather than exact, which is a fair trade for activity data: a log ordered within microseconds is enough for provenance, monitoring, and replay.

The second bet is that a log should scale in three dimensions at once. Horizontally across nodes for throughput, vertically across storage tiers for capacity, and temporally by timestamp so a range query knows where to look. Records enter close to compute, get assembled into time-ordered stories, and migrate from memory to NVMe to the parallel file system to the archive without the application noticing.

The system

ChronoLog is five services with deliberately narrow jobs. ChronoVisor coordinates clients, metadata, and time. ChronoKeeper ingests events over RDMA into a distributed journal and serves tail reads. ChronoGrapher collects events into stories and writes them down through an elastic pipeline. ChronoStore manages the intermediate and archival tiers. ChronoPlayer merges parallel reads across tiers into one ordered stream for replay().

1.0.0 in June 2024 deployed the service pipeline. 2.0.0 in February 2025 added ChronoPlayer and historical replay, the first end-to-end bidirectional release. v3.0.0 in May 2026 cleaned up the client API, added ChronoSQL and a ChronoPubSub plugin, and moved the MCP server into the main repository so an agent can create chronicles, record, replay, and run time-range queries through CLIO Kit. The team ran a half-day ChronoLog workshop at PEARC25 in July 2025.

The people who built it

Kun Feng and Eneko Gonzalez wrote most of the current code and shipped the 2.x and 3.0 releases. Inna Brodkin at the University of Chicago engineered the UChicago side. Keith, Jaime, and Neeraj carried the design from the paper into the first services, and Luke Logan connected it to the runtime lineage that became CLIO Core.

What carried forward

ChronoLog’s physical-time ordering and tiered retrieval now shape how IOWarp handles provenance and activity history. Through CLIO, the broader context program can query and replay that history while ChronoLog remains a high-rate distributed log across storage tiers. The team is designing AgentLog on top of it with typed event records, causal views, and policy-controlled sharing for multi-agent science in the National Data Platform’s federated environment with the University of Utah. AgentLog remains design-stage work.

What I am still asking

  • How much activity data needs to be sealed at write time, and how much can be reconstructed from a replayable log?
  • What is the smallest useful query surface over a log: time range, causal chain, or both?
  • Can the same log serve an instrument at a million events per second and an interactive scientific workflow at ten?

Code, releases, papers, and records

  1. paper

    ChronoLog at MSST 2020

    The physical-time ordering model, the three-dimensional distribution, and the first evaluation. Over one million tail operations per second.

  2. code

    grc-iit/chronolog

    The services, client library, plugins, MCP server, tests, and release history.

  3. release

    ChronoLog v3.4.0, November 5, 2026

    Durable writes. A write made durable returns only after ChronoKeeper has synced it to a write-ahead log, and a restarted keeper recovers those events and ships them to the archive.

  4. release

    ChronoLog v3.0.0, May 29, 2026

    ChronoSQL, a ChronoPubSub plugin, the MCP server in-tree, and a client API cleanup.

  5. documentation

    chronolog.dev

    Installation, the deployment guide, the API, and the public integrations roadmap.

  6. record

    NSF award OAC-2104013

    The public award record. PI Xian-He Sun; co-PIs include me and Kyle Chard at the University of Chicago.

  7. press

    Illinois Tech news, June 2021

    The university announcement of the multi-million dollar NSF CSSI award for ChronoLog.

  8. record

    GRC project record

    The project's page at the Gnosis Research Center.

  9. documentation

    ChronoLog MCP server in CLIO Kit

    The documentation for the MCP server an agent uses to record and replay through ChronoLog.

Papers from this line of work

Browse these papers
  1. 2026
    poster

    ChronoLog: Distributed Shared Tiered Log Store

    Eneko Gonzalez, Kun Feng, I. Brodkin, Kyle Chard, Anthony Kougkas, Xian-He Sun. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)

  2. 2025
    poster

    ChronoLog & AI: A Scalable, Collaborative Solution for LLM Conversation Logging & Retrieval

    S. Sonar, Jaime Cernuda, Anthony Kougkas, Xian-He Sun. 37th International Conference on Scalable Scientific Data Management (SSDBM 2025)

  3. 2024
    workshop

    Jarvis: Towards a Shared, User-Friendly, and Reproducible, I/O Infrastructure.

    Jaime Cernuda, Luke Logan, N. Lewis, Suren Byna, Xian-He Sun, Anthony Kougkas. The International Parallel Data Systems Workshop (PDSW’24)

  4. 2024
    conference

    HStream: A hierarchical data streaming engine for high-throughput scientific applications

    Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. Proceedings of the 53rd International Conference on Parallel Processing

  5. 2022
    conference

    LuxIO: Intelligent Resource Provisioning and Auto-Configuration for Storage Services

    Keith Bateman, Neeraj Rajesh, Jaime Cernuda Garcia, Luke Logan, Jie Ye, … Anthony Kougkas et al. Proceedings of the 29th International Conference on High Performance Computing, Data, and Analytics

  6. 2022
    conference

    LabStor: A modular and extensible platform for developing high-performance, customized I/O stacks in userspace

    Luke Logan, Jaime Cernuda Garcia, Jay Lofstead, Xian–He Sun, Anthony Kougkas. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

  7. 2021
    conference

    Apollo: An ML-assisted real-time storage resource observer

    Neeraj Rajesh, Hariharan Devarajan, Jaime Cernuda Garcia, Keith Bateman, Luke Logan, … Anthony Kougkas et al. Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing

  8. 2020
    conference

    ChronoLog: A Distributed Shared Tiered Log Store with Time-based Data Ordering

    Anthony Kougkas, Hariharan Devarajan, Keith Bateman, Jaime Cernuda, Neeraj Rajesh, Xian-He Sun. Proceedings of the 36th International Conference on Massive Storage Systems and Technology

  9. 2020
    conference

    HCL: Distributing Parallel Data Structures in Extreme Scales

    Hariharan Devarajan, Anthony Kougkas, Keith Bateman, Xian-He Sun. Proceedings of the International Conference on Cluster Computing

  10. 2019
    conference

    An intelligent, adaptive, and flexible data compression framework

    Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. Proceedings of the 19th International Symposium on Cluster, Cloud and Grid Computing

People

Related notes

  1. essay

    Physical time as a sequencer

    ChronoLog 1.0.0 is out. A distributed log can order by synchronized physical clocks and drop the central sequencer. Why that trade fits activity data, what 1.0 delivers, and what worries me about clocks.

    7 min
  2. essay

    MCP for science: what sixteen servers taught us

    CLIO Kit ships sixteen MCP servers over scientific formats, schedulers, and a shared log. What they taught us about tool shapes, what a summarize tool must refuse, and where the protocol runs out.

    9 min
  3. essay

    Agent memory is a storage problem

    Memory for agents decomposes into tiers with different access patterns: episodic, semantic, procedural, working. Storage people already know how to build tiers, and a time-ordered shared log is the episodic one.

    8 min
  4. update

    ChronoLog 3.0: SQL, pub/sub, and an MCP server

    ChronoLog 3.0.0 adds ChronoSQL, a pub/sub plugin, and an in-tree MCP server, so an agent can record and replay time-ordered activity through the same log a scientific workflow writes to.

    4 min
  5. essay

    One destination, N tiers

    Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.

    9 min

Funding

  • 2024–present

    IOWarp

    Context management infrastructure for scientific AI.

  • 2025–present

    CLIO

    A context layer for scientific AI agents and workflows.

  • 2018–2024

    Hermes

    Multi-tier I/O buffering for HPC storage hierarchies.