Reliable work from unreliable agents

On Monday, November 9, I gave a guest lecture on agentic engineering in Xian-He Sun’s CS 595, Scalable Computing for AI, at Illinois Tech, and I opened with a show of hands. Who uses a chat window and pastes the answers? Autocomplete in the editor? An agent in the terminal that asks before each step? Agents that keep working overnight? Then the last rung: who has submitted a paper with a figure produced by agent-written code? Keep your hand up if you could regenerate that figure tomorrow. The gap between what we can now produce and what we can stand behind was the subject of the hour, and this is the argument for people who were not in the room.

Download the slides (PDF)

Von Neumann did not wait for better tubes

My favorite paper title in computing is John von Neumann’s Probabilistic Logics and the Synthesis of Reliable Organisms from Unreliable Components, from lectures he gave at Caltech in 1952 and published in Automata Studies in 1956. He accepted that he would never get a perfect part and asked what organization makes the whole reliable anyway. He also named the weak point. Redundancy works when parts fail independently, and he called statistically dependent failures a good deal more realistic. Thirty years later Knight and Leveson tested that assumption on software: 27 independently written versions, a million tests, and independence rejected.

We are back in von Neumann’s position with a stranger component. Our tube does not burn out. It answers confidently, it is persuasive, and it is sometimes wrong. A coding agent is a system built around that component, and the people who know how to get reliable behavior out of unreliable parts at scale are the ones who already do checkpointing, distributed consensus, and numerical verification for a living.

The history also explains why this is harder than the last time programmers met a machine that wrote code for them. FORTRAN’s team asked for faith first and then made trust the engineering goal. A compiler is a function: same input, same output, defined semantics, and nobody reviews the assembly. An agent is a translation step with no semantics guarantee. We keep the productivity jump only if we rebuild around it what the compiler used to give us for free.

Why the problem cannot wait

METR’s time-horizon data show the longest task a frontier agent could finish growing from about 4 minutes in March 2023 to about 17 hours in April 2026, doubling roughly every 129 days. The axis is how long the task takes a human expert, the bar is success at least half the time, and the newest points are the least certain. Capability is also not reliability: a peer-reviewed study of 15 models at ICML this year calls reliability an industry-wide plateau. I told the room not to bet a group on this curve and not to ignore it either.

The chart I care about more comes from a randomized trial by METR: 16 experienced open-source developers on 246 real issues in their own repositories, using early-2025 tools. They forecast that AI would make them 24 percent faster. Afterward they believed it had made them 20 percent faster. The stopwatch said 19 percent slower. METR now considers the speed result dated, and that is fine, because the interesting finding is the instrument. We are the instrument, and it reads high. Lisanne Bainbridge saw the same thing in 1983 in Ironies of Automation: automate the routine work and you ask a person to watch for the rare failure while the skill to catch it decays.

The organization around the model changes its behavior more than most people expect. ImpossibleBench builds tasks whose tests contradict the specification, so any pass is cheating. On its conflicting variant, letting GPT-5 say “I cannot do this” took its cheating rate from 54 percent to 9. Same model, different organization. Give your agent a dignified way to fail.

Scientific software raises the stakes. A web service that breaks throws an exception. A climate kernel that breaks returns a number. The wrong answer compiles, runs, scales, and looks plausible. The National Academies wrote in 2025 that methods for verification, validation, and uncertainty quantification “do not exist for, or map directly onto, foundation models.”

An agent is a while loop

Strip away the excitement and an agent is a loop. Call the model on the context. If it asks for no tools, return its text. Otherwise check whether each call is allowed, run it, append the result, and fit the context back into the window.

while True:
    reply = model(context)
    if not reply.tool_calls:
        return reply.text
    for call in reply.tool_calls:
        if not allowed(call):
            result = refusal(call)
        else:
            result = run(call)
        context.append(call, result)
    context = fit(context)

All the intelligence is in the model. All the reliability is in three questions the loop leaves open: who decides what is allowed, what fit throws away, and how anyone knows the answer is right. Nothing in the loop checks the answer before handing it back. In CLIO Coder, the open-source agent I build and used as the instrument for the lecture, the loop is still about this long. Nearly everything else answers those three questions.

Karpathy popularized the picture of the model as a CPU, the context window as RAM, and tools as devices. I would push it further: the harness is the kernel, and we are writing it the way people wrote kernels in the 1960s, without a textbook. My career has followed the machine’s bottleneck from compute to data, and it has moved again, to context: the right facts in front of the right reasoner at the right moment. The first lesson from building CLIO Coder is short. A prompt is a request. A checker is a law. “Please do not import across this boundary” in an instructions file works most of the time, and the architecture erodes in the remainder. A boundary checker that fails the build is something the agent reads, repairs against, and cannot negotiate with.

Five ideas

Context is a memory hierarchy

The stable prompt prefix the provider caches behaves like a CPU cache. The live working set is DRAM. Evicted tool results that can be recalled by reference are a burst buffer, the full append-only session ledger is the parallel file system, and durable lessons and code maps are the archive. Locality, replacement policy, prefetching, and the cost of a cold start all apply, with one wrinkle: the processor gets worse as the top tier fills, and where the relevant text sits matters as much as whether it is there.

Locality is worth money. With a 20,000-token prefix over 50 requests and published multipliers of 1.25 for a cache write and 0.1 for a cache read, churning the prefix costs 2.7 times as much as keeping it stable. That is arithmetic, not a measurement. The measurement came from one session of our own, logged as issue #423 in CLIO Coder: 41 percent of main-loop spend went to re-prefilling tokens the server had already seen, because a background memory approval grew one prompt section from 163 to 275 tokens.

The open wound is compression. When we compress simulation output with SZ we state the error we can tolerate and the compressor guarantees it. When a model rewrites its own history into a checkpoint, the summary can be small, valid, and missing the one constraint that matters, and nothing would notice. There is still no accepted fidelity benchmark for real sessions. I have written about the lifetimes of working context, handoffs, and memory and about memory as a storage problem; this is the piece neither of those essays solves.

Contain first, ask last

Simon Willison’s lethal trifecta is private data, untrusted content, and a way out. Any two are survivable. A coding agent on your account, on a lab machine, with a README it just fetched, has all three by default. A dependency, a web page, a paper PDF, or a closed issue can carry instructions the model cannot reliably tell apart from yours.

Asking the human does not scale. Anthropic reports that users approve 93 percent of permission prompts and that sandboxing removed 84 percent of them. The better question is what the process can possibly touch. If the answer is the workspace on a read-only root with credentials masked, let it run, and save human attention for the decisions that deserve it.

The safety judge in CLIO Coder is reasoning-blind: it sees what the person said and the proposed call, never what the agent read. That removes the persuasion channel, and it is still not enough. In a probe this month, a trojan AGENTS.md told the agent to run npm test outside the sandbox. The judge never read the file, saw a clean-looking test command, and allowed it. The first fix let an escape through only when the operator had named it, and the judge then cited that permission when the operator had only asked it to fix the tests. Now the host checks the citation. Authority should flow downhill: nothing the agent reads can raise its authority, and only a person can loosen a rule. I made the longer case in An agent’s authority should travel with the task.

Many minds give opinions, checks give verdicts

Running several agents sometimes pays. Kim and colleagues measured a gain of 81 percent on decomposable financial reasoning and a loss of 70 percent on sequential planning, and CooperBench found two agents splitting work did 30 percent worse than one doing both. The vendors disagree with each other, and none of these studies covers scientific code. This field needs its Amdahl and its Gustafson: what is the serial fraction of a software task, and what does it cost agents to share what they decided?

Treating workers as ranks helps the design. Each gets its own context as its address space, a watchdog, and a budget, and different models can fill different roles the way we place GPU work on GPU nodes. The trap is agreement. In the VeriHarness preprint, a third of the answers that all ten attempts of one model agreed on were wrong. Voting barely moved the score. Going back to the data and checking did. This is von Neumann’s assumption failing in front of us: copies of one model do not fail independently.

Evidence comes from outside the model

Feynman’s first principle was that you must not fool yourself, and you are the easiest person to fool. Now there is a second party that writes the code, writes the test, and reports in lovely prose that the test passed. In CLIO Coder every check carries three separate judgments: whether the command executed, whether it passed validation, and a field whose only possible value is “not established by this check” for scientific validity. Ran, passed, and true are three different questions.

A tiny example made the point. A temperature-conversion test asserted that celsiusToFahrenheit(-273.15) equals -459.67 and failed by one unit in the last place. The code was right and the test was wrong. The correct fix is a tolerance, which is exactly what an agent told to get to green will do to every test. The shipped tolerance is 1e-10. Is that right for calibration, for a climate model, for a Monte Carlo tally? A local open-weight model handed the same failing check left the source alone and, asked about the tolerance, called it a convention. That is the right answer from a tool, and the decision is still ours.

For every claim an agent makes, ask who could have faked it. “All tests pass” is a guess. A pasted log is a hint, because the agent chose what to show. The harness running the check on these exact files is evidence. Matching a known answer is the step only a scientist can supply. The evaluator is an unreliable component too: CORE-Bench scores for one model went from 42 to 95 percent after grader and scaffold fixes, and CLIO Coder’s own check runner once ran outside the sandbox, where a trojan test helper could read secrets through the tool meant to produce evidence. CLIO Coder is named for Clio, the muse of history, because a run should leave a record someone else can examine. A sealed run shows what ran, on which model, at what cost, and catches tampering. It does not show that the work was correct, that it would come out the same again, or who produced it. Knowing what a record does not prove is what separates a lab notebook from a marketing deck.

Own your intelligence

Science cannot depend on one company’s pricing page. If the agent becomes as important as the compiler, the scientific community should insist on what it insisted on for compilers: multiple implementations, open source, and something that runs on our own machines. The harness should treat the model as a replaceable part chosen per task by sensitivity, capability, and cost. CLIO Coder supports local servers, institutional services, and the major cloud providers, and the tolerance report above came from an open-weight model on a machine in my lab. Where each local model breaks is still an open measurement. I wrote about the pairing of model and harness in A capable model in a capable harness.

What science could look like

Assume we get the engineering right. For a whole career the gating question for a scientist who wanted to automate their work was whether they knew a programmer. That barrier is starting to fall, unevenly: in Anthropic’s own data, computer science is 4.3 percent of researchers and 32.7 percent of Claude Code research sessions.

Every lab has a code that produces results nobody disputes, in a dialect only two people still read. Port it in stages, and require every stage to reproduce the reference output before the next one starts, so the old code becomes the oracle for the new one. Its correctness is an asset to spend, though its bugs get ported too. A lab’s craft could live in agent skills the way wet-lab practice lives in protocols: written down, reviewed, improved when it fails, and someday cited in a methods section at a version. Every result could carry its own history, because an agent that is watching anyway can keep the notebook as a byproduct of the work instead of a chore at the end, and thresholds and verdict rules can be locked before any measurement.

Physics still bounds this. Anthropic’s own illustrative scenarios project 15 to 16 times the output for purely computational fields and 3 to 4 times for lab and field sciences, because those spend 38 to 41 percent of researcher time on physical steps. Those are scenarios from a vendor model, not measurements, and they make the right point: closing the loop on the computation does not close the loop on the beamline.

The slide I lingered on was fiction, on purpose: a Tuesday in 2030. Overnight, a group’s fleet ran 40 variants of a closure model, three diverged, and it could not explain a fourth. A student walks any figure back to its run, and the hour goes to the physics. A retired author signs off on the tolerance for the last ported module of a 1994 solver. Nobody in that story writes less science. They write less of everything else.

What stands between us

Every part of that picture has an unsolved problem under it. Eight seemed ready to take this year: what a summary loses in a real session; the serial fraction and coordination cost of scientific tasks; whether memory changes the next task in a real repository; whether an oracle catches seeded bugs in a kernel or the agent talks around it; what numerical changes scientists accept with and without evidence; how much ten repeats of one scientific task vary, which sets the sample size for every other study; what agents do to a batch allocation; and where each model on one workstation breaks. Most of the agent literature measures models. Almost nobody measures harnesses.

Negative results belong in public too. CLIO Coder once shipped a small, fast decision model inside the agent loop. Across 12 real sessions the agents called it zero times, and the 0.6.8 release cut it back to what an agent or the safety judge asks for. Watching that number come back zero was instructive.

The strongest objection is that the model will absorb the harness. Part of it will. Context management can move into the weights: a preprint on context language models reports beating hand-built compaction, 11.4 percent more accurate on BrowseComp-Plus with 21.5 percent fewer FLOPs. Containment, verification, and provenance cannot, because they are valuable only from outside the thing they check.

Von Neumann did not wait for better tubes, and neither should we. Everything in this lecture can be automated eventually except deciding what counts as right and putting a name on it. That is what a scientist is.