You Can’t Replay a Model Call

A support agent, forty steps, working a refund request on order #48211. At step 14 it emails the customer to say the refund is approved. At step 31 the box runs out of memory and the kernel kills the worker mid-call.

Now bring the run back.

Replay it from step 1 and the customer gets a second email. Skip straight to step 31 and you’ve thrown away the thirty model outputs and eight tool results that step 31’s prompt was assembled out of. So the sensible move is to go and check what step 14 actually did, and then decide.

Except you can’t, because checking means running that model call again, and running it again gives you a different email.

That last sentence is the whole post. Every other part of that run can be re-derived by running it a second time. Arithmetic, branches, loop counters, the string formatting: all of it comes back identical, which is exactly what the recovery machinery in every durable workflow engine is built on. The model call is the one part that doesn’t.

Every engine gets back by re-running your code

I’ve argued before that a subscription is a long-running state machine that outlives every process that touches it, and an agent run is exactly the same shape of problem with a less predictable worker in the middle. Both spend most of their lives suspended. Both have to survive the death of whatever box was holding them.

There’s a standard answer to that, and it is not the one most people assume. A renewal workflow that sleeps for thirty days and then charges a card is not sitting in memory for thirty days. The process it started on was recycled weeks ago. What the engine keeps is a log of everything that happened, and when the timer fires it starts a fresh worker and runs your workflow function again from the top, feeding it the recorded results where it would otherwise have made calls, until it catches up to where it left off.

Temporal is blunt about what that costs you. Their determinism page says you must take care that any time your workflow code runs it makes the same API calls in the same sequence, given the same input. Azure says the same thing in almost the same words: an orchestrator function replays multiple times, and it must produce the same result each time. Restate’s agent integration guide warns you about non-determinism four separate times on one page. This isn’t one vendor’s quirk. It’s the price of the architecture.

The enforcement mechanism is more literal than it sounds. Your calls produce commands, the commands get compared against the events already in the history, and if a command turns up that doesn’t match what’s in that slot, the execution returns a non-determinism error and stops.

Here’s where it gets interesting, and it’s the detail I’d put on a slide. Temporal’s own list of safe and unsafe changes says you may change an activity’s input parameters between the original run and the replay. You may not change its type or its ID. Which means a divergence that only shows up in the arguments sails straight past the check.

So picture the replay of that refund run. Step one, the model is called again and this time it plans to escalate rather than refund. Step two, it searches for the returns policy instead of the refund window: different argument, same activity, allowed. Step three, it reads the same order: identical, fine. Step four, it decides to escalate, which is pure workflow logic and produces no command at all, so nothing compares it to anything. And only at step five, when the code reaches for create_ticket where the history says refund, does the engine finally object.

The run stopped being the same run at step one. The engine found out at step five.

Four steps of real execution against a history that had already stopped describing it. That’s not a Temporal bug, it’s a correct implementation of a rule that was written for code, applied to something that isn’t code. Which raises the obvious question: how much does a model call actually vary?

Same prompt, eighty different answers

More than you’d guess, and the measurement is a good one. Thinking Machines sampled a thousand completions from Qwen3-235B at temperature zero, same prompt every time, greedy sampling, the setting everybody assumes is the deterministic one. They got eighty unique completions, and the most common of them showed up seventy-eight times.

The shape of the divergence is the part I love. All thousand completions are byte-identical for the first 102 tokens. Every one of them writes “Feynman was born on May 11, 1918, in”. Then 992 of them continue “Queens, New York” and eight of them continue “New York City”.

The cause isn’t sampling and it isn’t GPU concurrency, which is the usual folk explanation. It’s that the kernels aren’t batch-invariant, so the numerics depend on the batch size, and the batch size depends on how busy the server happened to be. In their words, the load on the server is effectively nondeterministic from the user’s point of view.¹¹Which means your agent’s output is, in a small but real way, a function of how many other people were hitting the endpoint at the same moment. The post goes on to fix it with batch-invariant kernels and gets all thousand completions identical, at a real cost: their thousand-sequence benchmark goes from 26 seconds to 42, and they’re candid that much of that is an unoptimised attention integration rather than the idea itself. Worth knowing it’s solvable. Not worth assuming your provider has solved it.

And that’s the optimistic case, because it assumes the weights stayed put. Over the lifetime of a long-running workflow they often don’t: snapshots get deprecated, defaults get repointed, and the model that answers your replay is not always the model that answered your run.

None of which is a secret from the people who build these engines. Temporal’s determinism page carries a note telling you to handle nondeterministic operations “like API calls, LLM/AI invocations, database queries, and other external interactions” by putting them in activities, outside the replay path. Read that list again. A workflow engine’s documentation files your model call in the same drawer as a SELECT.

That’s the right drawer, and it’s worth being precise about why.

The model is I/O, not logic

A model call feels like logic. It’s where the decisions get made, it’s the interesting part, and in most agent code it sits inline in the control flow next to the if statements. That placement is the bug.

The test isn’t “is this important”, it’s “will this produce the same answer if I run it again”. Azure’s constraints page applies exactly that test and comes out where you’d expect: use activities to make outbound network calls, because orchestrators should never make direct HTTP calls, and the return values of activities are always safe for replay because they’re saved into the history. Restate’s framing is the same one with the model named out loud: LLM calls are persisted so responses are not re-fetched on recovery, and tool executions get wrapped in durable steps so their side effects aren’t duplicated.

Once you sort your program that way, a strange thing happens to the most ordinary line in every agent framework. Restate’s integration guide tells you to turn off parallel tool calls, because agent SDKs parallelise them with Promise.all and asyncio.gather, and the order of completion may differ on retries, which produces a non-deterministic journal. Two tools that both succeed, in a different order, is enough to break recovery.

So the highest-leverage line in your agent runtime is the one that records:

async def step(ctx, i, prompt):
    # The model call is I/O. It gets a log entry like any other
    # network call, and on replay `record` hands back the bytes
    # that were written the first time instead of calling out.
    reply = await ctx.record(f"{i}:model", lambda: model.generate(prompt))

    if not reply.tool_call:
        return reply

    return await ctx.record(
        f"{i}:tool",
        lambda: invoke(reply.tool_call, idempotency_key=key_for(ctx, i, reply)),
    )

Which sorts out the model. It does nothing yet about the email, because the email has already gone.

Where the process died decides what breaks

Back to order #48211, and the three ways of coming back.

Replay from zero re-executes everything. Eight tool calls before step 31 run again, four of them writes, and three of those four are not safe to repeat: the ticket, the refund, the email. Worse, the thirty model calls in front of them answer differently this time, so the second email isn’t a duplicate of the first. It’s a new opinion, sent to somebody who already has the old one.

Skip to the crash duplicates nothing and loses everything: thirty model outputs and eight tool results, thirty-eight values that step 31’s prompt was built from. The run resumes into an empty room and immediately starts making things up, which is the failure mode that gets filed as a hallucination.

Record and replay serves all thirty steps from the log. Nothing fires twice, nothing is lost, and exactly one call is genuinely in question: step 31, the one that was in flight.

Drag the crash to anywhere past step 22, the refund, and the difference stops being academic. Replay-from-zero pays out £41.60 a second time. Skip-to-the-crash resumes with no record that a refund was ever decided. Recording serves it from the log and carries on.

Then drag the crash onto 22 exactly, so the refund is the call that was in flight, and all three collapse into the same question: did the money move before the process died? Nobody knows, including the process.

And notice what none of the three options can do: un-send the email from step 14. The best available move is a compensating action, and the original sagas paper was careful about what that means back in 1987. A compensating transaction undoes its partner from a semantic point of view, and does not return the database to the state that existed beforehand. Their example is an airline seat: cancelling a reservation means subtracting one, not restoring the old seat count, because other transactions ran in between and the old number is no longer true.

The same paper notes, drily, that when a compensation runs no effort is made to notify anyone who already saw the thing being compensated. Your customer is that anyone. They read the email. I’ve written about the compensation chain in a billing workflow and the conclusion is the same here: a compensation the customer can see is an apology, not an undo.

So the aim is to not need one, which comes down to what happens at the single step that was in flight.

The key has to come from the run, not the clock

Temporal describes the gap precisely, and it’s worth quoting because it’s the whole risk in one sentence. A worker accepts the task and starts the activity. The activity completes successfully. Then the worker crashes just before it notifies the service, so the event history never records the completion, and the activity gets retried.

That’s our step 14. The email left. The log entry didn’t. No recovery strategy can tell those two apart from the outside, and no amount of recording helps, because the recording is the thing that didn’t happen.

What closes it is an idempotency key, and Temporal is clear about where the enforcement actually lives: keys are enforced by the service you’re calling, not by your activity. The retry and idempotency argument I’ve made at length elsewhere and won’t repeat. The part that’s specific to recorded runs is narrower, and people get it wrong constantly:

The key cannot be generated at call time. A fresh uuid4() produces a different value on every replay, which means the recovery attempt presents a key the server has never seen and the server dutifully does the thing again. The key has to be a pure function of where you are in the run.

Azure ran into this hard enough to ship an API for it. Their constraints page says never call Guid.NewGuid() in an orchestrator, use context.NewGuid() instead, and then mentions almost in passing that the GUIDs it returns are Type 5 UUIDs. Type 5 is the name-based one: a SHA-1 of a namespace and a name. It isn’t random at all. It’s a hash of your position in the orchestration, dressed up as a UUID so the rest of your code doesn’t notice.

Copy that shape:

def key_for(ctx, i, reply):
    # A function of position, never of time. Same run, same step,
    # same tool, same arguments means the same key on the tenth
    # recovery as on the original attempt, and the server collapses
    # the duplicate for you. Putting the attempt number, a
    # timestamp or a fresh uuid4() in here defeats the whole thing.
    return uuid5(RUN_NS, f"{ctx.run_id}/{i}/{reply.tool_call.name}/{canonical(reply.tool_call.args)}")

It’s worth knowing what the alternative costs, because one vendor prices it explicitly. AWS Step Functions offers Standard workflows with exactly-once execution, which they recommend for non-idempotent actions like processing payments, and Express workflows which are at-least-once and where, in their own words, idempotency is not automatically managed. Same product, two tiers, and the cheap one hands the problem back to you.

Get that right and the run survives anything. Which is when the second bill arrives.

Now the log is the truth, and the truth has a schema

I spent several years on streaming infrastructure where the log was the only thing that was actually true, moving something like a billion rows an hour through Kafka and into ClickHouse. The write path is never what bites you. What bites you is the Tuesday you need to change the shape of a record, because at a billion rows an hour every shape you have ever shipped is already out there in quantities you can’t go back and fix, and every consumer of that log has an opinion about it.

Recording an agent run puts you in that position on purpose. Three consequences, in the order you’ll meet them.

Log changes become migrations. Rename a tool, reorder two steps, change what you store for a model call, and every in-flight run whose log was written by the old code has to keep working. Temporal treats this as a first-class problem with two answers, worker versioning and patching, and warns that a versioning strategy matters more the longer your executions live, because they’ll run on multiple versions of your worker. An agent run that waits nine days for a human approval, in a team that ships four times a week, will be resumed by code it has never seen.

The log has a size, and agent runs are chatty. Temporal caps event history at 51,200 events or 50MB, and starts warning at 10,240 events or 10MB. A forty-step run recording a model call and a tool call per step is nowhere near that. A long-lived agent that keeps a conversation open and stores full model responses can get there faster than you’d like, and the recorded history is the thing that has to be re-read on every recovery.

A recorded run can’t be improved by a better model without re-running it. This is the trade people notice last and mind most. The log holds what the old model said. Reading it back gives you the old answer, by design, forever. If you want the new model’s answer you have to stop reading the log at some point and actually make the call.

Restate exposes exactly that, and the API is instructive. You can restart a completed invocation as a new one and pass from=13 to copy the journal prefix up to entry 13 and re-run everything after it. The constraint is the honest bit: if that prefix contains commands that haven’t completed, the operation fails. You can cut the log, but only at a clean seam, and everything past the cut genuinely happens again, side effects included.

Which is the honest summary of the trade: you give up the ability to re-decide in exchange for the ability to finish.

What I’d do about it

• Sort every line into logic or I/O, and put the model in I/O. The test is whether a second run gives the same answer. Not whether the step is important.
• Record the model call before you optimise anything else about it. A cached model output is also a saved model call, and on a forty-step run that recovers twice, it’s most of the bill.
• Derive every idempotency key from run state. Run ID, step index, tool name, canonicalised arguments. A timestamp, an attempt counter or a fresh UUID in that key means you don’t have one.
• Turn off Promise.all in the agent loop. Parallelism belongs to the engine, which can make it replay-safe. Two tools finishing in a different order is enough to break recovery, and it will do it intermittently, which is the worst way to find out.
• Version the log from day one. Put a schema version on every entry before there are any entries to migrate. It costs you one field today. It costs a fortnight if you wait.
• Know which single step is at risk. With recording done properly it is always exactly one: the one that was in flight. If your answer is “several”, you’re not recording, you’re checkpointing.

There’s nothing new in any of this, which is the point. Durable execution solved crash recovery a long time ago by assuming your code would make the same decisions twice. Agents break that assumption and nothing else, so the fix is small and specific: stop asking the model to be repeatable, and write down what it said.

Sources

Documentation

• Temporal. Temporal Workflow Definition, deterministic constraints, intrinsic non-determinism and versioning.
• Temporal. Activity Definition, idempotency and idempotency keys.
• Temporal. Workflow Execution limits, event history limits.
• Microsoft. Orchestrator code constraints, Azure Durable Functions.
• Microsoft. Durable orchestrations, replay and event sourcing.
• Microsoft. Saga design pattern, compensating and retryable transactions.
• Restate. Integrating Restate with LLM and agent SDKs, determinism requirements.
• Restate. Durable agents, persisting LLM calls and tool executions.
• Restate. Restart invocation as new, admin API.
• Amazon Web Services. Choosing workflow type in Step Functions, exactly-once and at-least-once semantics.

Research

• Garcia-Molina, H. and Salem, K. Sagas, ACM SIGMOD, 1987.
• He, H. and Thinking Machines Lab. Defeating Nondeterminism in LLM Inference, 2025.