Your Agent Isn’t Confused. It’s Timing Out.

The first thing everyone does when an agent run fails is read the transcript looking for the moment the model got confused. Sometimes that is what happened. Usually it is not.

In the runs I have actually traced, the failure is somewhere else entirely: a tool call that hit its deadline and got retried against a non-idempotent endpoint, a worker that came back after the planner had already moved on, or a context assembly step that dropped the middle of the history because it did not fit and told nobody. The model behaved reasonably given what it was handed. What it was handed was wrong.

The maths nobody does

Start with the part that is not a judgement call. If a step succeeds with probability p, and a run is n steps long, the run finishes with probability pⁿ. That is it. That is the whole model.

The reason this catches people is that p looks excellent right up until you raise it to a power:

A 99% tool call, which most people would call reliable and stop optimising, gets you a 67% completion rate over forty steps. Push the run to eighty steps and you are below half. No amount of prompt engineering moves that number, because it is not a prompting problem.

I should be honest that pⁿ is the optimistic version. It assumes every step is as likely to succeed as the first one, and at least one careful measurement says otherwise: per-step accuracy actually degrades as a run gets longer, so the real curve falls away faster than the power law does (Sinha et al.). The same paper makes the cheerful version of the point, that small gains in single-step accuracy compound into large gains in how long a task you can finish. Both halves are worth holding at once: the arithmetic below is a floor you will not beat, not a forecast.

If you want measured numbers rather than a model, τ-bench is the uncomfortable one. Even strong function-calling agents finish well under half of its tasks, and running the same task eight times drops the success rate under 25% in the retail domain (Yao et al.).¹¹That second number, which the authors call pass^8, is the one worth internalising. It is not measuring whether a model can do the task. It is measuring whether it can do it reliably, which is a different and much harder question, and the gap between the two is most of what makes agents hard to ship.

An agent’s reliability is the product of its steps, not the quality of its reasoning. You cannot prompt your way out of exponentiation.

All of which sounds like bad news, and it reads as bad news, and it is actually the most encouraging paragraph in this post. Because an exponent cuts both ways.

The same arithmetic, read forwards

Take the 99% call that gets you 67% over forty steps, and ask what it would take to run eighty steps at that same 67%.

The answer is 99.5%. Not 99.9, not some unreachable number of nines. Halve the per-step failure rate from 1% to 0.5% and you have doubled the length of task your agent can finish. Halve it again to 0.1% and you are at four hundred steps, still finishing 67% of them.

Each halving of the failure rate doubles the horizon. That is the same exponent, read in the direction that pays.

And it appears to be what has actually been happening. METR timed people on real software tasks and then asked how long a task a model can complete with 50% success, which they call the 50% time horizon. That horizon has been doubling roughly every seven months since 2019. Crucially, when they look at what is driving it, the answer is not mainly reasoning. It is greater reliability and a better ability to recover from mistakes (Kwa et al.).²²The paper is careful about its own limits, and so am I in citing it: the tasks are drawn from RE-Bench, HCAST and 66 shorter ones, which is software work and not everything, and the authors flag that the trend may have accelerated in 2024 rather than held steady. The direction is the claim I am leaning on, not the slope.

So the outside measurement and the inside arithmetic agree, and they point at the same engineering. The thing that buys you a longer run is not a better prompt. It is a smaller per-step failure rate, which is the boring stuff: retries, idempotency, deadlines, and not lying about what got truncated.

Retries are the cheapest of those by a distance. One retry per step takes that 67% to 99.6%, and it costs almost nothing, because a retry only fires on a call that already failed. They are the highest-leverage thing in an agent runtime and they are usually the last thing built.

There is a catch, and it is a big one.

So retries have to be safe

A retry is only free if the call is idempotent, meaning it does no harm when run twice, and most interesting tool calls are not. Take send_email, create_ticket, execute_trade or POST /orders. Retry any of those against a backend that is slow but alive and you haven’t recovered from a failure. You’ve caused one.

The fix is the one distributed systems settled on decades ago and agent frameworks are still rediscovering. HTTP’s own specification is blunt about it: a client should not automatically retry a non-idempotent request unless it has some way to know the semantics are actually idempotent, or some way to detect the original never landed (RFC 9110). Stripe’s idempotency keys are the same idea with an implementation attached, letting you repeat a request after a connection error without creating a second object (Stripe).

async def call_tool(tool, args, *, step_id, attempt):
    # The key is derived from what the step IS, not from when it ran,
    # so a retry of the same logical step reuses the same key and the
    # server can collapse the duplicate. Including `attempt` here (a
    # mistake I have now made twice) defeats the entire mechanism.
    key = idempotency_key(run_id, step_id, tool.name, canonical(args))

    if cached := await results.get(key):
        return cached                      # someone already did this

    result = await tool.invoke(args, idempotency_key=key, timeout=tool.deadline)
    await results.put(key, result)
    return result

If a tool cannot offer that, the runtime has to know. Marking tools as retryable: false and letting those steps fail loudly is a worse completion rate and a much better system, because the alternative is a duplicate side effect nobody notices for a week.

Safe, though, is only half of it. A retry that is perfectly safe can still take down the thing you are retrying against.

And then they have to be counted

Google’s SRE book has the arithmetic, and it is the second multiplication in this post that people consistently get wrong.

Suppose the database is overloaded and returning errors. The backend retries three times. The frontend, not knowing that, retries three times. The JavaScript in the browser retries three times. Four attempts at each of three layers, and one user action arrives at the database as sixty-four attempts, because the layers multiply rather than add (Google SRE).

Every layer there is behaving correctly in isolation. Every layer is also making the outage worse, at the exact moment the database is failing because it is overloaded.

Now count the layers in an agent stack. The planner may re-plan a failed step. The agent runtime retries the tool call. The vendor SDK retries on 429 and 5xx, usually by default and usually without telling you. The HTTP client underneath it has its own policy. That is four before the request has left your process, and the planner’s retry is the sneaky one, because it is semantic rather than transport-level, so nothing lower in the stack can see it as a duplicate at all.

The fix in the SRE book is a server-wide retry budget: allow some fixed number of retries per minute in a process, say sixty, and when the budget is gone, stop retrying and fail. The run still fails. It just stops taking the tool down with it, which turns a global cascading failure into some dropped queries.

For an agent runtime the same rule has a specific shape. Decide which single layer owns retrying, give it a budget, and turn retries off everywhere else. If your SDK retries by default, that is a layer you did not choose and did not count.

That leaves the question of when a call counts as failed in the first place, which is a number somebody picked out of the air.

A timeout is a choice you’re making

I spent years on consensus protocols arguing that a failure detector’s timeout is a policy decision wearing a constant’s clothing. I now make the same argument about TOOL_TIMEOUT_S = 30 roughly once a month.

Thirty seconds is a claim: I would rather abandon a call that would have returned at thirty-one seconds than wait longer. Sometimes that is right. But it should be derived from the tool’s observed latency distribution, not from the fact that thirty is a round number:

class ToolDeadline:
    """A deadline that names the trade it is making."""
    def __init__(self, p99: float, slack: float = 2.5, floor: float = 2.0):
        self.p99, self.slack, self.floor = p99, slack, floor

    def seconds(self) -> float:
        return max(self.p99 * self.slack, self.floor)

slack: 2.5 invites the question “why two and a half?” in a way that 30 never does. That is the entire benefit, and it is worth more than it sounds, because the number that gets reviewed is the number that gets fixed.

Here is the whole argument running. One planner, five workers, real tool calls at about a fortieth of real speed. Nothing is scripted: calls time out on their own, back off, and give up after three attempts. Time a tool out yourself and watch what the retry policy buys, and what it costs against the budget.

Every failure in that simulation is at least honest about being a failure. The last one isn’t.

Cutting context is a failure too

The one that took me longest to see. When the context assembler cannot fit the history, most implementations drop the middle and continue. The model then answers confidently from a history with a hole in it, and the trace shows a reasoning failure.

It is not a reasoning failure. It is a partial failure that was silently converted into a successful-looking call. Every other layer of the stack would have raised something. This one returns a plausible string.

It is made worse by where models are weakest. Performance is highest when the relevant information sits at the beginning or end of the context and drops off when the model has to reach into the middle, even in models sold on their long context (Liu et al.). So the middle is both the part your assembler is most likely to drop and the part the model was already worst at reading.

That is a genuinely unlucky coincidence, and it means a truncation is not a neutral loss of some fraction of the history. It is a targeted removal of the material the model would have struggled with anyway, replaced with nothing, with no indication that anything is missing.

Treat it the way you would treat a truncated RPC response:

• Count it. A truncation rate is a health metric. If you cannot report yours, you have one.
• Fail the step, do not soften it. Then let the retry path summarise, re-rank, or split the work, deliberately and with a record.
• Never drop the middle silently. Whatever you evict, the model should be told it was evicted.

What I’d do about it

• Compute pⁿ before optimising anything. If the arithmetic says the run cannot finish, no prompt will save it, and you need fewer steps rather than better ones.
• Chase the failure rate, not the success rate. Going from 99% to 99.5% sounds like a rounding error and is a doubling of how long a task you can finish. That is the number to put on a dashboard.
• Make retries the default and idempotency the requirement. A tool that cannot be safely retried is a tool that caps your completion rate; that should be visible in the tool registry, not discovered in production.
• Pick one layer to own retrying and give it a budget. Then go and turn off the retries in the other three, including the one in your SDK you did not know about.
• Derive every deadline from a measured distribution. A constant is a claim about latency you have not measured.
• Instrument truncation like packet loss. It is packet loss.

None of this is about models, and none of it is new. It is the same list you would write for any pipeline of unreliable stages calling each other over a network. That’s exactly what an agent is, with an unpredictable worker in the middle and a much better demo.

Sources

Research

• Kwa, T. and others. Measuring AI Ability to Complete Long Software Tasks, arXiv:2503.14499, 2025.
• Sinha, A. and others. The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs, arXiv:2509.09677, 2025.
• Yao, S., Shinn, N., Razavi, P. and Narasimhan, K. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, arXiv:2406.12045, 2024.
• Liu, N. F. and others. Lost in the Middle: How Language Models Use Long Contexts, arXiv:2307.03172, 2023.

Standards and documentation

• Fielding, R., Nottingham, M. and Reschke, J. HTTP Semantics, RFC 9110, section 9.2.2, 2022.
• Stripe. Idempotent requests, API reference.
• Beyer, B. and others. Addressing Cascading Failures, Site Reliability Engineering, chapter 22, Google, 2016.