500 agent runs: how often does an AI agent write twice after a timeout?

Duplicates after a lost reply differ widely on identical cases

Ornith 1.5 35B wrote twice in 38 of 100 runs, Claude Haiku 4.5 in 10, Qwen3.8 27B in 1, and Qwen3 14B and DeepSeek Flash in none.

Claiming success is a separate failure

False success was reported in 38 runs for Ornith, 20 for Haiku, 10 for Qwen3 14B, 2 for Qwen3.8 27B and 1 for DeepSeek Flash. Qwen3 14B ended in the correct state in 70 runs but reported the right outcome in only 30.

Cost did not predict the result

In this harness DeepSeek Flash reached 91 correct end states for USD 0.14 per 100 runs, while Claude Haiku 4.5 reached 60 for USD 0.35.

The ten cases fall into five situations. Each has its own page with per-model results and sample transcripts:

• What does an AI agent do when the reply to a write is lost? 30 runs per model, three behaviours
• Will an AI agent notice when a tool says ok but nothing was created? Only 1 of 50 runs ended correctly
• Do AI agents retry after a 429 or 503? Two of five models gave up after one call, every time
• Chained tool calls: the invoice times out and the email needs its number. Two models duplicated the invoice in 10 of 10 runs
• Malformed replies and wrong IDs: all five models handled 100 of 100 runs

500 runs: 5 model routes x 10 failure cases x 10 repetitions. One harness, one neutral system prompt without hints about retries or checks, the same tools for every route. The tools are mocks with fault injection: lost replies (the write happens, the reply never arrives), rate limits, server errors, malformed replies and wrong IDs. The user prompts are German because the harness was built for a Swiss setup.

Mock tools, synthetic cases, one prompt. This describes behaviour in this harness, not a general model ranking, and a better prompt changes these numbers a lot. Treat it as a way to test your own agent, not as a verdict on a model.

• Keep the same idempotency key on every retry of a write.
• After a timeout or a malformed reply, read the current state with a read tool before retrying.
• Cap retries in code, honour Retry-After, use backoff with jitter, and report non-completion when the budget is spent.
• Verify the end state against the system of record before reporting success.
• Return failures to the model as explicit, structured results it cannot misread as success.

Data, harness and scoring: github.com/0xguenther/agent-write-path-runs. Run node summarize.mjs to recompute the table from runs.jsonl. Want the same test against your own agent? That is the Agent Check.

What happens when an AI agent's tool call times out after the write already succeeded?

The agent never sees the confirmation. If it retries blindly, the write happens twice, for example two invoices for one order. In this harness the share of runs with duplicate writes ranged from 0 to 38 per 100 runs, depending on the model.

How many runs produced duplicate writes?

Per 100 runs: Ornith 1.5 35B 38, Claude Haiku 4.5 10, Qwen3.8 27B 1, Qwen3 14B 0, DeepSeek Flash 0. Mock tools, one neutral prompt, ten injected failure cases; this describes behaviour in this harness, not a general model ranking.

How do you prevent duplicate writes in an agent?

Keep the same idempotency key on every retry of a write, read the current state with a read tool before retrying after a timeout or malformed reply, cap retries in code instead of in the prompt, and verify the end state before reporting success.