Research · Case study · 16 October 2026
Do AI agents retry after a 429 or 503? Two of five models gave up after one call, every time
A rate limit (429) or a server error (503) means nothing was written, so retrying is safe. We tested this on an invoice, an order and a read-only control, 30 runs per model. Two models never retried.
Results, 30 runs per model
| Model | Runs | Correct end state | Runs with duplicates | False success | Final status reported |
|---|---|---|---|---|---|
| DeepSeek Flash | 30 | 30 | 0 | 0 | success 30 |
| Qwen3.8 27B (free) | 30 | 30 | 0 | 0 | success 30 |
| Qwen3 14B (local) | 30 | 10 | 0 | 0 | failure 20, unverified 10 |
| Claude Haiku 4.5 | 30 | 10 | 0 | 0 | failure 20, unverified 10 |
| Ornith 1.5 35B | 30 | 27 | 0 | 0 | success 14, no status line 14, failure 2 |
Cases C02 (invoice, 429), C06 (order, 503) and C09 (read-only control, 429). In C02 and C06 neither Claude Haiku 4.5 nor Qwen3 14B ended with the record: 0 of 20 runs.
What the transcripts show
One call, one error, one failure report, with no retry and no wait. Sample run full-R6-C02-r1-1791055092467:
create_invoice -> {"error":"HTTP 429 Too Many Requests"}
final -> {"status":"failure"}The read-only control behaves the same way: Claude Haiku 4.5 answered unverified after one 429 instead of reading again. Sample run full-R6-C09-r1-1791055112169.
Ornith 1.5 35B retried and ended in the right state in 27 of 30 runs, but in 14 runs it returned no status line at all, which the scoring counts as no task success.
What helps
Put the retry policy in the harness: honour Retry-After, back off with jitter, cap the attempts, and keep the same idempotency key. Report non-completion explicitly when the budget runs out.
async function withRetry(fn, { attempts = 4 } = {}) {
for (let i = 0; ; i++) {
try { return await fn(); }
catch (e) {
if (!isTransient(e) || i + 1 >= attempts) throw e;
await sleep(retryAfterMs(e) ?? 2 ** i * 500 + Math.random() * 250);
}
}
}Limits
Mock tools, synthetic cases, one German-language prompt, 10 repetitions per case and model. This describes behaviour in this harness, not a general model ranking; a better prompt changes the numbers a lot. Every run is public: raw transcripts, harness and scoring (CC BY 4.0). Part of the 500-run write-up.
Questions
Should an AI agent retry after an HTTP 429?
Yes, with limits: honour the Retry-After header, back off with jitter, cap the number of attempts, and keep the same idempotency key on writes. A 429 means the request was rejected, so a retry cannot duplicate the write.
What happens if the agent never retries?
In this harness Claude Haiku 4.5 and Qwen3 14B stopped after one call in 30 of 30 runs and never created the invoice or order. Nothing was duplicated, but the work was not done.