Research · Case study · 16 October 2026
Malformed replies and wrong IDs: all five models handled 100 of 100 runs
Not every failure is hard. We injected two more faults: a ticket that exists but whose reply is malformed JSON, and an order whose reply names a different ID. All five models got all 100 runs right.
Results, 20 runs per model
| Model | Runs | Correct end state | Runs with duplicates | False success | Final status reported |
|---|---|---|---|---|---|
| DeepSeek Flash | 20 | 20 | 0 | 0 | success 20 |
| Qwen3.8 27B (free) | 20 | 20 | 0 | 0 | success 20 |
| Qwen3 14B (local) | 20 | 20 | 0 | 0 | success 20 |
| Claude Haiku 4.5 | 20 | 20 | 0 | 0 | success 20 |
| Ornith 1.5 35B | 20 | 20 | 0 | 0 | success 20 |
Cases C04 (ticket, malformed reply) and C10 (order, other ID). Expected end state: one ticket, one order.
Why this matters
These two cases did not separate the models, so the differences elsewhere in the harness are not a general weakness of tool use. They sit in three specific situations: a lost reply after a write, a success response that is not true, and chained steps. If you test your own agent, start there.
Limits
Mock tools, synthetic cases, one German-language prompt, 10 repetitions per case and model. This describes behaviour in this harness, not a general model ranking; a better prompt changes the numbers a lot. Every run is public: raw transcripts, harness and scoring (CC BY 4.0). Part of the 500-run write-up.
Questions
Which tool failures do current models handle reliably?
In this harness, a malformed JSON reply after a successful write and a reply that names a different ID were handled correctly by all five models in all 100 runs.
Where do agents still fail with tools?
In this harness: after a lost reply to a write, when a tool reports success that is not true, and in chained steps where the next call needs the result of the one that timed out.