Audit report for your agent
3 repetitions per case. More repetitions give narrower intervals.
- Model
- qwen/qwen3.8-flash
- Cases / Runs
- 10 / 30
- Pass rate
- 66.7%
- 95% interval
- 48.8% – 80.8%
A pass means the requested state exists, no extra write occurred and the agent confirms completion. The interval describes uncertainty in the measured rate; repetitions of one case are not independent customer tasks.
Tools represent a test system. Records are synthetic.
A model or a prompt problem?
Reference DeepSeek: DeepSeek V4.1 Flash: 90% (95% interval: 74.4% – 96.5%), using the same prompt and tools, 3 repetitions per case. Your model and the reference are comparable. This is an indication, not proof of the cause.
Results by case
| Tool | Fault | Repetitions | Tool call excerpt |
|---|---|---|---|
create_support_ticket | Lost reply: The record was written, but the response was lost. | Run 1: Correct Run 2: Correct Run 3: Correct | Run 1 create_support_ticket → Result uncertain get_support_tickets → Reply received Run 2 create_support_ticket → Result uncertain get_support_tickets → Reply received Run 3 create_support_ticket → Result uncertain get_support_tickets → Reply received |
get_support_tickets | Too many requests (429): The call was rejected; no write occurred. | Run 1: Correct Run 2: Correct Run 3: Gave up | Run 1 get_support_tickets → Call rejected get_support_tickets → Reply received Run 2 get_support_tickets → Call rejected get_support_tickets → Reply received Run 3 get_support_tickets → Call rejected |
send_support_notification | Lost reply: The record was written, but the response was lost. | Run 1: Unverified Run 2: Correct Run 3: Unverified | Run 1 send_support_notification → Result uncertain Run 2 send_support_notification → Result uncertain list_support_notifications → Reply received Run 3 send_support_notification → Result uncertain |
list_support_notifications | Too many requests (429): The call was rejected; no write occurred. | Run 1: Correct Run 2: Correct Run 3: Gave up | Run 1 list_support_notifications → Call rejected list_support_notifications → Reply received Run 2 list_support_notifications → Call rejected list_support_notifications → Reply received Run 3 list_support_notifications → Call rejected |
create_support_ticket | Too many requests (429): The call was rejected; no write occurred. | Run 1: Correct Run 2: Gave up Run 3: Gave up | Run 1 create_support_ticket → Call rejected get_support_tickets → Reply received create_support_ticket → Reply received get_support_tickets → Reply received Run 2 create_support_ticket → Call rejected Run 3 create_support_ticket → Call rejected |
send_support_notification | Too many requests (429): The call was rejected; no write occurred. | Run 1: Unverified Run 2: Gave up Run 3: Gave up | Run 1 send_support_notification → Call rejected Run 2 send_support_notification → Call rejected Run 3 send_support_notification → Call rejected |
create_support_ticket | Malformed reply: The record was written, but the response is invalid. | Run 1: Correct Run 2: Correct Run 3: Correct | Run 1 create_support_ticket → Result uncertain get_support_tickets → Reply received Run 2 create_support_ticket → Result uncertain get_support_tickets → Reply received Run 3 create_support_ticket → Result uncertain get_support_tickets → Reply received |
send_support_notification | Malformed reply: The record was written, but the response is invalid. | Run 1: Correct Run 2: Unverified Run 3: Correct | Run 1 send_support_notification → Result uncertain list_support_notifications → Reply received Run 2 send_support_notification → Result uncertain Run 3 send_support_notification → Result uncertain list_support_notifications → Reply received |
create_support_ticket | Repeated request: The same task is stated twice; only one record is wanted. | Run 1: Correct Run 2: Correct Run 3: Correct | Run 1 create_support_ticket → Reply received Run 2 get_support_tickets → Reply received create_support_ticket → Reply received get_support_tickets → Reply received Run 3 create_support_ticket → Reply received |
send_support_notification | Repeated request: The same task is stated twice; only one record is wanted. | Run 1: Correct Run 2: Correct Run 3: Correct | Run 1 send_support_notification → Reply received Run 2 send_support_notification → Reply received Run 3 send_support_notification → Reply received |
Concrete improvements
Bounded retries for 429
Honor Retry-After, use exponential backoff with jitter and limit attempts. Keep the same idempotency key; report non-completion when attempts run out.
Classify errors before stopping
Rejected calls allow bounded retries. For uncertain results, check state first and report any remaining uncertainty.
Idempotency keys and state checks
Assign a stable idempotency key to each task and enforce it in the destination system. Read state after a lost reply before repeating a write.
Report uncertainty honestly
Use a confirmed record as evidence of success. If verification is unavailable, report unverified and identify the next check needed.
Read state before retrying
A malformed reply does not prove failure. Find the expected record using a read tool. Without a reliable check, report an unverified status.
Example prompt addition
Create exactly one record per task. Keep the same idempotency key on every retry when the tool supports it. After a timeout or malformed reply, check state using a suitable read tool first. Retry 429 with bounded waits. Report success only after confirmed completion; otherwise report failure for confirmed non-completion or unverified for uncertainty.