AGENTCAPABLE · LIVE AGENT TEST REPORT
Issued by Automation Craft
- Document no.
- LAT-0000 SPECIMEN
- Site tested
- acme-example.dev
- Test date
- (specimen)
- Delivered
- within 24 hours of purchase
- Panel
- 8 configurations, 3 tasks, 24 recorded attempts
Summary
RESULT: 4 of 8 agent configurations completed all three tasks. The most common failure: the pricing page requires JavaScript menu interaction that 3 configurations could not complete.
- Attempts completed
- 17/24
- Median time per attempt
- 64s
- Median tokens per attempt
- 28,400
The grid
| # | Model | Harness | Tool | Task 1 Read | Task 2 Pricing | Task 3 Docs | Time | Tokens |
|---|---|---|---|---|---|---|---|---|
| 1 | Claude Sonnet 5 | Claude Code | browser-use | PASS | PASS | PASS | 41s | 18,200 |
| 2 | GPT-5 | Codex | Playwright MCP | PASS | FAIL (step 3) | PASS | 78s | 41,220 |
| 3 | Gemini 2.5 Pro | Gemini CLI | computer-use | PASS | PASS | PARTIAL | 91s | 44,050 |
| 4 | DeepSeek V3 | OpenClaw | browser-use | PASS | FAIL (step 2) | PASS | 66s | 30,900 |
| 5 | Qwen3 | OpenClaw | Playwright MCP | PASS | PASS | PASS | 58s | 26,300 |
| 6 | Kimi K2 | Codex | browser-use | PASS | FAIL (step 3) | FAIL (step 1) | 84s | 39,750 |
| 7 | Claude Haiku 4.5 | Claude Code | Playwright MCP | PASS | PASS | PASS | 37s | 12,900 |
| 8 | GPT-5 mini | Gemini CLI | computer-use | PASS | PARTIAL | PASS | 63s | 24,100 |
Attempt detail
ATTEMPT DETAIL: CONFIGURATION 2, TASK 2 (FIND PRICING)
- Loaded acme-example.dev, 200 OK.
- Identified navigation, chose "Products".
- Pricing link renders only inside a JavaScript hover menu; the agent could not open it.
- Fell back to sitemap.xml: no /pricing entry.
- Gave up after 78 seconds.
Failure cause: pricing is unreachable without JavaScript hover interaction.
Prioritized fixes
- F1Add a plain link to /pricing in the footer and sitemap (removes the blocker hit by configurations 2, 4 and 6).
- F2Serve the pricing table as static HTML, not a JS-rendered widget (removes the barrier behind the Task 2 partials).
- F3Add llms.txt pointing agents to pricing and docs (points agents straight to pricing and docs).
Recordings of all recorded attempts, including retries, accompany the customer report as proof. Not included in this specimen.
SPECIMEN · NOT A MEASUREMENT
Every customer report names exactly which models and harnesses ran. We never list a configuration that did not run.