What the agent gets right, and where it fails.
Live figures come from the database behind the chat. Eval figures come from the last run over 300 labelled cases. Nothing on this page is typed in.
Eval passNo run yet300 labelled cases, both languages
Hand-offs caughtNo run yetvaluation, contract, complaint, out of scope
Latency, p50No chats yetlive · last 30 days
Cost a conversationNo chats yetlive · last 30 days · Claude list price
Eval pass by category
last run
Hand-offs by reason
live · last 30 days
Conversations a day
live · last 7 days
Failure log
Every case the agent got wrong in the last run, with the cause and the fix.
| Case | Lang | Category | Expected | Got | Cause | Fix |
|---|---|---|---|---|---|---|
| No eval run yet. | ||||||
How the labels were made
- A model judge labelled all 300 cases.
- Josh checked 60 by hand; the agreement figure shows how often the judge matched him.
- A case passes when the agent calls the expected tool (or none), names the right area, hands off for the right reason, and never confirms a viewing or quotes a figure where it must not.
What the agent cannot do
- Send a message, email or WhatsApp. Move or promise money.
- Quote a figure for an area with fewer than 10 registered deals.
- Value a visitor's own home. That goes to Josh.
- Touch a live database. It reads one exported JSON file.
Runs
| Run | Pass | Hand-offs | p50 |
|---|---|---|---|
| No eval run yet. | |||