Most decision models beat LLMs on intent routing
On the BANKING77 benchmark, given only the intent names, four of five decision models beat all three LLMs I tested at routing customer messages. Three of the four also cost less.
Why I ran this test
Routing customer messages to the right topic is a rote decision. Since autumn 2026, vendors have been selling "decision models": small models that don't generate text, just pick from options you give them and return a confidence score. They claim these are cheaper, faster and better calibrated than prompting an LLM.
I wanted to check. My earlier test used synthetic emails. This time I used a public benchmark others have published on, so the results can be compared with published numbers.
The benchmark: BANKING77
BANKING77 (PolyAI, Casanueva et al. 2020, CC BY 4.0) contains short customer messages to a bank, each labelled with one of 77 intents: card_arrival, exchange_rate, declined_transfer, and so on. The official test split has 3,080 messages. I took 13 per intent (1,001 messages) with a fixed random seed.
Cloudflare publishes its own BANKING77 numbers for its decision models. Academics have published LLM results. This makes comparison possible.
Next to published results
My runs with their 95% intervals, and published numbers on the full test set. Hover a point for the source.
Setup
Every model saw the same thing: the message and the list of 77 intent names. No examples, no descriptions, no tuning. Zero-shot.
Decision models (via OpenRouter's decisions endpoint):
- Cloudflare Clef Flash
- Cloudflare Clef
- Liquid D1.2 Ultra
- Liquid D1.2
- Typesafe Jev 1.14
LLMs (short JSON prompt):
- DeepSeek V4.1 Flash (OpenRouter API)
- GPT-6 Luna (Codex CLI)
- Claude Haiku 5.5 (Claude Code CLI)
For Luna and Haiku, cost is estimated from API list prices; latency was not measured. Run date: 17 November 2026.
Making the numbers trustworthy
- 95% confidence intervals on every accuracy (Wilson score). At 1,001 messages, this is roughly ±1.5 to ±3 points.
- Paired exact McNemar tests against the best model, Holm-corrected. "Better" means the difference is unlikely to be noise.
- DeepSeek run twice with intent list in different order: 82.1% and 81.5%, same answer on 93% of messages. Prompt order moves results by under a point.
- Cross-check against published numbers and vendor reports.
Results
Accuracy vs. cost
Cost per 1,000 messages, log scale. Higher and further left is better. Quadrants split at the median of all eight models.
Key findings:
- Cloudflare Clef Flash: 95.5% accuracy (94.0–96.6), $0.047 per 1,000 messages, 0.54 s median latency. Best overall.
- Cloudflare Clef: 94.4% (92.8–95.7), $0.40 per 1,000, 0.69 s. Not significantly different from Clef Flash (p = 0.07), but about 8× the cost.
- Liquid D1.2 Ultra: 90.9% (89.0–92.5), $0.020, 0.37 s.
- Liquid D1.2: 88.1% (86.0–90.0), $0.009, 0.36 s. Cheapest tested.
- DeepSeek V4.1 Flash: 82.1% (79.6–84.4), $0.093, 1.18 s. Best LLM, but significantly below Clef Flash.
- GPT-6 Luna: 80.2% (77.6–82.6), ~$0.067 estimated.
- Typesafe Jev 1.14: 79.6% (77.0–82.0), $0.071, 0.80 s.
- Claude Haiku 5.5: 63.7% (60.7–66.7), ~$0.067 estimated. Also produced 10 invalid answers (intent not in list).
Every model except Clef is significantly less accurate than Clef Flash.
Against published numbers
- Fine-tuned BERT on 10,003 training examples: 93.7% (Casanueva et al. 2020). Clef Flash, with zero training examples: 95.5% on my sample.
- GPT-4 with one example per intent: 80.4%. Claude 2 same setup: 76.8% (Loukas et al. 2023). Today's LLMs without examples land in the same 80–82% range.
- Vendor check: Cloudflare reports macro-F1 of 94.2 for Clef, 90.9 for Clef Flash, 79.7 for Jev. I measured 94.4, 95.5, 78.7. Clef and Jev match closely; Clef Flash scored higher than Cloudflare reports.
Where models fail
Many errors are between near-identical intents:
- card_arrival vs card_delivery_estimate
- order_physical_card vs get_physical_card
One failure mode is instructive. Messages labelled get_physical_card are mostly PIN questions ("do i need a pin", "If I need a PIN for my card, where is it located?"). Luna, Haiku and Jev each sent 12 of 13 to change_pin. Loukas et al. saw GPT-4 do the same. Real label sets have these overlaps. The labels matter as much as the model.
Speed and cost
Response time
Median latency, end to end, for the models called through an API.
Liquid D1.2 models answer in ~0.37 s. Clef Flash in 0.54 s. DeepSeek in 1.18 s. All are cheap per message. At 1 million messages monthly:
- Clef Flash: ~$47
- Clef: ~$397
The entire experiment cost $0.73 in API fees.
Confidence thresholds
Decision models return confidence scores. For Clef Flash:
Accuracy vs. automation
{BEST_DECISION_NAME}, routing automatically only the messages above a confidence threshold. Hover a point for the threshold.
- At 80% confidence: routes 84.4% automatically at 98.8% accuracy; 15.6% to a person.
- At 90% confidence: routes 64.2% at 99.7% accuracy.
This is the practical trade-off: coverage versus precision.
Limits of this test
- Banking messages only, English only, single domain.
- Sample of 1,001 from the test split, not the full 3,080.
- One run per model (except DeepSeek's order check).
- Zero-shot with intent names only. LLMs might improve with examples or descriptions.
- Luna and Haiku ran through CLIs with default settings.
- BANKING77 messages are short single questions. Real inboxes contain threads, signatures, multiple languages.
What I would do for a client
Take 100–300 real messages. Agree on the label set and label them together. Run the same comparison. Pick the cheapest model that is accurate enough. Set a confidence threshold. Send everything below it to a person.
That is my 7-day pilot.