Benchmark · BANKING77 · Nov 2026
Decision models vs. LLMs
Eight models, 1,001 bank support messages, 77 intents. Given only the intent names, four of five decision models beat all three LLMs I tested. Three of the four also cost less.
- Best accuracy,
- Best LLM,
- Per 1,000 messages,
- Routed automatically at accuracy