A study conducted by technology company Saturn has revealed that leading chatbots, including ChatGPT, Claude, Copilot, Grok, and Gemini, delivered incorrect responses to 57% of financial inquiries on average. The research assessed 18 different models through 121 questions, yielding over 10,000 responses.
The findings indicate a significant rise in errors for complex financial tasks, with an alarming 88% error rate noted. Some models, in particular, even reached error rates as high as 99%. Claude Opus 5 performed the best in reasoning mode, yet it still made mistakes 39% of the time. A notable example highlighted was advice from Claude Haiku 4.5 that could have led to a tax penalty of £17,500 (around $23,400).
