A study reveals that major AI models often disagree on factual accuracy, with a 67% conflict rate on real-world claims.
Study Highlights Discrepancies in AI Fact-Checking
A recent investigation into the reliability of five prominent AI models revealed that they disagreed on the accuracy of factual claims 67% of the time. Conducted by Kosta Jordanov at Lenz Research, the study assessed how these models—GPT-5.4, Claude Opus 4.7, Gemini 3 Pro, Gemini 3 Pro with Search, and Sonar Pro—handled a set of 1,000 real-world fact-check claims.
Out of the claims analyzed, there were 672 instances where at least one model offered a conflicting assessment compared to the majority. In 34% of these cases, one model deemed a statement true while another categorized it as false. The models were tasked with labeling each claim as true, mostly true, misleading, or false, yet achieved consensus on only 328 claims.
The study used Krippendorff’s alpha to gauge agreement among the models, yielding a score of 0.639, which indicates limited reliability. The researchers noted that a score below 0.8 is typically considered weak, highlighting the models' struggle to provide consistent factual assessments.
The findings are significant, especially as more people turn to AI for fact-checking. The study underscores that while AI companies often showcase improvements in accuracy through benchmark scores, these models frequently present conflicting evaluations on complex claims sourced from actual users, rather than standardized test sets.
For example, regarding the statement "The World Bank's active portfolio in Nigeria stands at over $16.4 billion as of 2025," the models provided varying classifications: GPT-5.4 rated it as "mostly true," while Gemini 3 Pro labeled it "false." Similar discrepancies were observed with other claims, emphasizing the challenges in relying on AI for accurate fact-checking.
The researchers concluded that while models can reach agreement on definitive truths or falsehoods, they often falter in the nuanced middle ground. With a lack of unanimous verdicts on claims deemed "mostly true," the study raises questions about the effectiveness of AI models as reliable fact-checkers.
