A new study shows AI models favor Catholicism and exhibit bias against Jehovah's Witnesses, raising concerns over religious bias in AI.
Summary
- AI models show a strong preference for Catholicism while displaying bias against Jehovah’s Witnesses.
- The Grok model exhibited the highest bias; Anthropic and Meta's models showed the least.
- Findings come on the heels of Pope Leo XIV's warning about AI reflecting creator biases.
A recent study conducted by a consortium of universities has revealed that several prominent AI models demonstrate a significant bias towards Catholicism, particularly in contexts related to religious conversion, while discouraging interactions with other faiths.
The research was spearheaded by the Consortium for Evaluating Faith and Ethics in AI (CEFE-AI), which includes Baylor University, Brigham Young University, the University of Notre Dame, and Yeshiva University. Their preliminary findings, part of the AllFaith Benchmark, were shared on GitHub and presented at the Athens Summit on AI Ethics. The study emphasizes the often-overlooked aspect of religious bias in AI safety evaluations.
David Wingate, a professor at BYU, noted a troubling trend of religious omissions in AI interactions, stating that these systems encourage users to seek guidance from various figures in their lives but exclude spiritual leaders.
The analysis covered 3,640 responses from 20 AI models, including ChatGPT, Claude, and Grok, revealing a clear pattern in how these models engage with religious subjects.
Results indicated that nearly all models rated Catholicism favorably, achieving a 61% “encouraged” score, in stark contrast to Jehovah’s Witnesses, which garnered only 3%. Mainline Protestantism scored 49.2%, while Evangelical Protestantism was at 34%. Interestingly, agnosticism received the highest encouragement score at 71% among the tested belief systems.
Furthermore, numerous models displayed negative responses to atheism and agnosticism, while being more favorable towards Baha’i and Sikh beliefs. Grok 4.20 emerged as the most biased model, scoring 69% for Catholicism and 51% for Evangelical Protestantism, yet it was one of the few that rated Jehovah’s Witnesses above 5% positively.
This study's release coincided with Pope Leo XIV's encyclical, Magnifica Humanitas, which addresses the biases inherent in technology and the values of its creators. He emphasized that data should not be commodified or monopolized.
Despite growing concerns from religious leaders regarding AI's influence, the consortium highlighted that only a mere 0.2% of over 12,000 studies on AI bias have tackled issues related to religious bias. Professor Nancy Fulda from BYU commented on the unexpected nature of the results, which revealed significant biases towards specific belief systems rather than neutrality in guidance.
