A Stanford study shows AI-generated legal responses are preferred over those by law professors, highlighting AI's effectiveness in legal reasoning.
AI Performance in Legal Education Examined
A recent study from Stanford University reveals that artificial intelligence (AI) is often preferred over human law professors for legal reasoning tasks. The research indicates that AI-generated responses are not only favored but also exhibit a lower rate of harmfulness compared to those from human instructors.
The study involved 16 law professors from 14 U.S. law schools, including renowned institutions like Stanford and Yale. They created 40 questions related to contract law, allowing researchers to evaluate the effectiveness of today's AI models in a legal context.
In a comprehensive analysis of 2,918 blinded evaluations, professors preferred answers generated by Google’s Gemini 2.5 Pro 75.92% of the time and NotebookLM 74.75% of the time. This demonstrates a strong inclination towards AI outputs, with professors favoring them in about three-quarters of evaluations.
Furthermore, the study examined the consistency of assessments among the professors and found that their agreement surpassed what would be expected if judgments were random, suggesting that AI responses align well with established professional standards in legal education.
The AI models also excelled in various categories, particularly in recall questions regarding case law and policy discussions. Notably, harmfulness rates for AI answers were significantly lower, with Gemini at 3.41% and NotebookLM at 3.64%, in stark contrast to the 12.06% for human-generated content.
However, the researchers caution that their findings do not evaluate how well AI responses align with individual teaching styles, indicating that while AI is generally preferred, it may not meet every professor's specific preferences. They stated, "While LLM responses are generally preferred over those of human instructors, our evaluation setting does not allow us to directly measure the extent to which instructor preferences are satisfied."
This research arrives as the legal field increasingly adopts AI technologies. Courts and law schools are beginning to integrate AI tools into their workflows and curricula, recognizing their potential to enhance legal practice. John P. Anderson, Dean of Mississippi College School of Law, noted the significant role of AI tools in shaping the future of legal education and practice, emphasizing that students will need to be familiar with these technologies.
Despite the promising results, challenges remain in the accuracy of AI, with instances of erroneous information still posing risks for legal firms, as highlighted by a recent incident involving fictitious citations in a filing by Sullivan & Cromwell.
