AI can be found everywhere, from drafting emails to coding apps to diagnosing diseases. But when it comes to providing legal advice, how close are we to AI being a reliable legal expert?
Linklaters recently put AI through a series of law exams, and the tests revealed that AI can answer legal questions with surprising competence. However, the subtleties of legal reasoning, contextual interpretation, and factual application remain problematic for AI.
With 50 tough questions across 10 practice areas, including contract law, corporate governance, tax, intellectual property, and banking. The test was aimed at the standard of a mid-level lawyer level with two years of post-qualification experience.
Each AI model was scored out of 10 based on three areas: substance (accuracy), citations (reliability of legal references), and clarity (how understandable the answers were).
The best performer, OpenAI o1, scored 6.4, followed by Google's Gemini 2.0 at 6.0.
Although these scores represent an improvement from earlier models like GPT-3 and GPT-4, the gap between AI and human expertise remains substantial.
The hallucination problem: AI's Achilles' heel
AI models can infamously generate information that sounds plausible but isn't real. In legal settings, this can be disastrous. The Linklaters study found that earlier AI models fabricated case law and statutes in nearly one-third of responses.
Although newer models like OpenAI o1 and Gemini 2.0 have reduced this rate to 9%, that's still too high when the stakes involve real-world legal outcomes. A lawyer unknowingly referencing a fabricated case would be the stuff of nightmares for any law firm. Credibility would be lost, and client trust would be shattered.
AI's tendency to lie confidently is an obvious concern for law firms. AI's overconfidence can lead users to false security, believing that an AI-generated answer is accurate when it isn't. This highlights why expert human review remains indispensable.
Where AI shines and where it doesn't
Interestingly, the models excelled at summarizing established legal principles. AI performed reasonably well for tasks like explaining the difference between "best endeavors" and "reasonable endeavors" or summarizing privacy regulations. These answers can be sourced from widely available legal content – a territory where AI feels comfortable.
AI also stumbled when applying legal principles to specific facts or interpreting complex contractual clauses. The AI models provided confident but legally incorrect interpretations when asked about the legal implications of a flawed bonus clause. The nuance required to understand contractual intentions eludes the AI, especially in ambiguous cases.
Legal advice requires understanding the broader context of a client's needs. Nuance is everything. Only humans possess the perfect blend of logic, creativity, and judgment to take the next actionable steps in law.
Interpreting a clause influenced by unique business relationships or anticipating how a court might view ambiguous contract language requires human understanding. The problem is even the most advanced AI models still lack the human touch that can impact legal decisions.
"It's far easier to refine or improve something than to start from scratch, and when more people (or tools) contribute, creativity and outcomes improve. But AI alone isn't enough. You need a well-designed product that seamlessly integrates AI for legal work.”
Kira Söderlind
Why testing reflects reality
Some might argue that AI would perform better with more tailored prompts. Linklaters anticipated this and chose a realistic testing approach. Instead of optimizing prompts for maximum AI performance, they kept the process simple and standard.
Most notably, AI models performed best in real estate law and worst in corporate law. Real estate questions primarily required summarization, a known strength of AI. On the other hand, corporate law demanded more analytical reasoning, such as applying laws to complex fact patterns.
The disparity shows that while AI may support some legal functions, it isn't ready for the strategic and interpretive work that defines high-level legal practice.
A partner, not a replacement
Could AI eventually master the complexities of legal reasoning? The jury's still out. Linklaters' benchmark exam indicates that while AI might continue improving, it remains a "stochastic parrot" that repeats learned information without proper understanding. It can't reason contextually, adapt to nuanced scenarios, or offer the strategic foresight clients expect from top-tier legal advice.
Even if AI can answer complex legal questions accurately, the broader challenge of understanding the question behind the question remains. The problem is that clients rarely approach lawyers with neatly packaged problems. They bring business goals, concerns, and risks. Translating these into legal solutions requires wisdom as well as knowledge.
According to Kira Söderlind (CEO and co-founder of Pocketlaw), "AI can deliver legal expertise, but the real value isn't just in replacing lawyers – it's about augmenting their capabilities, enhancing both speed and quality. AI is an incredible sparring partner if the goal is the best possible result in the fastest way.”
"It's far easier to refine or improve something than to start from scratch, and when more people (or tools) contribute, creativity and outcomes improve. But AI alone isn't enough. You need a well-designed product that seamlessly integrates AI for legal work,” she said.
Söderlind also offered a timely reminder that the real value comes from optimizing both the AI infrastructure and the user experience for legal operations, ensuring the technology enhances expertise rather than just automating for the sake of it.
“The best legal AI tools don't just use LLMs – they make them work for lawyers. Remember that AI isn't the tool but the electricity that powers your house. The real value lies in finding the right house for your legal work.”
The promise of AI in law is real, but its real potential can only be unlocked when paired with the insight and judgment only humans can provide.
James Grice, Head of Innovation and AI at Lawfront Group, is also optimistic. "AI is transforming legal work by accelerating processes and handling routine tasks while leaving the nuanced, complex reasoning to human experts."
"AI should be seen not as a replacement for lawyers, but as a powerful tool that augments their capabilities. By partnering with legal professionals, we can shift the narrative away from the threat of replacement and highlight AI's collaborative value instead.”
Grice also shared with me how this approach ensures that while AI enhances efficiency, critical human judgment, ethical considerations, and complex problem-solving skills remain at the core of legal practice.
"It is less a replacement for an experienced lawyer; it is more like a junior lawyer that can perform first-stage drafts or analysis, leaving them to apply their experience and remove the time drain of more administrative tasks."
Lawfront is just one of many examples of a law firm exploring how AI and Legal Tech can play the "Digital Associate" role in a way that all legal teams will see the benefit.
AI is increasingly seen as a powerful legal partner, pairing its speed and efficiency with human lawyers' judgment, creativity, and empathy. Hopefully, this perfect collaboration will finally retire the dated stereotype that the legal industry is slow to adapt to technological change.
Your email address will not be published. Required fields are markedmarked