AI chatbot can pass national lawyer ethics exam, study finds – Reuters
AI (Artificial Intelligence) letters are placed on computer motherboard in this illustration taken June 23, 2023. REUTERS/Dado Ruvic Acquire Licensing Rights
Nov 16 (Reuters) – Popular AI chatbot GPT-4 outperforms most aspiring lawyers on the legal ethics exam required by nearly every state in order to practice law, a new study has found.
GPT-4 answered 74% of the questions correctly on a simulated Multistate Professional Responsibility Exam (MPRE), compared with an estimated 68% average among human test takers nationwide, according to a report released on Thursday by LegalOn Technologies — which sells AI software that reviews contracts.
“Our study indicates that in the future it may be possible to develop AI to assist lawyers with ethical compliance and operate, where relevant, in alignment with lawyers’ professional responsibilities,” the study reads.
It joins a growing body of research examining AI within legal education and attorney licensure. An earlier study found that the previous version of GPT-4 earned passing but not stellar scores on law school final exams. Another more recent study found that GPT-4 can pass the bar exam. Earlier this month, researchers found that access to GPT-4 improved speed on legal writing assignments but didn’t bolster the quality law students’ work.
GPT-4 is a large language model from Microsoft-backed OpenAI that generates human-like text based on user queries.
A spokesperson for the National Conference of Bar Examiners, which develops the MPRE, said that it could not assess the LegalOn report's claims that GPT-4 can pass its ethics test.
"The legal profession is always evolving in its use of technology, and will continue to do so," said National Conference spokesperson Sophie Martin. She added that "attorneys have a unique set of skills that AI cannot currently match."
Every state besides Wisconsin requires law students to pass the 60-multiple-choice MPRE before they are admitted to practice, in addition to passing the bar exam. Tested subjects include conflicts of interest, lawyer-client relationships and confidentiality. Most people take the MPRE while in law school.
GPT-4 performed particularly well on questions about conflicts of interest, with a 91% rate of correct responses. It also answered 88% of questions about the client-lawyer relationship correctly. But its accuracy fell with questions on communications about legal services and safekeeping funds and other property, which it answered correctly a respective 71% and 72% of the time.
“This research demonstrates for the first time that top-performing generative AI models can apply black-letter ethical rules as effectively as aspiring lawyers,” the study reads.
Read more:
AI improves legal writing speed, not quality – study
Bar exam score shows AI can keep up with 'human lawyers,' researchers say
Get the latest legal news of the day delivered straight to your inbox with The Afternoon Docket.
Reporting by Karen Sloan
Our Standards: The Thomson Reuters Trust Principles.
Thomson Reuters
Karen Sloan reports on law firms, law schools, and the business of law. Reach her at karen.sloan@thomsonreuters.com
Henry Engler
Diana Novak Jones
Andrew Goudsward
Sara Merken
Reuters, the news and media division of Thomson Reuters, is the world’s largest multimedia news provider, reaching billions of people worldwide every day. Reuters provides business, financial, national and international news to professionals via desktop terminals, the world's media organizations, industry events and directly to consumers.
Build the strongest argument relying on authoritative content, attorney-editor expertise, and industry defining technology.
The most comprehensive solution to manage all your complex and ever-expanding tax and compliance needs.
The industry leader for online information for tax, accounting and finance professionals.
Access unmatched financial data, news and content in a highly-customised workflow experience on desktop, web and mobile.
Browse an unrivalled portfolio of real-time and historical market data and insights from worldwide sources and experts.
Screen for heightened risk individual and entities globally to help uncover hidden risks in business relationships and human networks.
All quotes delayed a minimum of 15 minutes. See here for a complete list of exchanges and delays.
© 2023 Reuters. All rights reserved
