According to recently published study, the ChatGPT bot that sparked criticism succeeded on several law school exams.
According to a report from Reuters on Wednesday, lead study author Jonathan H. Choi of the University of Minnesota Law School said of the study, “Alone, ChatGPT would be a pretty mediocre law student.” Theoretically, according to Choi’s research, a human attorney may use the bot to prepare documents in order to “make their practice that much more effective.”
On a paper titled “ChatGPT Goes to Law School,” Choi worked with Kristin E. Hickman (University of Minnesota Twin Cities School of Law), Amy Monahan (University of Minnesota Law School), and Daniel Schwarcz (University of Minnesota Law School). Researchers wanted to know “how well” these models would perform on a law school exam without the aid of humans.
The model ultimately succeeded in passing all four classes (Constitutional Law: Federalism and Separation of Powers, Employee Benefits, Taxation, and Torts) with an overall average grade of C+. It is noted that this average, in a real-world setting, would put this “student” on academic probation. However, if this performance level was sustained over the course of law school, this so-called “student” could be able to complete the program.
Researchers found that ChatGPT performed better with essay questions than it did with multiple choice components in terms of specific capabilities. However, the model’s performance was “highly uneven” even within the essay-writing context.