3 Min
After analyzing the LSAT results of OpenAI's ChatGPT, a language model capable of accurately responding to human language, Blueprint Prep - a leading test preparation and professional training company, discovered that ChatGPT-3.5 series failed to meet the typical scores necessary for admission to a top 14 law school.
Upon analyzing the test results, a veteran Blueprint Prep LSAT instructor found that ChatGPT's main errors were due to a lack of consistent application of logical and critical reasoning, as well as an incapacity to differentiate crucial information from unnecessary material. These are areas where input from an experienced instructor would help ChatGPT better understand the questions and how to interpret them. However, even though ChatGPT's scores were not impressive, it remained convinced that it was answering the test questions correctly while taking the test.
"We know that the LSAT is designed to measure a student's analytical reasoning, critical thinking, and reading comprehension skills, which are essential for success in law school," said Gene Suhir, LSAT Academic Manager at Blueprint Prep. "These skills can be significantly strengthened, but to do so requires leveraging proven strategies and top-scoring instructors who can help students get into the mindsets of the test makers. This form of LSAT test prep not only enables the student to process information like a lawyer would, but it's been proven that strengthening reasoning skills via LSAT test prep can help wire students' brains to think more like a lawyer. This is not the specific kind of reasoning that ChatGPT is innately useful for, although it can learn these skills."
The LSAT, which is graded on a 120-180 scale, considers a score of 151 as an approximate average. Students aspiring to get admitted into one of the top 14 law schools in the nation generally need a score in the 170s range. ChatGPT attempted the LSAT twice and achieved scores of 148 (37th percentile) and 157 (70th percentile), respectively. Despite doing better in the second test, the chatbot still got around one-third of the questions wrong.
In terms of performance, ChatGPT showed its weakest abilities in logic games, displayed average results in logic reasoning, and scored best in reading comprehension, which is the LSAT's most straightforward test section. The chatbot encountered difficulties in identifying when a newly introduced rule was only applicable to a specific question and not transferable to the following questions. Furthermore, the LSAT intentionally added irrelevant information to questions to divert the test taker's attention, and ChatGPT was unable to distinguish between the peripheral and essential information.
"ChatGPT had not prepared for taking the LSAT and clearly showed skill gaps in the reasoning abilities that are relevant to law school," said Matt Riley, CEO and Co-Founder of Blueprint Prep. "Even with its phenomenal ability to scrape the internet for existing knowledge, ChatGPT's untutored scores would probably not get the bot accepted to a top law school, demonstrating the value of selecting a proven prep course to achieve high LSAT scores."
When asked whether it wished to take the LSAT again, ChatGPT replied, "As an AI language model, ChatGPT does not have a personal ambition or desire to pursue a legal career that requires taking the LSAT test." The bot did recognize, however, that taking an LSAT prep course "helps students improve their chances of success, gain expert guidance and support, and feel more confident and prepared."
