Copyright © 2025 All Rights Reserved
3 Min
Blueprint Prep analyzed the LSAT results of OpenAI’s ChatGPT, a language model skilled at responding to human language. They found that the ChatGPT-3.5 series failed to achieve the scores needed for admission to a top 14 law school.
A veteran Blueprint Prep LSAT instructor analyzed the test results. They found ChatGPT’s main errors were due to inconsistent application of logical and critical reasoning. Additionally, ChatGPT struggled to differentiate crucial information from unnecessary material. These are areas where input from an experienced instructor would help ChatGPT better understand the questions and how to interpret them. However, despite its unimpressive scores, ChatGPT remained convinced it was answering the test questions correctly during the test.
“We know that the LSAT is designed to measure a student’s analytical reasoning, critical thinking, and reading comprehension skills, which are essential for success in law school,” said Gene Suhir, LSAT Academic Manager at Blueprint Prep.
“These skills can be significantly strengthened, but to do so requires leveraging proven strategies and top-scoring instructors who can help students get into the mindsets of the test makers. This form of LSAT test prep not only enables the student to process information like a lawyer would but it’s also been proven that strengthening reasoning skills via LSAT test prep can help wire students’ brains to think more like lawyers. This is not the specific kind of reasoning that ChatGPT is innately useful for, although it can learn these skills.”
The LSAT grades on a 120-180 scale, with 151 as the approximate average. Students aiming for the top 14 law schools generally need scores in the 170s. ChatGPT attempted the LSAT twice. It scored 148 (37th percentile) and 157 (70th percentile). Despite improving in the second test, the chatbot still got about one-third of the questions wrong.
In terms of performance, ChatGPT showed its weakest abilities in logic games. It also displayed average results in logic reasoning and scored best in reading comprehension, which is the LSAT’s most straightforward test section. The chatbot struggled to identify when a new rule applied only to a specific question and not to subsequent ones. Additionally, the LSAT added irrelevant information to distract test takers. ChatGPT failed to distinguish between peripheral and essential information.
“ChatGPT had not prepared for taking the LSAT and clearly showed skill gaps in the reasoning abilities that are relevant to law school,” said Matt Riley, CEO and Co-Founder of Blueprint Prep. “Even with its phenomenal ability to scrape the internet for existing knowledge, ChatGPT’s untutored scores would probably not get the bot accepted to a top law school, demonstrating the value of selecting a proven prep course to achieve high LSAT scores.”
When asked whether it wished to take the LSAT again, ChatGPT replied,
“As an AI language model, ChatGPT does not have a personal ambition or desire to pursue a legal career that requires taking the LSAT test.” The bot did recognize, however, that taking an LSAT prep course “helps students improve their chances of success, gain expert guidance and support, and feel more confident and prepared.”