<story><title>Re-Evaluating GPT-4&apos;s Bar Exam Performance</title><url>https://link.springer.com/article/10.1007/s10506-024-09396-9</url></story><parent_chain></parent_chain><comment><author>fnordpiglet</author><text>Scoring 96 percentile among humans taking the exam without moving goal posts would have been science fiction two years ago. Now it’s suddenly not good enough and the fact a computer program can score decent among passing lawyers and first time test takers is something to sneer at.&lt;p&gt;The fact I can talk to the computer and it responds to me  idiomatically and understands my semantic intent well enough to be nearly indistinguishable from a human being is breath taking. Anyone who views it as anything less in 2024 and asserts with a straight face they wouldn’t have said the same thing in 2020 is lying.&lt;p&gt;I do however find the paper really useful in contextualizing the scoring with a much finer grain. Personally I didn’t take the 96 percentile score to be anything other than “among the mass who take the test,” and have enough experience with professional licensing exams to know a huge percentage of test takers fail and are repeat test takers. Placing the goal posts quantitatively for the next levels of achievement is a useful exercise. But the profusion of jaded nerds makes me sad.</text></comment>
<story><title>Re-Evaluating GPT-4&apos;s Bar Exam Performance</title><url>https://link.springer.com/article/10.1007/s10506-024-09396-9</url></story><parent_chain></parent_chain><comment><author>thehoneybadger</author><text>It is difficult to comment without sounding obnoxious, but having taken the bar exam, I found the exam simple. Surprisingly simple. I think it was the single most over hyped experience of my life. I was fed all this insecurity and walked into the convention center expecting to participate in the biggest intellectual challenge in my life. Instead, it was endless multiple choice questions and a couple contrived scenarios for essays.&lt;p&gt;It may also be surprising to some to understand that legal writing is prized for its degree of formalism. It aims to remove all connotation from a message so as to minimize misunderstanding, much like clean code.&lt;p&gt;It may also be surprising, but the goal when writing a legal brief or judicial opinion is not to try to sound smart. The goal is to be clear, objective, and thereby, persuasive. Using big words for the sake of using big words, using rare words, using weasel words like &amp;quot;kind of&amp;quot; or &amp;quot;most of the time&amp;quot; or &amp;quot;many people are saying&amp;quot;, writing poetically, being overly obtuse and abstract, these are things that get your law school application rejected, your brief ridiculed, and your bar exam failed.&lt;p&gt;The simpler your communication, the more formulaic, the better. The more your argument is structured, akin to a computer program, the better.&lt;p&gt;As compared to some other domain, such as fiction, good legal writing much easier for an attention model to simulate. The best exam answers are the ones that are the most formulaic and that use the smallest lexicon and that use words correctly.&lt;p&gt;I only want to add this comment because I want to inform how non-lawyers perceive the bar exam. Getting an attention model to pass the bar exam is a low bar. It is not some great technical feat. A programmer can practically write a semantic disambiguation algorithm for legal writing from scratch with moderate effort.&lt;p&gt;It will be a good accomplishment, but it will only be a stepping stone. I am still waiting for AI to tackle messages that have greater nuance and that are truly free form. LLMs are still not there yet.</text></comment>