<story><title>Re-Evaluating GPT-4&apos;s Bar Exam Performance</title><url>https://link.springer.com/article/10.1007/s10506-024-09396-9</url></story><parent_chain></parent_chain><comment><author>fnordpiglet</author><text>Scoring 96 percentile among humans taking the exam without moving goal posts would have been science fiction two years ago. Now it’s suddenly not good enough and the fact a computer program can score decent among passing lawyers and first time test takers is something to sneer at.&lt;p&gt;The fact I can talk to the computer and it responds to me  idiomatically and understands my semantic intent well enough to be nearly indistinguishable from a human being is breath taking. Anyone who views it as anything less in 2024 and asserts with a straight face they wouldn’t have said the same thing in 2020 is lying.&lt;p&gt;I do however find the paper really useful in contextualizing the scoring with a much finer grain. Personally I didn’t take the 96 percentile score to be anything other than “among the mass who take the test,” and have enough experience with professional licensing exams to know a huge percentage of test takers fail and are repeat test takers. Placing the goal posts quantitatively for the next levels of achievement is a useful exercise. But the profusion of jaded nerds makes me sad.</text></comment>
<story><title>Re-Evaluating GPT-4&apos;s Bar Exam Performance</title><url>https://link.springer.com/article/10.1007/s10506-024-09396-9</url></story><parent_chain></parent_chain><comment><author>radford-neal</author><text>A basic problem with evaluations like these is that the test is designed to discriminate between &lt;i&gt;humans&lt;/i&gt; who would make good lawyers and humans who would not make good lawyers.  The test is not necessarily any good at telling whether a non-human would make a good lawyer, since it will not test anything that pretty much all humans know, but non-humans may not.&lt;p&gt;For example, I doubt that it asks whether, for a person of average wealth and income, a $1000 fine is a more or less severe punishment than a month in jail.</text></comment>