<story><title>Comparing humans, GPT-4, and GPT-4V on abstraction and reasoning tasks</title><url>https://arxiv.org/abs/2311.09247</url></story><parent_chain><item><author>krona</author><text>This paper evaluates performance compared to a &amp;#x27;human&amp;#x27; which presumably is an average adult human without cognitive impairment. I had to dig in to the references:&lt;p&gt;&lt;i&gt;In the first batch of participants collected via Amazon Mechanical Turk, each received 11 problems
(this batch also only had two “minimal Problems,” as opposed to three such problems for everyone
else). However, preliminary data examination showed that some participants did not fully follow
the study instructions and had to be excluded (see Section 5.2). In response, we made the screening
criteria more strict (requiring a Master Worker qualification, 99% of HITs approved with at least
2000 HIT history, as opposed to 95% approval requirement in the first batch). Participants in all
but the first batch were paid $10 upon completing the experiment. Participants in the first batch
were paid $5. In all batches, the median pay-per-hour exceeded the U.S. minimal wage.&lt;/i&gt;&lt;p&gt;(Arseny Moskvichev et al)&lt;p&gt;So in conclusion, this isn&amp;#x27;t a random sample of (adult) humans, and the paper doesn&amp;#x27;t give standard deviations.&lt;p&gt;It would&amp;#x27;ve been more interesting of they had sampled an age range of humans which we would place GPT-4 on rather than just &amp;#x27;it&amp;#x27;s not as good&amp;#x27; which is all this paper can say, really.</text></item></parent_chain><comment><author>svnt</author><text>What is your concern exactly?&lt;p&gt;This was a first-pass study in a field addressing some of the criticisms leveraged against an earlier study where the spatial reasoning problems were viewed to be too hard. They seemingly made the spatial reasoning questions as easy as they could.&lt;p&gt;The qualifications they put on MTurk are pretty standard if you want humans who care about what they are doing responding to your study. It costs more to do this.&lt;p&gt;It is a limitation of science that is both budgetary and procedural.&lt;p&gt;By calling into question their results you seem to be suggesting that an average human would only 33% of the time be able to tell e.g. how many points are inside a box, or whether more points are inside or outside of a box. This is extremely basic spatial reasoning we are talking about.&lt;p&gt;The problem they were addressing with the settings is just noise in the results by cheap bots and clicky humans trying to earn $0.50. It is endemic on MTurk.</text></comment>
<story><title>Comparing humans, GPT-4, and GPT-4V on abstraction and reasoning tasks</title><url>https://arxiv.org/abs/2311.09247</url></story><parent_chain><item><author>krona</author><text>This paper evaluates performance compared to a &amp;#x27;human&amp;#x27; which presumably is an average adult human without cognitive impairment. I had to dig in to the references:&lt;p&gt;&lt;i&gt;In the first batch of participants collected via Amazon Mechanical Turk, each received 11 problems
(this batch also only had two “minimal Problems,” as opposed to three such problems for everyone
else). However, preliminary data examination showed that some participants did not fully follow
the study instructions and had to be excluded (see Section 5.2). In response, we made the screening
criteria more strict (requiring a Master Worker qualification, 99% of HITs approved with at least
2000 HIT history, as opposed to 95% approval requirement in the first batch). Participants in all
but the first batch were paid $10 upon completing the experiment. Participants in the first batch
were paid $5. In all batches, the median pay-per-hour exceeded the U.S. minimal wage.&lt;/i&gt;&lt;p&gt;(Arseny Moskvichev et al)&lt;p&gt;So in conclusion, this isn&amp;#x27;t a random sample of (adult) humans, and the paper doesn&amp;#x27;t give standard deviations.&lt;p&gt;It would&amp;#x27;ve been more interesting of they had sampled an age range of humans which we would place GPT-4 on rather than just &amp;#x27;it&amp;#x27;s not as good&amp;#x27; which is all this paper can say, really.</text></item></parent_chain><comment><author>a1j9o94</author><text>Another thing I take issue with is this doesn&amp;#x27;t seem to be using known ways to improve performance of LLMs such as chain of thought and tree of thought prompting.</text></comment>