<story><title>“Emergent” abilities in LLMs actually develop gradually and predictably – study</title><url>https://www.quantamagazine.org/how-quickly-do-large-language-models-learn-unexpected-skills-20240213/</url></story><parent_chain><item><author>Gisbitus</author><text>Just like it&amp;#x27;s mentioned later in the article: it doesn&amp;#x27;t really matter if you get an addition &lt;i&gt;mostly&lt;/i&gt; right. You either get it right or you don&amp;#x27;t. I still appreciate their effort though, because even after altering the grading system, there were still some emergent abilities.</text></item><item><author>a_wild_dandan</author><text>There are several issues with the study:&lt;p&gt;1. Replacing pass&amp;#x2F;fail accuracy with smoother alternatives (e.g token edit distance) could be a terrible proxy for skill, depending on the task.&lt;p&gt;2. Even by the authors&amp;#x27; metrics, they _still_ find a few potentially emergent abilities.&lt;p&gt;3. Hindsight is 20-20. Yes, we can revisit the data and fiddle until we find transforms that erase emergence from aptitude plots. The fact is, folk used commonplace test accuracy measurements, and the results were unpredictable and surprising. That&amp;#x27;s the true notable phenomenon.&lt;p&gt;I think there&amp;#x27;s value in the paper. Just...don&amp;#x27;t take its conclusions too far.</text></item></parent_chain><comment><author>arka2147483647</author><text>Assume we have a child, and we test him regularly:&lt;p&gt;- Test 1: First he can just draw squiggles on the math test&lt;p&gt;- Test 2: Then he can do arithmetic correctly&lt;p&gt;- Test 3: He fails on the last details on the algebraic calculation.&lt;p&gt;Now, event though he fails on all tests, any reasonable parent would see that he improving nicely, and would be able to work in his chosen field in a year or so.&lt;p&gt;Or alternatively, if we talk about AI, we can set the Test as a threshold, and we see the results are continuously trending upwards, and we can expect the curve to breach the threshold in the future.&lt;p&gt;That is; measuring improvement, instead of pass&amp;#x2F;fail, allows one to predict when we might be able to use the AI for something.</text></comment>
<story><title>“Emergent” abilities in LLMs actually develop gradually and predictably – study</title><url>https://www.quantamagazine.org/how-quickly-do-large-language-models-learn-unexpected-skills-20240213/</url></story><parent_chain><item><author>Gisbitus</author><text>Just like it&amp;#x27;s mentioned later in the article: it doesn&amp;#x27;t really matter if you get an addition &lt;i&gt;mostly&lt;/i&gt; right. You either get it right or you don&amp;#x27;t. I still appreciate their effort though, because even after altering the grading system, there were still some emergent abilities.</text></item><item><author>a_wild_dandan</author><text>There are several issues with the study:&lt;p&gt;1. Replacing pass&amp;#x2F;fail accuracy with smoother alternatives (e.g token edit distance) could be a terrible proxy for skill, depending on the task.&lt;p&gt;2. Even by the authors&amp;#x27; metrics, they _still_ find a few potentially emergent abilities.&lt;p&gt;3. Hindsight is 20-20. Yes, we can revisit the data and fiddle until we find transforms that erase emergence from aptitude plots. The fact is, folk used commonplace test accuracy measurements, and the results were unpredictable and surprising. That&amp;#x27;s the true notable phenomenon.&lt;p&gt;I think there&amp;#x27;s value in the paper. Just...don&amp;#x27;t take its conclusions too far.</text></item></parent_chain><comment><author>raincole</author><text>Human beings do arithmetic problems wrong all the time so I&amp;#x27;m not sure &amp;quot;doing addition 100% right&amp;quot; is a merit of intelligence.&lt;p&gt;I&amp;#x27;m not saying LLM will achieve AGI (I don&amp;#x27;t know if it will, or when it does we&amp;#x27;ll even know). But somehow people seem to be judging AI&amp;#x27;s intelligence with this simple procedural:&lt;p&gt;1. Find a task that AI can&amp;#x27;t do perfectly.
2. Gotcha! AI isn&amp;#x27;t intelligent.&lt;p&gt;It just makes me question humans&amp;#x27; intelligence if anything.</text></comment>