<story><title>Getting 50% (SoTA) on Arc-AGI with GPT-4o</title><url>https://redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt</url></story><parent_chain><item><author>mikeknoop</author><text>(ARC Prize co-founder here).&lt;p&gt;Ryan&amp;#x27;s work is legitimately interesting and novel &amp;quot;LLM reasoning&amp;quot; research! The core idea:&lt;p&gt;&amp;gt; get GPT-4o to generate around 8,000 python programs which attempt to implement the transformation, select a program which is right on all the examples (usually there are 3 examples), and then submit the output this function produces when applied to the additional test input(s)&lt;p&gt;Roughly, he&amp;#x27;s implemented an outer loop and using 4o to sample reasoning traces&amp;#x2F;programs from training data and test. Hybrid DL + program synthesis approaches are solutions we&amp;#x27;d love to see more of.&lt;p&gt;A couple important notes:&lt;p&gt;1. this result is on the public eval set vs private set (ARC Prize $).&lt;p&gt;2. the current private set SOTA ~35% solution also performed ~50% on the public set. so this new result &lt;i&gt;might&lt;/i&gt; be SOTA but hasn&amp;#x27;t been validated or scrutinized yet.&lt;p&gt;All said, I do expect verified public set results to flow down to the private set over time. We&amp;#x27;ll be publishing all the SOTA scores and open source reproductions here once available: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;arcprize.org&amp;#x2F;leaderboard&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;arcprize.org&amp;#x2F;leaderboard&lt;/a&gt;&lt;p&gt;EDIT: also, congrats and kudos to Ryan for achieving this and putting the effort in to document and share his approach. we hope to inspire more frontier AI research sharing like this</text></item></parent_chain><comment><author>refreshingdrink</author><text>Also worth nothing that Ryan mentions&lt;p&gt;&amp;gt; In addition to iterating on the training set, I also did a small amount of iteration on a 100 problem subset of the public test set&lt;p&gt;and&lt;p&gt;&amp;gt; it&amp;#x27;s unfortunate that these sets aren’t IID: it makes iteration harder and more confusing&lt;p&gt;It’s not unfortunate: generalizing beyond the training distribution is a crucial part of intelligence that ARC is trying to measure! Among other reasons, developing with test-set data is a bad practice in ML because it hides the difficulty this challenge. Even worse, writing about a bunch of tricks that help results on this subset is extending the test-set leakage the blog post&amp;#x27;s readers. This is why I&amp;#x27;m glad the ARC Prize has a truly hidden test set</text></comment>
<story><title>Getting 50% (SoTA) on Arc-AGI with GPT-4o</title><url>https://redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt</url></story><parent_chain><item><author>mikeknoop</author><text>(ARC Prize co-founder here).&lt;p&gt;Ryan&amp;#x27;s work is legitimately interesting and novel &amp;quot;LLM reasoning&amp;quot; research! The core idea:&lt;p&gt;&amp;gt; get GPT-4o to generate around 8,000 python programs which attempt to implement the transformation, select a program which is right on all the examples (usually there are 3 examples), and then submit the output this function produces when applied to the additional test input(s)&lt;p&gt;Roughly, he&amp;#x27;s implemented an outer loop and using 4o to sample reasoning traces&amp;#x2F;programs from training data and test. Hybrid DL + program synthesis approaches are solutions we&amp;#x27;d love to see more of.&lt;p&gt;A couple important notes:&lt;p&gt;1. this result is on the public eval set vs private set (ARC Prize $).&lt;p&gt;2. the current private set SOTA ~35% solution also performed ~50% on the public set. so this new result &lt;i&gt;might&lt;/i&gt; be SOTA but hasn&amp;#x27;t been validated or scrutinized yet.&lt;p&gt;All said, I do expect verified public set results to flow down to the private set over time. We&amp;#x27;ll be publishing all the SOTA scores and open source reproductions here once available: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;arcprize.org&amp;#x2F;leaderboard&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;arcprize.org&amp;#x2F;leaderboard&lt;/a&gt;&lt;p&gt;EDIT: also, congrats and kudos to Ryan for achieving this and putting the effort in to document and share his approach. we hope to inspire more frontier AI research sharing like this</text></item></parent_chain><comment><author>refibrillator</author><text>Do you have any perspectives to share on Ryan&amp;#x27;s observation of a potential scaling law for these tasks and his comment that &amp;quot;ARC-AGI will be one benchmark among many that just gets solved by scale&amp;quot;?</text></comment>