<story><title>GPT-4 performs significantly worse on coding problems not in its training data</title><url>https://twitter.com/cHHillee/status/1635790330854526981</url></story><parent_chain><item><author>nopinsight</author><text>“Reflection-Based GPT-4 Agent is State-of-the-Art on Code Gen&lt;p&gt;Iteratively refines code, shifting “accuracy bottleneck” from correct code gen to correct test gen&lt;p&gt;HumanEval accuracy:&lt;p&gt;-Reflexion-based GPT-4 88%&lt;p&gt;-GPT-4 67.0%&lt;p&gt;-CodeT 65.8%&lt;p&gt;-PaLM 26.2%”&lt;p&gt;with link to code in the Tweet:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;mobile.twitter.com&amp;#x2F;johnjnay&amp;#x2F;status&amp;#x2F;1639362071807549446&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;mobile.twitter.com&amp;#x2F;johnjnay&amp;#x2F;status&amp;#x2F;16393620718075494...&lt;/a&gt;&lt;p&gt;21% improvement after adding a feedback loop and self-reflection to GPT-4, which just went public 12 days ago. (The approach is based on a preprint published 4 days ago.)&lt;p&gt;Human coders often need a feedback loop and self-reflection to properly “generate” code for problems novel to them as well.&lt;p&gt;-----&lt;p&gt;A larger question: Are we hurling ourselves toward a (near) future of unaligned AGI with self-improvement capabilities?</text></item></parent_chain><comment><author>nopinsight</author><text>Someone asked GPT-4 to build a complete app from scratch. It&amp;#x27;s now on the store. He seems to apply good prompt engineering techniques.&lt;p&gt;Screenshots, video demo, and process here: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;mobile.twitter.com&amp;#x2F;mortenjust&amp;#x2F;status&amp;#x2F;1639276571574894594&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;mobile.twitter.com&amp;#x2F;mortenjust&amp;#x2F;status&amp;#x2F;163927657157489...&lt;/a&gt;&lt;p&gt;It seems plausible to me now that a junior developer position would be hard to find in 2-3 years (I thought it would be ~5 years).</text></comment>
<story><title>GPT-4 performs significantly worse on coding problems not in its training data</title><url>https://twitter.com/cHHillee/status/1635790330854526981</url></story><parent_chain><item><author>nopinsight</author><text>“Reflection-Based GPT-4 Agent is State-of-the-Art on Code Gen&lt;p&gt;Iteratively refines code, shifting “accuracy bottleneck” from correct code gen to correct test gen&lt;p&gt;HumanEval accuracy:&lt;p&gt;-Reflexion-based GPT-4 88%&lt;p&gt;-GPT-4 67.0%&lt;p&gt;-CodeT 65.8%&lt;p&gt;-PaLM 26.2%”&lt;p&gt;with link to code in the Tweet:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;mobile.twitter.com&amp;#x2F;johnjnay&amp;#x2F;status&amp;#x2F;1639362071807549446&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;mobile.twitter.com&amp;#x2F;johnjnay&amp;#x2F;status&amp;#x2F;16393620718075494...&lt;/a&gt;&lt;p&gt;21% improvement after adding a feedback loop and self-reflection to GPT-4, which just went public 12 days ago. (The approach is based on a preprint published 4 days ago.)&lt;p&gt;Human coders often need a feedback loop and self-reflection to properly “generate” code for problems novel to them as well.&lt;p&gt;-----&lt;p&gt;A larger question: Are we hurling ourselves toward a (near) future of unaligned AGI with self-improvement capabilities?</text></item></parent_chain><comment><author>isaacfrond</author><text>the paper: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;pdf&amp;#x2F;2303.11366.pdf&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;pdf&amp;#x2F;2303.11366.pdf&lt;/a&gt;&lt;p&gt;it reminds me of the thinking-fast vs. thinking-slow dichotomy. Current llms are the thinking fast type. Funnily people’s complains about its errors are reminiscent of this. It answers just to quick and only with its instant response neural net. A thinking slow answer would be more akin to a chain of thought answer. Allowing the llm a more flexible platform than CoT promptin might well be the next step. Of course it would als multiply compute cost. So it might not be in your 20$ subscription</text></comment>