<story><title>RLHF is just barely RL</title><url>https://twitter.com/karpathy/status/1821277264996352246</url></story><parent_chain><item><author>cs702</author><text>Programming has a clear reward function when the problem being solving is well-specified, e.g., &amp;quot;we need a program that grabs data from these three endpoints, combines their data in this manner, and returns it in this JSON format.&amp;quot;&lt;p&gt;The same is true for math. There is a clear reward function when the goal is well-specified, e.g., &amp;quot;we need a sequence of mathematical statements that prove this other important  mathematical statement is true.&amp;quot;</text></item><item><author>discreteevent</author><text>Does programming have a clear reward function? A vague description from a business person is not it. By the time someone (a programmer?) has written a reward function that is clear enough, how would that function look compared to a program?</text></item><item><author>gizmo</author><text>This is why AI coding assistance will leap ahead in the coming years. Chat AI has no clear reward function (basically impossible to judge the quality of responses to open-ended questions like historical causes for a war). Coding AI can write tests, write code, compile, examine failed test cases, search for different coding solutions that satisfy more test cases or rewrite the tests, all in an unsupervised loop. And then whole process can turn into training data for future AI coding models.&lt;p&gt;I expect language models to also get crazy good at mathematical theorem proving. The search space is huge but theorem verification software will provide 100% accurate feedback that makes real reinforcement learning possible. It&amp;#x27;s the combination of vibes (how to approach the proof) and formal verification that works.&lt;p&gt;Formal verification of program correctness never got traction because it&amp;#x27;s so tedious and most of the time approximately correct is good enough. But with LLMs in the mix the equation changes. Having LLMs generate annotations that an engine can use to prove correctness might be the missing puzzle piece.</text></item></parent_chain><comment><author>danpalmer</author><text>I’m not sure I would agree. By the time you’ve written a full spec for it, you may as well have just written a high level programming language anyway. You can make assumptions that minimise the spec needed… but also programming APIs can have defaults so that’s no advantage.&lt;p&gt;I’d suggest that the Python code for your example prompt with reasonable defaults is not actually that far from the prompt itself in terms of the time necessary to write it.&lt;p&gt;However, add tricky details like how you want to handle connection pooling, differing retry strategies, short circuiting based on one of the results, business logic in the data combination step, and suddenly you’ve got a whole design doc in your prompt and you need a senior engineer with good written comms skills to get it to work.</text></comment>
<story><title>RLHF is just barely RL</title><url>https://twitter.com/karpathy/status/1821277264996352246</url></story><parent_chain><item><author>cs702</author><text>Programming has a clear reward function when the problem being solving is well-specified, e.g., &amp;quot;we need a program that grabs data from these three endpoints, combines their data in this manner, and returns it in this JSON format.&amp;quot;&lt;p&gt;The same is true for math. There is a clear reward function when the goal is well-specified, e.g., &amp;quot;we need a sequence of mathematical statements that prove this other important  mathematical statement is true.&amp;quot;</text></item><item><author>discreteevent</author><text>Does programming have a clear reward function? A vague description from a business person is not it. By the time someone (a programmer?) has written a reward function that is clear enough, how would that function look compared to a program?</text></item><item><author>gizmo</author><text>This is why AI coding assistance will leap ahead in the coming years. Chat AI has no clear reward function (basically impossible to judge the quality of responses to open-ended questions like historical causes for a war). Coding AI can write tests, write code, compile, examine failed test cases, search for different coding solutions that satisfy more test cases or rewrite the tests, all in an unsupervised loop. And then whole process can turn into training data for future AI coding models.&lt;p&gt;I expect language models to also get crazy good at mathematical theorem proving. The search space is huge but theorem verification software will provide 100% accurate feedback that makes real reinforcement learning possible. It&amp;#x27;s the combination of vibes (how to approach the proof) and formal verification that works.&lt;p&gt;Formal verification of program correctness never got traction because it&amp;#x27;s so tedious and most of the time approximately correct is good enough. But with LLMs in the mix the equation changes. Having LLMs generate annotations that an engine can use to prove correctness might be the missing puzzle piece.</text></item></parent_chain><comment><author>seanthemon</author><text>&lt;p&gt;&lt;pre&gt;&lt;code&gt;  &amp;gt;when the problem being solving is well-specified
&lt;/code&gt;&lt;/pre&gt;
Phew! Sounds like i&amp;#x27;ll be fine, thank god for product owners.</text></comment>