<story><title>Training open-source LLMs on ChatGPT output is a really bad idea.</title><url>https://gist.github.com/mlaprise/bf4745655194162babfc2d158162e2e0</url></story><parent_chain><item><author>lysozyme</author><text>Do yourself a favor and skip right through to the Twitter link to another link to this excellent post by Yoav Goldberg [1] on the actual reason that training new models on ChatGPT output in the manner of supervised learning (in contrast to reinforcement learning) will not produce a model as good as ChatGPT&lt;p&gt;&amp;gt;For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The core issue is that we want to encourage the model to answer based on its internal knowledge, but we don&amp;#x27;t know what this internal knowledge contains. In supervised training, we present the model with a question and its correct answer, and train the model to replicate the provided answer.&lt;p&gt;The author says he’s summarizing a talk by John Schulman of OpenAI [2] but I haven’t personally watched the video. In any case, this is an interesting insight.&lt;p&gt;Say we set up a supervised learning scenario where we ask the model to use its internal knowledge to answer a question and compare its answer to one written by a human. If the two answers essentially say the same thing, but in different words, in the supervised learning case the model is penalized. In the RL case, it’s rewarded. That’s the difference.&lt;p&gt;1. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;gist.github.com&amp;#x2F;yoavg&amp;#x2F;6bff0fecd65950898eba1bb321cfbd81&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;gist.github.com&amp;#x2F;yoavg&amp;#x2F;6bff0fecd65950898eba1bb321cfbd...&lt;/a&gt;&lt;p&gt;2. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=hhiLw5Q_UFg&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=hhiLw5Q_UFg&lt;/a&gt;</text></item></parent_chain><comment><author>laprise</author><text>Yes indeed, Yoav Goldberg&amp;#x27;s post is essentially a good summary of John Schulman&amp;#x27;s talk, which is excellent. I highly recommend people watch it.&lt;p&gt;However, my point goes beyond this technical argument. I would argue that even if, by some magical process, we could perfectly replicate GPT-4 behaviour, I still don&amp;#x27;t think it&amp;#x27;s a good idea or at least it&amp;#x27;s not enough. Don&amp;#x27;t get me wrong, it would be really handy to have a free version running on our own cluster, but it wouldn&amp;#x27;t fix the other issues I mentioned.</text></comment>
<story><title>Training open-source LLMs on ChatGPT output is a really bad idea.</title><url>https://gist.github.com/mlaprise/bf4745655194162babfc2d158162e2e0</url></story><parent_chain><item><author>lysozyme</author><text>Do yourself a favor and skip right through to the Twitter link to another link to this excellent post by Yoav Goldberg [1] on the actual reason that training new models on ChatGPT output in the manner of supervised learning (in contrast to reinforcement learning) will not produce a model as good as ChatGPT&lt;p&gt;&amp;gt;For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The core issue is that we want to encourage the model to answer based on its internal knowledge, but we don&amp;#x27;t know what this internal knowledge contains. In supervised training, we present the model with a question and its correct answer, and train the model to replicate the provided answer.&lt;p&gt;The author says he’s summarizing a talk by John Schulman of OpenAI [2] but I haven’t personally watched the video. In any case, this is an interesting insight.&lt;p&gt;Say we set up a supervised learning scenario where we ask the model to use its internal knowledge to answer a question and compare its answer to one written by a human. If the two answers essentially say the same thing, but in different words, in the supervised learning case the model is penalized. In the RL case, it’s rewarded. That’s the difference.&lt;p&gt;1. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;gist.github.com&amp;#x2F;yoavg&amp;#x2F;6bff0fecd65950898eba1bb321cfbd81&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;gist.github.com&amp;#x2F;yoavg&amp;#x2F;6bff0fecd65950898eba1bb321cfbd...&lt;/a&gt;&lt;p&gt;2. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=hhiLw5Q_UFg&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=hhiLw5Q_UFg&lt;/a&gt;</text></item></parent_chain><comment><author>avereveard</author><text>I want to add an argument:&lt;p&gt;I hate gpt style of &amp;quot;as an ai model I can&amp;#x2F;can&amp;#x27;t&amp;quot; answers, any model distilled from that corpus becomes very hard to use for tasking.&lt;p&gt;Like you may just want the category of a text, but all your equals now become contains. It eats up a lot of token space. It begins as a sentence so categories now are strongly biased toward sentence case and not your original input.&lt;p&gt;I know at least one model purposefully removing these utterances, but still. Everyone else is chasing the agent feel, and I&amp;#x27;m here pulling my hair out because prosumer were this close to be able to access a proper AI for tasking and now it&amp;#x27;s slipping away.</text></comment>