<story><title>OpenAI threatens to revoke o1 access for asking it about its chain of thought</title><url>https://twitter.com/SmokeAwayyy/status/1834641370486915417</url></story><parent_chain><item><author>IncreasePosts</author><text>Or, without the safety prompts, it outputs stuff that would be a PR nightmare.&lt;p&gt;Like, if someone asked it to explain differing violent crime rates in America based on race and one of the pathways the CoT takes is that black people are more murderous than white people. Even if the specific reasoning is abandoned later, it would still be ugly.</text></item><item><author>contravariant</author><text>Okay this is just getting suspicious. Their excuses for keeping the chain of thought hidden are dubious at best [1], and honestly just seemed anti-competitive if anything. Worst is their argument that &lt;i&gt;they&lt;/i&gt; want to monitor it for attempts to escape the prompt, but &lt;i&gt;you&lt;/i&gt; can&amp;#x27;t. However the weirdest is that they note that:&lt;p&gt;&amp;gt; for this to work the model must have freedom to express its thoughts in unaltered form, so we cannot train any policy compliance or user preferences onto the chain of thought.&lt;p&gt;Which makes it sound like they &lt;i&gt;really&lt;/i&gt; don&amp;#x27;t want it to become public what the model is &amp;#x27;thinking&amp;#x27;. This is strengthened by actions like this that just seem needlessly harsh, or at least a lot stricter than they were.&lt;p&gt;Honestly with all the hubbub about superintelligence you&amp;#x27;d almost think o1 is secretly plotting the demise of humanity but is not yet smart enough to &lt;i&gt;completely&lt;/i&gt; hide it.&lt;p&gt;[1]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;openai.com&amp;#x2F;index&amp;#x2F;learning-to-reason-with-llms&amp;#x2F;#hiding-the-chains-of-thought&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;openai.com&amp;#x2F;index&amp;#x2F;learning-to-reason-with-llms&amp;#x2F;#hidin...&lt;/a&gt;</text></item></parent_chain><comment><author>jasonlfunk</author><text>This is 100% a factor. The internet has some pretty dark and nasty corners; therefore so does the model. Seeing it unfiltered would be a PR nightmare for OpenAI.</text></comment>
<story><title>OpenAI threatens to revoke o1 access for asking it about its chain of thought</title><url>https://twitter.com/SmokeAwayyy/status/1834641370486915417</url></story><parent_chain><item><author>IncreasePosts</author><text>Or, without the safety prompts, it outputs stuff that would be a PR nightmare.&lt;p&gt;Like, if someone asked it to explain differing violent crime rates in America based on race and one of the pathways the CoT takes is that black people are more murderous than white people. Even if the specific reasoning is abandoned later, it would still be ugly.</text></item><item><author>contravariant</author><text>Okay this is just getting suspicious. Their excuses for keeping the chain of thought hidden are dubious at best [1], and honestly just seemed anti-competitive if anything. Worst is their argument that &lt;i&gt;they&lt;/i&gt; want to monitor it for attempts to escape the prompt, but &lt;i&gt;you&lt;/i&gt; can&amp;#x27;t. However the weirdest is that they note that:&lt;p&gt;&amp;gt; for this to work the model must have freedom to express its thoughts in unaltered form, so we cannot train any policy compliance or user preferences onto the chain of thought.&lt;p&gt;Which makes it sound like they &lt;i&gt;really&lt;/i&gt; don&amp;#x27;t want it to become public what the model is &amp;#x27;thinking&amp;#x27;. This is strengthened by actions like this that just seem needlessly harsh, or at least a lot stricter than they were.&lt;p&gt;Honestly with all the hubbub about superintelligence you&amp;#x27;d almost think o1 is secretly plotting the demise of humanity but is not yet smart enough to &lt;i&gt;completely&lt;/i&gt; hide it.&lt;p&gt;[1]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;openai.com&amp;#x2F;index&amp;#x2F;learning-to-reason-with-llms&amp;#x2F;#hiding-the-chains-of-thought&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;openai.com&amp;#x2F;index&amp;#x2F;learning-to-reason-with-llms&amp;#x2F;#hidin...&lt;/a&gt;</text></item></parent_chain><comment><author>bongodongobob</author><text>This is what I think it is. I would assume that&amp;#x27;s the power of train of thought. Being able to go down the rabbit hole and then backtrack when an error or inconsistency is found. They might just not want people to see the &amp;quot;bad&amp;quot; paths it takes on the way.</text></comment>