<story><title>ChatGPT&apos;s Chess Elo is 1400</title><url>https://dkb.blog/p/chatgpts-chess-elo-is-1400</url></story><parent_chain><item><author>latexr</author><text>&amp;gt; These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…)&lt;p&gt;&amp;gt; With this prompt ChatGPT almost always plays fully legal games.&lt;p&gt;&amp;gt; Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table (…)&lt;p&gt;&amp;gt; (…) with GPT4 (…) in the two games I attempted, it made numerous illegal moves.&lt;p&gt;So you’ve ostensibly¹ found a way to reduce the error rate and then deliberately ignored the cases where it failed. In short: it may play valid chess &lt;i&gt;under certain conditions&lt;/i&gt; but can’t be trusted to do so. That doesn’t contradict previous findings.&lt;p&gt;¹ 19 games is a small sample and the supposedly more advanced system failed in your tries.</text></item></parent_chain><comment><author>vidarh</author><text>Fuller context from the article:&lt;p&gt;&amp;gt; Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” &lt;i&gt;So whenever it wanted to make an illegal move, it resigned.&lt;/i&gt;&lt;p&gt;(my emphasis)&lt;p&gt;So the illegal moves are at least part of the reasons for the 6 losses, and factored into the rating. Quickly scanning the game, it seems 3 of the losses ended in checkmate, so that leaves 3 illegal moves in 19 games.&lt;p&gt;Could be better, but for a system not intentionally built to play chess, it&amp;#x27;s pretty decent.</text></comment>
<story><title>ChatGPT&apos;s Chess Elo is 1400</title><url>https://dkb.blog/p/chatgpts-chess-elo-is-1400</url></story><parent_chain><item><author>latexr</author><text>&amp;gt; These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…)&lt;p&gt;&amp;gt; With this prompt ChatGPT almost always plays fully legal games.&lt;p&gt;&amp;gt; Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table (…)&lt;p&gt;&amp;gt; (…) with GPT4 (…) in the two games I attempted, it made numerous illegal moves.&lt;p&gt;So you’ve ostensibly¹ found a way to reduce the error rate and then deliberately ignored the cases where it failed. In short: it may play valid chess &lt;i&gt;under certain conditions&lt;/i&gt; but can’t be trusted to do so. That doesn’t contradict previous findings.&lt;p&gt;¹ 19 games is a small sample and the supposedly more advanced system failed in your tries.</text></item></parent_chain><comment><author>ballenf</author><text>The illegal moves were counted as losses&amp;#x2F;resignations, not ignored.</text></comment>