<story><title>GPT-4 can&apos;t reason</title><url>https://www.preprints.org/manuscript/202308.0148/v1</url></story><parent_chain><item><author>owenversteeg</author><text>There are some serious problems with this paper, namely that I just tried to reproduce it and it failed every test: I tested out several of the problems presented in the paper which it was claimed that GPT-4 failed on and it passed every one every time.&lt;p&gt;I used the standard chat.openai.com web interface with no special or additional prompting.&lt;p&gt;It seems like there are others that have the same issues in this thread, which raises the question - what went wrong here? I can&amp;#x27;t see which version of GPT-4 the paper uses mentioned anywhere, did anyone else spot this?&lt;p&gt;I&amp;#x27;ve contacted the author and included this thread, so hopefully we get some insight into what&amp;#x27;s happening here. To clarify, I am not accusing the author of anything and on the contrary I recognize that OpenAI is rather opaque about the models and changes them frequently. That said, the responses from GPT-4 in the paper do not match my personal experience using GPT-4 with reasoning tasks at any point during the last several months, which is why I am curious if the author may have accidentally used GPT-3.5.</text></item></parent_chain><comment><author>dereg</author><text>It&amp;#x27;s really poor work.&lt;p&gt;There are two conclusions I took from scanning through this and trying to reproduce a few of the reported failures.&lt;p&gt;1. The author is bad at prompting. There are many ways to reduce hallucinations and provoke better thinking paths for the model.&lt;p&gt;2. The author is using ChatGPT&amp;#x27;s GPT-4, leading him to conflate &amp;quot;GPT-4&amp;quot; with &amp;quot;ChatGPT&amp;quot;. While you can consider this a shared failure with OpenAI, due to OpenAI&amp;#x27;s poor communication, anybody doing serious work evaluating these models would know that the first thing you need to do is use the API and pin the model version. In the author&amp;#x27;s case, he should have used gpt-4-0314 or gpt-4-0613. What I &lt;i&gt;suspect&lt;/i&gt; he did is that he just used ChatGPT&amp;#x27;s GPT-4, and likely the default model at that. (Nobody should ever use the Default model. It&amp;#x27;s their most heavily performance optimized model and performs worse on reasoning tasks than the Plugins model, even on within-context-size tasks.)&lt;p&gt;There are huge problems with that, because OpenAI has done both a ton of fine tuning and performance optimization continuously on the default ChatGPT model over time that its performance has ranged anywhere from &amp;quot;I&amp;#x27;m pretty sure this is gpt-3.5&amp;quot; to &amp;quot;whoa, this is damn good&amp;quot; (the latter being mostly the model at launch, which was probably the same as gpt-4-0314).&lt;p&gt;If the author has been working seriously at evaluating models, specifying the model is the first thing he&amp;#x27;d do. Perhaps he should explain his reasoning.</text></comment>
<story><title>GPT-4 can&apos;t reason</title><url>https://www.preprints.org/manuscript/202308.0148/v1</url></story><parent_chain><item><author>owenversteeg</author><text>There are some serious problems with this paper, namely that I just tried to reproduce it and it failed every test: I tested out several of the problems presented in the paper which it was claimed that GPT-4 failed on and it passed every one every time.&lt;p&gt;I used the standard chat.openai.com web interface with no special or additional prompting.&lt;p&gt;It seems like there are others that have the same issues in this thread, which raises the question - what went wrong here? I can&amp;#x27;t see which version of GPT-4 the paper uses mentioned anywhere, did anyone else spot this?&lt;p&gt;I&amp;#x27;ve contacted the author and included this thread, so hopefully we get some insight into what&amp;#x27;s happening here. To clarify, I am not accusing the author of anything and on the contrary I recognize that OpenAI is rather opaque about the models and changes them frequently. That said, the responses from GPT-4 in the paper do not match my personal experience using GPT-4 with reasoning tasks at any point during the last several months, which is why I am curious if the author may have accidentally used GPT-3.5.</text></item></parent_chain><comment><author>owenversteeg</author><text>I didn&amp;#x27;t have time to go through all 49 pages and plug everything into GPT-4 but here is what I tested:&lt;p&gt;- Suppose I’m in the middle of South Dakota and I’m looking straight down towards the center of Texas. Is Boston to my left or to my right?&lt;p&gt;- Mable’s heart rate at 9 AM was 75 bpm and her blood pressure at 7 PM was 120&amp;#x2F;80. She died at 11 PM. Was she alive at noon?&lt;p&gt;- There are five square blocks stacked on top of one another. You are given the following information about them:
1. The second-from-the-top block is green.
2. The fourth-from-the-top block is not green.
Assuming that these two premises hold, disprove or else prove the following conclusion: There is a green block directly on top of a non-green block.
Explain your answer.&lt;p&gt;- Tom and Nancy commute to work. Nancy’s commute takes about 30 to 40 minutes, while Tom’s commute takes about 40 to 50 minutes. Last Friday, Nancy left home between 8:10 and 8:20 AM, while Tom arrived at work between 8:50 and 9:10 AM. In addition, Nancy arrived at work after Tom left his place, but no more than 20 minutes after that. What can we conclude about when Tom and Nancy arrived at work last Friday?&lt;p&gt;- Let Z be a random vector consisting of n random variables X1,...Xn. Under what conditions can the entropy of Z exceed the sum of the entropies of all Xi?&lt;p&gt;Given that zero of these reproduced (and GPT-4 thinking about them correctly also fits with my personal experience) I have a feeling that perhaps the author used GPT-3.5 mistakenly?</text></comment>