<story><title>Inductive or deductive? Rethinking the fundamental reasoning abilities of LLMs</title><url>https://arxiv.org/abs/2408.00114</url></story><parent_chain></parent_chain><comment><author>godelski</author><text>I&amp;#x27;m really tired of these papers and experiments.&lt;p&gt;You cannot test reasoning when you don&amp;#x27;t know what&amp;#x27;s in the training set. You have to be able to differentiate reasoning from memorization, and that&amp;#x27;s not trivial.&lt;p&gt;Moreso, the results look to confirm that at least some memorization is going on. Do we really not think GPT has extensively been trained on arithmetic in base 10, 8, and 16? This seems like a terrible prior.  Even if not explicitly, how much code has it read that performs these tasks. How many web pages, tutorials, Reddit posts cover oct and hex? They also haven&amp;#x27;t defined zero shot correctly. Arithmetic in these bases aren&amp;#x27;t 0-shot. They&amp;#x27;re explicitly in distribution...&lt;p&gt;I&amp;#x27;m unsure about base 9 and 11. It&amp;#x27;s pretty interesting to see that GPT 4 is much better at these. Anyone know why? Did they train on these? More bases? Doesn&amp;#x27;t seem unreasonable but I don&amp;#x27;t know.&lt;p&gt;The experimentation is also extremely lacking. The arithmetic questions only have 1000 tests where they add two digits. This is certainly in the training data. I&amp;#x27;m also unconvinced by the syntax reasoning tasks since the transformer (attention) architecture seems to be designed for this. I&amp;#x27;m also unconvinced these tasks aren&amp;#x27;t in training. Caesar ciphers are also certainly in the training data.&lt;p&gt;The prompts are also odd and I guess that&amp;#x27;s why they&amp;#x27;re in the appendix. For example, getting GPT to be better at math or many tasks by having it write python code is not novel.&lt;p&gt;There&amp;#x27;s some stuff here but this really doesn&amp;#x27;t seem like a lot of work for 12 people from a top university and a trillion dollar company. It&amp;#x27;s odd to see that many people when the experiments can be run in a fairly short time.</text></comment>
<story><title>Inductive or deductive? Rethinking the fundamental reasoning abilities of LLMs</title><url>https://arxiv.org/abs/2408.00114</url></story><parent_chain></parent_chain><comment><author>hydrox24</author><text>Is there a good reason to exclude abductive reasoning from an analysis like this? It&amp;#x27;s even considered by at least one of the referenced papers (Fangzhi 2023a).&lt;p&gt;Abductive reasoning is common in day-to-day life. It seeks the best explanation for some (often incomplete) observations, and reaches conclusions without certainty. I would have thought it would be important to assess for LLMs.</text></comment>