<story><title>What can LLMs never do?</title><url>https://www.strangeloopcanon.com/p/what-can-llms-never-do</url></story><parent_chain><item><author>jncfhnb</author><text>I think one should feel comfortable arguing that AGI must be stateful and experience continuous time at least. Such that a plain old LLM is definitively not ever going to be AGI; but an LLM called in a do while true for loop might.</text></item><item><author>dragonwriter</author><text>&amp;gt; There are problems that are easy for human beings but hard for current LLMs (and maybe impossible for them; no one knows). Examples include playing Wordle and predicting cellular automata (including Turing-complete ones like Rule 110). We don’t fully understand why current LLMs are bad at these tasks.&lt;p&gt;I thought we did know for things like playing Wordle, that its because they deal with words as sequence of tokens that correspond to whole words not sequences of letters, so a game that involves dealing with sequences of letters constrained to those that are valid words doesn’t match the way they process information?&lt;p&gt;&amp;gt; Providing an LLM with examples and step-by-step instructions in a prompt means the user is figuring out the “reasoning steps” and handing them to the LLM, instead of the LLM figuring them out by itself. We have “reasoning machines” that are intelligent but seem to be hitting fundamental limits we don’t understand.&lt;p&gt;But providing examples with &lt;i&gt;different&lt;/i&gt;, contextually-appropriate sets of reasoning steps results can enable the model to choose its own, more-or-less appropriate, set of reasoning steps for particular questions not matching the examples.&lt;p&gt;&amp;gt; It’s unclear if better prompting and bigger models using existing attention mechanisms can achieve AGI.&lt;p&gt;Since there is no objective definition of AGI or test for it, there’s no basis for any meaningful speculation on what can or cannot achieve it; discussions about it are quasi-religious, not scientific.</text></item><item><author>cs702</author><text>Fantastic essay. Highly recommended!&lt;p&gt;I agree with all key points:&lt;p&gt;* There are problems that are easy for human beings but hard for &lt;i&gt;current&lt;/i&gt; LLMs (and maybe impossible for them; no one knows). Examples include playing Wordle and predicting cellular automata (including Turing-complete ones like Rule 110). We don&amp;#x27;t fully understand why &lt;i&gt;current&lt;/i&gt; LLMs are bad at these tasks.&lt;p&gt;* Providing an LLM with examples and step-by-step instructions in a prompt means &lt;i&gt;the user&lt;/i&gt; is figuring out the &amp;quot;reasoning steps&amp;quot; and &lt;i&gt;handing them to the LLM&lt;/i&gt;, instead of the LLM figuring them out by itself. We have &amp;quot;reasoning machines&amp;quot; that are intelligent but seem to be hitting fundamental limits we don&amp;#x27;t understand.&lt;p&gt;* It&amp;#x27;s unclear if better prompting and bigger models using existing attention mechanisms can achieve AGI. As a model of computation, attention is very rigid, whereas human brains are always undergoing synaptic plasticity. There may be a more flexible architecture capable of AGI, but we don&amp;#x27;t know it yet.&lt;p&gt;* For now, using current AI models &lt;i&gt;requires&lt;/i&gt; carefully constructing long prompts with right and wrong answers for computational problems, priming the model to reply appropriately, and applying lots of external guardrails (e.g., LLMs acting as agents that review and vote on the answers of other LLMs).&lt;p&gt;* Attention seems to suffer from &amp;quot;goal drift,&amp;quot; making reliability hard without all that external scaffolding.&lt;p&gt;Go read the whole thing.</text></item></parent_chain><comment><author>PopePompus</author><text>I don&amp;#x27;t understand why you believe it must experience continuous time. If you had a system which clearly could reason, which could learn new tasks on its own, which didn&amp;#x27;t hallucinate any more than humans do, but it was only active for the period required for it to complete an assigned task, and was completely dormant otherwise, why would that dormant period disqualify it as AGI?   I agree that such a system should probably not be considered conscious, but I think it&amp;#x27;s an open question whether or not consciousness is required for intelligence.</text></comment>
<story><title>What can LLMs never do?</title><url>https://www.strangeloopcanon.com/p/what-can-llms-never-do</url></story><parent_chain><item><author>jncfhnb</author><text>I think one should feel comfortable arguing that AGI must be stateful and experience continuous time at least. Such that a plain old LLM is definitively not ever going to be AGI; but an LLM called in a do while true for loop might.</text></item><item><author>dragonwriter</author><text>&amp;gt; There are problems that are easy for human beings but hard for current LLMs (and maybe impossible for them; no one knows). Examples include playing Wordle and predicting cellular automata (including Turing-complete ones like Rule 110). We don’t fully understand why current LLMs are bad at these tasks.&lt;p&gt;I thought we did know for things like playing Wordle, that its because they deal with words as sequence of tokens that correspond to whole words not sequences of letters, so a game that involves dealing with sequences of letters constrained to those that are valid words doesn’t match the way they process information?&lt;p&gt;&amp;gt; Providing an LLM with examples and step-by-step instructions in a prompt means the user is figuring out the “reasoning steps” and handing them to the LLM, instead of the LLM figuring them out by itself. We have “reasoning machines” that are intelligent but seem to be hitting fundamental limits we don’t understand.&lt;p&gt;But providing examples with &lt;i&gt;different&lt;/i&gt;, contextually-appropriate sets of reasoning steps results can enable the model to choose its own, more-or-less appropriate, set of reasoning steps for particular questions not matching the examples.&lt;p&gt;&amp;gt; It’s unclear if better prompting and bigger models using existing attention mechanisms can achieve AGI.&lt;p&gt;Since there is no objective definition of AGI or test for it, there’s no basis for any meaningful speculation on what can or cannot achieve it; discussions about it are quasi-religious, not scientific.</text></item><item><author>cs702</author><text>Fantastic essay. Highly recommended!&lt;p&gt;I agree with all key points:&lt;p&gt;* There are problems that are easy for human beings but hard for &lt;i&gt;current&lt;/i&gt; LLMs (and maybe impossible for them; no one knows). Examples include playing Wordle and predicting cellular automata (including Turing-complete ones like Rule 110). We don&amp;#x27;t fully understand why &lt;i&gt;current&lt;/i&gt; LLMs are bad at these tasks.&lt;p&gt;* Providing an LLM with examples and step-by-step instructions in a prompt means &lt;i&gt;the user&lt;/i&gt; is figuring out the &amp;quot;reasoning steps&amp;quot; and &lt;i&gt;handing them to the LLM&lt;/i&gt;, instead of the LLM figuring them out by itself. We have &amp;quot;reasoning machines&amp;quot; that are intelligent but seem to be hitting fundamental limits we don&amp;#x27;t understand.&lt;p&gt;* It&amp;#x27;s unclear if better prompting and bigger models using existing attention mechanisms can achieve AGI. As a model of computation, attention is very rigid, whereas human brains are always undergoing synaptic plasticity. There may be a more flexible architecture capable of AGI, but we don&amp;#x27;t know it yet.&lt;p&gt;* For now, using current AI models &lt;i&gt;requires&lt;/i&gt; carefully constructing long prompts with right and wrong answers for computational problems, priming the model to reply appropriately, and applying lots of external guardrails (e.g., LLMs acting as agents that review and vote on the answers of other LLMs).&lt;p&gt;* Attention seems to suffer from &amp;quot;goal drift,&amp;quot; making reliability hard without all that external scaffolding.&lt;p&gt;Go read the whole thing.</text></item></parent_chain><comment><author>pixl97</author><text>A consistent stateful experience may be needed, but not sure about continuous time. I mean human consciousness doesn&amp;#x27;t do that.</text></comment>