<story><title>The False Promise of Chomskyism</title><url>https://scottaaronson.blog/?p=7094</url></story><parent_chain><item><author>mjburgess</author><text>Scott mischaracterises the reasons, which concern capacities of the system &lt;i&gt;not&lt;/i&gt; properties of its output.&lt;p&gt;(1) It lacks the capacity to interpret sentences. It&amp;#x27;s &lt;i&gt;output&lt;/i&gt; possess the property of seeming-to-have-interpreted.&lt;p&gt;(2) It lacks the capacity to learn. Learning is, in part, explaining; it is, in part, causal contact with an environment. It&amp;#x27;s output has the property of seeming-to-be-generated-by-a-learner.&lt;p&gt;(3) It lacks the capacity to speak either truthfully or otherwise. Lacking the capacity to explain, because at least it lacks the capacity to imagine (counterfactuals), it cannot thereby determine whether a claim accords with the way the world is. It&amp;#x27;s output has the property: seeming-to-be-true.&lt;p&gt;(4) It lacks the capacity to be moral. For all the above, and more: it lacks the capacity to &lt;i&gt;care&lt;/i&gt;. It&amp;#x27;s output has the property: seeming-to-take-moral-stands.&lt;p&gt;The &amp;quot;Jesuit&amp;quot; here, I fear, is the AI fanatic who have turned their telescopes away from reality (intelligence as it actually exists), and have stolen the lenses to read scripture (intelligence as theorised by partisan academics).&lt;p&gt;One has to agree with Chomsky here at the end, &amp;quot;given the amorality, faux science and linguistic incompetence of these systems, we can only laugh or cry at their popularity.&amp;quot;&lt;p&gt;The morality point I think should hit home most extremely: how horrifying to treat generated text output as-if it were constructed by an agent who &lt;i&gt;cares&lt;/i&gt;.&lt;p&gt;Apparent properties of the output are &lt;i&gt;not&lt;/i&gt; properties of the system; ChatGPT does not think, know, care, intend, speak, commnuicate, etc.&lt;p&gt;One can only &amp;quot;laugh or cry&amp;quot; at how absurd this sales pitch: what a horror to be invited to treat ChatGPT as possessing any of these capacities, simply because correlations across a whole internet of text seems to.</text></item></parent_chain><comment><author>FredPret</author><text>He writes a great reply to this in his own comment section:&lt;p&gt;&amp;quot;Michael #3: In a certain sense you’re right. The language models now being adopted by millions of programmers don’t write working code; they only seem-to-write-working-code. They’re not, unfortunately, already doing millions of students’ homework for them; they’re only seeming-to-do-the-homework. Even if in a few years they help me and my colleagues do our research, they won’t actually be helping, but only seeming-to-help. They won’t change civilization; they’ll only seem-to-change-it.&amp;quot;&lt;p&gt;[0] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;scottaaronson.blog&amp;#x2F;?p=7094#comment-1947189&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;scottaaronson.blog&amp;#x2F;?p=7094#comment-1947189&lt;/a&gt;</text></comment>
<story><title>The False Promise of Chomskyism</title><url>https://scottaaronson.blog/?p=7094</url></story><parent_chain><item><author>mjburgess</author><text>Scott mischaracterises the reasons, which concern capacities of the system &lt;i&gt;not&lt;/i&gt; properties of its output.&lt;p&gt;(1) It lacks the capacity to interpret sentences. It&amp;#x27;s &lt;i&gt;output&lt;/i&gt; possess the property of seeming-to-have-interpreted.&lt;p&gt;(2) It lacks the capacity to learn. Learning is, in part, explaining; it is, in part, causal contact with an environment. It&amp;#x27;s output has the property of seeming-to-be-generated-by-a-learner.&lt;p&gt;(3) It lacks the capacity to speak either truthfully or otherwise. Lacking the capacity to explain, because at least it lacks the capacity to imagine (counterfactuals), it cannot thereby determine whether a claim accords with the way the world is. It&amp;#x27;s output has the property: seeming-to-be-true.&lt;p&gt;(4) It lacks the capacity to be moral. For all the above, and more: it lacks the capacity to &lt;i&gt;care&lt;/i&gt;. It&amp;#x27;s output has the property: seeming-to-take-moral-stands.&lt;p&gt;The &amp;quot;Jesuit&amp;quot; here, I fear, is the AI fanatic who have turned their telescopes away from reality (intelligence as it actually exists), and have stolen the lenses to read scripture (intelligence as theorised by partisan academics).&lt;p&gt;One has to agree with Chomsky here at the end, &amp;quot;given the amorality, faux science and linguistic incompetence of these systems, we can only laugh or cry at their popularity.&amp;quot;&lt;p&gt;The morality point I think should hit home most extremely: how horrifying to treat generated text output as-if it were constructed by an agent who &lt;i&gt;cares&lt;/i&gt;.&lt;p&gt;Apparent properties of the output are &lt;i&gt;not&lt;/i&gt; properties of the system; ChatGPT does not think, know, care, intend, speak, commnuicate, etc.&lt;p&gt;One can only &amp;quot;laugh or cry&amp;quot; at how absurd this sales pitch: what a horror to be invited to treat ChatGPT as possessing any of these capacities, simply because correlations across a whole internet of text seems to.</text></item></parent_chain><comment><author>krona</author><text>(1) As it is with humans. It&amp;#x27;s not possible to attach a debugger to a human brain and determine the nature of understanding; you have to ask the human questions about that understanding and interpret the response yourself.&lt;p&gt;(2) This is a strange one to me. It&amp;#x27;s a bit like saying an essential quality of a human is to experience the world as a human, and therefore nothing except a human can be human.&lt;p&gt;(3) If only all humans met this arbitrary standard of secular scientific rationalism.&lt;p&gt;(4) I&amp;#x27;d say 5% of humanity fail to meet this criteria.</text></comment>