<story><title>AI’s Language Problem</title><url>https://www.technologyreview.com/s/602094/ais-language-problem/?set=602129</url></story><parent_chain><item><author>Houshalter</author><text>I believe we can get AI from just text. Obviously that won&amp;#x27;t work for babies, because babies get bored quickly looking at text. AIs can be forced to read billions of words in mere hours!&lt;p&gt;Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words. The famous example is the vector &amp;quot;king&amp;quot; minus the vector for &amp;quot;man&amp;quot; plus &amp;quot;woman&amp;quot; equals &amp;quot;queen&amp;quot;. It&amp;#x27;s learning the meaning of words, and the relationships between them.&lt;p&gt;Recurrent NNs, using similar techniques, but with much more complexity, can learn to predict the next word in a sentence very well. They can learn to write responses that are almost indistinguishable from humans. And it&amp;#x27;s incredible this works at all, given RNNs have only a few thousand neurons at most, and a few days of training, compared to humans&amp;#x27; billions of neurons trained over a lifetime.&lt;p&gt;All of the information of our world is contained in text. Humans have produced billions of books, papers, articles, and internet comments. Billions of times more information than any human could read in their entire lifetime. Any information you can imagine is contained in text somewhere. I don&amp;#x27;t think it&amp;#x27;s necessary for AIs to be able to see, or interact with the world in any way.&lt;p&gt;If you can predict the word a human would say next, with enough accuracy, then you could also produce answers indistinguishable from theirs. Meaning you could pass the Turing test, and perform any language task they could do just as well. So language prediction alone may be sufficient for AGI.&lt;p&gt;This is the theory behind the Hutter Prize, which proposes that predicting (compressing) wikipedia&amp;#x27;s text is a measure of AI progress. The Hutter Prize isn&amp;#x27;t perfect (it&amp;#x27;s only a sample of wikipedia, which is very small compared to all the text humans have produced), but the idea is solid.</text></item><item><author>erikpukinskis</author><text>No one would ever imagine that locking a baby in a featureless room with a giant stack of books would give them general intelligence. I don&amp;#x27;t understand why AI researchers think it will work for AIs. They need bodies that are biologically connected with the rest of the biosphere, with an intrinsic biological imperative, if they are ever to understand the world. I&amp;#x27;m not saying they have to be exactly like us, but they will only be able to understand us to the extent that they have body parts and social experiences that are analogous to ours.&lt;p&gt;This isn&amp;#x27;t an engineering problem, it&amp;#x27;s a philosophical problem: We are blind to most things. We can only see what we can relate to personally. We can use language and other symbol systems to expand our understanding by permuting and recombining our personal experiences, but everything is grounded in our interactive developmental trajectory.&lt;p&gt;The kitten-in-a-cart experiment demonstrates this clearly: &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;io9.gizmodo.com&amp;#x2F;the-seriously-creepy-two-kitten-experiment-1442107174&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;io9.gizmodo.com&amp;#x2F;the-seriously-creepy-two-kitten-exper...&lt;/a&gt;  Interaction is crucial for perception. Sensation is not experience.&lt;p&gt;And here&amp;#x27;s the rub: Once you give an AI an animal-like personal developmental trajectory to use for grounding the semantics of their symbol systems, you end up with something which is not particularly different or better than a human cyborg.</text></item></parent_chain><comment><author>ACow_Adonis</author><text>&amp;gt; All of the information of our world is contained in text&lt;p&gt;Communicate the concept of &amp;quot;green&amp;quot; to me in text.&lt;p&gt;The sound of a dog barking, a motor turning over, a sonic boom, or the experience of a Doppler shift.  Beethoven&amp;#x27;s symphony.&lt;p&gt;Sour. Sweet. What does &amp;quot;mint&amp;quot; taste like? Shame. Merit.  Learn facial and object recognition via text.&lt;p&gt;Vertigo.&lt;p&gt;Tell a boxer how to box by reading?&lt;p&gt;Hand eye coordination, bodies in 3 dimensional space.&lt;p&gt;Look, I love text, maybe even more than yourself.  But all these things imbibe, structure and influence or text, but are not contained in them.&lt;p&gt;To make substantial inroads to something that looks like human esque AI, text is not enough.  The division of these fields are artificial and based on our current limited tech and the specialisation of our researchers, faculties and limitations.&lt;p&gt;When we read, we play back memories, visions, sounds, feelings, etc, and inherent ideas gained through experience of ourselves as physical bodies in space.&lt;p&gt;Strong AI, at least to be vaguely recognised as such, must work with algorithms and machinery that understand these things, but which then works at that next level of abstraction to combine them into proper human type concepts.&lt;p&gt;Of course, there is the question about why we would want to create a human like AI, it&amp;#x27;s my contention that human like AI isn&amp;#x27;t actually what many of us would want, but that&amp;#x27;s another topic...</text></comment>
<story><title>AI’s Language Problem</title><url>https://www.technologyreview.com/s/602094/ais-language-problem/?set=602129</url></story><parent_chain><item><author>Houshalter</author><text>I believe we can get AI from just text. Obviously that won&amp;#x27;t work for babies, because babies get bored quickly looking at text. AIs can be forced to read billions of words in mere hours!&lt;p&gt;Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words. The famous example is the vector &amp;quot;king&amp;quot; minus the vector for &amp;quot;man&amp;quot; plus &amp;quot;woman&amp;quot; equals &amp;quot;queen&amp;quot;. It&amp;#x27;s learning the meaning of words, and the relationships between them.&lt;p&gt;Recurrent NNs, using similar techniques, but with much more complexity, can learn to predict the next word in a sentence very well. They can learn to write responses that are almost indistinguishable from humans. And it&amp;#x27;s incredible this works at all, given RNNs have only a few thousand neurons at most, and a few days of training, compared to humans&amp;#x27; billions of neurons trained over a lifetime.&lt;p&gt;All of the information of our world is contained in text. Humans have produced billions of books, papers, articles, and internet comments. Billions of times more information than any human could read in their entire lifetime. Any information you can imagine is contained in text somewhere. I don&amp;#x27;t think it&amp;#x27;s necessary for AIs to be able to see, or interact with the world in any way.&lt;p&gt;If you can predict the word a human would say next, with enough accuracy, then you could also produce answers indistinguishable from theirs. Meaning you could pass the Turing test, and perform any language task they could do just as well. So language prediction alone may be sufficient for AGI.&lt;p&gt;This is the theory behind the Hutter Prize, which proposes that predicting (compressing) wikipedia&amp;#x27;s text is a measure of AI progress. The Hutter Prize isn&amp;#x27;t perfect (it&amp;#x27;s only a sample of wikipedia, which is very small compared to all the text humans have produced), but the idea is solid.</text></item><item><author>erikpukinskis</author><text>No one would ever imagine that locking a baby in a featureless room with a giant stack of books would give them general intelligence. I don&amp;#x27;t understand why AI researchers think it will work for AIs. They need bodies that are biologically connected with the rest of the biosphere, with an intrinsic biological imperative, if they are ever to understand the world. I&amp;#x27;m not saying they have to be exactly like us, but they will only be able to understand us to the extent that they have body parts and social experiences that are analogous to ours.&lt;p&gt;This isn&amp;#x27;t an engineering problem, it&amp;#x27;s a philosophical problem: We are blind to most things. We can only see what we can relate to personally. We can use language and other symbol systems to expand our understanding by permuting and recombining our personal experiences, but everything is grounded in our interactive developmental trajectory.&lt;p&gt;The kitten-in-a-cart experiment demonstrates this clearly: &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;io9.gizmodo.com&amp;#x2F;the-seriously-creepy-two-kitten-experiment-1442107174&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;io9.gizmodo.com&amp;#x2F;the-seriously-creepy-two-kitten-exper...&lt;/a&gt;  Interaction is crucial for perception. Sensation is not experience.&lt;p&gt;And here&amp;#x27;s the rub: Once you give an AI an animal-like personal developmental trajectory to use for grounding the semantics of their symbol systems, you end up with something which is not particularly different or better than a human cyborg.</text></item></parent_chain><comment><author>dmreedy</author><text>&amp;gt; All of the information of our world is contained in text&lt;p&gt;Even if this were a true statement, it&amp;#x27;s still the case that it might not be enough. There is a class of functions that are simply not learnable without some prerequisite knowledge. This is directly analogous to a one-time pad in crypto. It is entirely possible that the function &amp;#x27;language&amp;#x27; is in this class of unlearnable functions. While it may be the case that certain varieties of intelligence are learnable tabula rasa from a powerful neural net, the surface form of human natural language (the part your recommending measuring) may simply not have enough information in it to decode the whole picture. It is entirely possible that you need to supply some of your own information as well to the picture, in a specific manner so as to act as a kind of decryption key. A record needs a record player, even if you can make similar sounds with cassettes and CDs.&lt;p&gt;And so, I&amp;#x27;m willing to bet that you simply cannot, using raw, uninformed statistical techniques, predict what word a human would say next. You need to understand more of the underlying structure of humans first.&lt;p&gt;I will agree, however, that the success towards the Hutter Prize is a valuable demonstration of AI progress. Simply because I believe that maximal compression and the kind of intelligence I&amp;#x27;m talking about are one and the same thing. You need to offload as much of the semantic weight of the corpus into the encryption algorithm as you can. That means building a very complex model of natural language. And if you accept the premise that this model is not simply learnable by observing the surface form, then that means building Strong AI</text></comment>