<story><title>Better Language Models and Their Implications</title><url>https://blog.openai.com/better-language-models/</url></story><parent_chain><item><author>cs702</author><text>This kind of &amp;quot;blocking-and-tackling&amp;quot; work is important.&lt;p&gt;The authors take a well-known architecture, the Transformer[a], configure it with a progressively larger number of parameter, train it to predict the next word conditioned on previous text, using a large dataset consisting of 40GB of text scraped from the Web, and test each trained model on a range of zero-shot transfer-learning tasks.&lt;p&gt;Remarkably, the performance of a Transformer in the tested tasks improves &lt;i&gt;log-linearly&lt;/i&gt; with the number of parameters, suggesting that even the largest model tested, with 1.5B parameters, still &lt;i&gt;underfits&lt;/i&gt; 40GB of text.&lt;p&gt;This is &lt;i&gt;compelling evidence&lt;/i&gt; that we do NOT need new architectures, NOR new kinds of training objectives, NOR new theories, for better language modeling!  We can get better language modeling simply by increasing model capacity (i.e., by adding more parameters to existing models), which becomes easier and simpler to do as hardware continues to improve over time.&lt;p&gt;Great work.&lt;p&gt;PS. In case it&amp;#x27;s not clear: I&amp;#x27;m not saying we should suddenly stop searching for new, better ideas and architectures. That would be silly. Please don&amp;#x27;t attack a straw-man :-)&lt;p&gt;[a] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;1706.03762&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;1706.03762&lt;/a&gt;</text></item></parent_chain><comment><author>ninjin</author><text>&amp;gt; …we do NOT need new architectures…&lt;p&gt;You do realise the irony in stating this regarding transformers that arguably made their first appearance as decomposable attention in 2016 [1] and then as transformers in 2017 [2]? It is not as if this is a vanilla RNN straight out of the 90s sweeping the floor with decades of model innovations, rather it looks like we are seeing the rise of a new, simple model category that works remarkably well – akin to how Mikolov et al. reconsidered how to learn vector representations back in 2013 [3] to ingest magnitudes more data than previously possible.&lt;p&gt;[1]: &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;www.aclweb.org&amp;#x2F;anthology&amp;#x2F;D&amp;#x2F;D16&amp;#x2F;D16-1244.pdf&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;www.aclweb.org&amp;#x2F;anthology&amp;#x2F;D&amp;#x2F;D16&amp;#x2F;D16-1244.pdf&lt;/a&gt;&lt;p&gt;[2]: &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;papers.nips.cc&amp;#x2F;paper&amp;#x2F;7181-attention-is-all-you-need&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;papers.nips.cc&amp;#x2F;paper&amp;#x2F;7181-attention-is-all-you-need&lt;/a&gt;&lt;p&gt;[3]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;1301.3781&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;1301.3781&lt;/a&gt;&lt;p&gt;I would not call myself a model-focused researcher so I would love to play down their importance, but to be intellectually honest one should call out hyperbole where ever one sees it.</text></comment>
<story><title>Better Language Models and Their Implications</title><url>https://blog.openai.com/better-language-models/</url></story><parent_chain><item><author>cs702</author><text>This kind of &amp;quot;blocking-and-tackling&amp;quot; work is important.&lt;p&gt;The authors take a well-known architecture, the Transformer[a], configure it with a progressively larger number of parameter, train it to predict the next word conditioned on previous text, using a large dataset consisting of 40GB of text scraped from the Web, and test each trained model on a range of zero-shot transfer-learning tasks.&lt;p&gt;Remarkably, the performance of a Transformer in the tested tasks improves &lt;i&gt;log-linearly&lt;/i&gt; with the number of parameters, suggesting that even the largest model tested, with 1.5B parameters, still &lt;i&gt;underfits&lt;/i&gt; 40GB of text.&lt;p&gt;This is &lt;i&gt;compelling evidence&lt;/i&gt; that we do NOT need new architectures, NOR new kinds of training objectives, NOR new theories, for better language modeling!  We can get better language modeling simply by increasing model capacity (i.e., by adding more parameters to existing models), which becomes easier and simpler to do as hardware continues to improve over time.&lt;p&gt;Great work.&lt;p&gt;PS. In case it&amp;#x27;s not clear: I&amp;#x27;m not saying we should suddenly stop searching for new, better ideas and architectures. That would be silly. Please don&amp;#x27;t attack a straw-man :-)&lt;p&gt;[a] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;1706.03762&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;1706.03762&lt;/a&gt;</text></item></parent_chain><comment><author>rosstaylor90</author><text>I think your conclusion is too strong. Yes, we know bigger models and data generally lead to better performance (e.g. BigGAN results last year) but progress in architecture can still speed up progress in ML tasks. If we were still stuck using RNNs and LSTMs for language modeling we wouldn&amp;#x27;t be talking about this news today.</text></comment>