<story><title>σ-GPTs: A new approach to autoregressive models</title><url>https://arxiv.org/abs/2404.09562</url></story><parent_chain><item><author>cs702</author><text>This looks &lt;i&gt;great&lt;/i&gt;.&lt;p&gt;The authors randomly permute (i.e., shuffle) input tokens in training and add two positional encodings to each token: one with the token&amp;#x27;s position and another with the position of the token to be predicted. Otherwise, the model is a standard autoregressive GPT. The consequences of this seemingly &amp;quot;simple&amp;quot; modification are significant:&lt;p&gt;* The authors can prompt the trained model with part of a sequence and then decode the missing tokens, all at once, in parallel, regardless of order -- i.e., the model can in-fill in parallel.&lt;p&gt;* The authors can compute conditional probability densities for every missing token in a sequence, again in parallel, i.e., densities for all missing tokens at once.&lt;p&gt;* The authors propose a rejection-sampling method for generating in-fill tokens, again in parallel. Their method seems to work well in practice.&lt;p&gt;I&amp;#x27;ve added this to my reading list. Thank you for sharing it on HN.</text></item></parent_chain><comment><author>toxik</author><text>This problem formulation has been around for a while, it’s kind of the holy grail of modeling. What is new compared to PixelCNN and related is this position embedding idea.</text></comment>
<story><title>σ-GPTs: A new approach to autoregressive models</title><url>https://arxiv.org/abs/2404.09562</url></story><parent_chain><item><author>cs702</author><text>This looks &lt;i&gt;great&lt;/i&gt;.&lt;p&gt;The authors randomly permute (i.e., shuffle) input tokens in training and add two positional encodings to each token: one with the token&amp;#x27;s position and another with the position of the token to be predicted. Otherwise, the model is a standard autoregressive GPT. The consequences of this seemingly &amp;quot;simple&amp;quot; modification are significant:&lt;p&gt;* The authors can prompt the trained model with part of a sequence and then decode the missing tokens, all at once, in parallel, regardless of order -- i.e., the model can in-fill in parallel.&lt;p&gt;* The authors can compute conditional probability densities for every missing token in a sequence, again in parallel, i.e., densities for all missing tokens at once.&lt;p&gt;* The authors propose a rejection-sampling method for generating in-fill tokens, again in parallel. Their method seems to work well in practice.&lt;p&gt;I&amp;#x27;ve added this to my reading list. Thank you for sharing it on HN.</text></item></parent_chain><comment><author>taneq</author><text>Wow, if that works that&amp;#x27;s wild (and also has that &amp;quot;damn, now you say it it&amp;#x27;s obvious&amp;quot; flavour that so many really cool discoveries share...)</text></comment>