<story><title>Reformer, the Efficient Transformer</title><url>https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html</url></story><parent_chain></parent_chain><comment><author>gwern</author><text>Discussion &amp;amp; links to various implementations: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.reddit.com&amp;#x2F;r&amp;#x2F;MachineLearning&amp;#x2F;comments&amp;#x2F;eg1wr3&amp;#x2F;reformer_the_efficient_transformer_anonymous_et&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.reddit.com&amp;#x2F;r&amp;#x2F;MachineLearning&amp;#x2F;comments&amp;#x2F;eg1wr3&amp;#x2F;ref...&lt;/a&gt;</text></comment>
<story><title>Reformer, the Efficient Transformer</title><url>https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html</url></story><parent_chain></parent_chain><comment><author>lapink</author><text>There is no argument for why the LSH would work well, especially at the beginning of training. As the weights are initially random, bucket assignment  would be random as well. If predicting at position A requires info from position B,  but they are not in the same bucket, there will be no gradient to get the query embedding of A closer to the key embedding of B. The reversible layer trick is neat though.</text></comment>