<story><title>Rethinking Attention with Performers</title><url>https://ai.googleblog.com/2020/10/rethinking-attention-with-performers.html</url></story><parent_chain><item><author>desmap</author><text>OT: I want to get into transformers for NLP, what&amp;#x27;s the best way?&lt;p&gt;About me: Mostly done TS the last years. Dipped into Python, a bit pandas, a bit numpy, a bit Kaggle for the last 3-4 weeks.&lt;p&gt;Why I ask: It&amp;#x27;s so easy to get lost, this field is wide, e.g. I spent days with spaCy, CoreNLP, etc. before I learned that transformers-based stuff exist and outperform former.</text></item></parent_chain><comment><author>FL33TW00D</author><text>I&amp;#x27;ve just recently been on a journey to understand transformers in and out, here are the resources I&amp;#x27;ve found managed to drill it into my head:&lt;p&gt;1. Chapters 7, 9, 10&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;web.stanford.edu&amp;#x2F;~jurafsky&amp;#x2F;slp3&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;web.stanford.edu&amp;#x2F;~jurafsky&amp;#x2F;slp3&amp;#x2F;&lt;/a&gt;&lt;p&gt;This was really useful to really build up to the concepts of attention (although the actual attention section is still brief).&lt;p&gt;2. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;jalammar.github.io&amp;#x2F;visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;jalammar.github.io&amp;#x2F;visualizing-neural-machine-transl...&lt;/a&gt;&lt;p&gt;This was great for visualization and understanding that attention wasn&amp;#x27;t exclusively for transformers and actually was for RNNs first.&lt;p&gt;3. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;jalammar.github.io&amp;#x2F;illustrated-transformer&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;jalammar.github.io&amp;#x2F;illustrated-transformer&amp;#x2F;&lt;/a&gt;&lt;p&gt;Getting to understand how transformers actually work visually.&lt;p&gt;4. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=S27pHKBEp30&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=S27pHKBEp30&lt;/a&gt;&lt;p&gt;This lecture by Leo Dirac was extremely helpful to finish off with, not only because it actually includes some pseudocode but it also revisits some key topics and covers why transformers are needed.&lt;p&gt;One of the big confusion points for me was that the concept of ATTENTION and SELF-ATTENTION are not the same thing.&lt;p&gt;Hope this helps.</text></comment>
<story><title>Rethinking Attention with Performers</title><url>https://ai.googleblog.com/2020/10/rethinking-attention-with-performers.html</url></story><parent_chain><item><author>desmap</author><text>OT: I want to get into transformers for NLP, what&amp;#x27;s the best way?&lt;p&gt;About me: Mostly done TS the last years. Dipped into Python, a bit pandas, a bit numpy, a bit Kaggle for the last 3-4 weeks.&lt;p&gt;Why I ask: It&amp;#x27;s so easy to get lost, this field is wide, e.g. I spent days with spaCy, CoreNLP, etc. before I learned that transformers-based stuff exist and outperform former.</text></item></parent_chain><comment><author>kanche</author><text>Everyone has a different way of approaching. I like video walk-throughs of papers. If you are the same, for the theory bit you may find Yannic Kilcher&amp;#x27;s videos [0,1] helpful.&lt;p&gt;On implementation side, the best project out there is [2] with good documentation. Also spacy has transformer package [3] (I personally haven&amp;#x27;t tried it), so maybe it will be easier for you to jump in if you have prior experience with spacy.&lt;p&gt;[0] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=iDulhoQ2pro&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=iDulhoQ2pro&lt;/a&gt;&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=-9evrZnBorM&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=-9evrZnBorM&lt;/a&gt;&lt;p&gt;[2] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;huggingface.co&amp;#x2F;transformers&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;huggingface.co&amp;#x2F;transformers&amp;#x2F;&lt;/a&gt;&lt;p&gt;[3] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;explosion.ai&amp;#x2F;blog&amp;#x2F;spacy-transformers&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;explosion.ai&amp;#x2F;blog&amp;#x2F;spacy-transformers&lt;/a&gt;</text></comment>