<story><title>The Transformer Family</title><url>https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/</url></story><parent_chain></parent_chain><comment><author>hn_throwaway_99</author><text>Somewhat off topic. As someone who did some neural network programming in Matlab a couple decades ago, I always feel a bit dismayed that I&amp;#x27;m able to understand so little about modern AI given the explosion in advances in the field starting in about the late 00s or so with things like convolutional neural networks and deep learning, transformers, large language models, etc.&lt;p&gt;Can anyone recommend some great courses or other online resources for getting up to speed on the state-of-the-art with respect to AI? Not really so much looking for an &amp;quot;ELI5&amp;quot; but more of a &amp;quot;you have a strong programming and very-old-school AI background, here are the steps&amp;#x2F;processes you need to know to understand modern tools&amp;quot;.&lt;p&gt;Edit: thanks for all the great replies, super helpful!</text></comment>
<story><title>The Transformer Family</title><url>https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/</url></story><parent_chain></parent_chain><comment><author>simonw</author><text>On the one hand, this looks really useful.&lt;p&gt;On the other hand:&lt;p&gt;&amp;gt; There are various forms of attention &amp;#x2F; self-attention, Transformer (Vaswani et al., 2017) relies on the scaled dot-product attention: given a query matrix , a key matrix and a value matrix , the output is a weighted sum of the value vectors, where the weight assigned to each value slot is determined by the dot-product of the query with the corresponding key&lt;p&gt;There HAS to be a better way of communicating this stuff. I&amp;#x27;m honestly not even sure where to start decoding and explaining that paragraph.&lt;p&gt;We really need someone with the explanatory skills of &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;jvns.ca&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;jvns.ca&amp;#x2F;&lt;/a&gt; to start helping people understand this space.</text></comment>