<story><title>What are transformer models and how do they work?</title><url>https://txt.cohere.ai/what-are-transformer-models/</url></story><parent_chain><item><author>jaidhyani</author><text>Skimming it, there are a few things about this explanation that rub me just slightly the wrong way.&lt;p&gt;1. Calling the input token sequence a &amp;quot;command&amp;quot;. It probably only makes sense to think of this as a &amp;quot;command&amp;quot; on a model that&amp;#x27;s been fine-tuned to treat it as such.&lt;p&gt;2. Skipping over BPE as part of tokenization - but almost every transformer explainer does this, I guess.&lt;p&gt;3. Describing transformers as using a &amp;quot;word embedding&amp;quot;. I&amp;#x27;m actually not aware of any transformers that use actual word embeddings, except the ones that incidentally fall out of other tokenization approaches sometimes.&lt;p&gt;4. Describing positional embeddings as multiplicative. They are generally (and very counterintuitively to me, but nevertheless) additive with token embeddings.&lt;p&gt;5. &amp;quot;what attention does is it moves the words in a sentence (or piece of text) closer in the word embedding&amp;quot; No, that&amp;#x27;s just incorrect.&lt;p&gt;6. You don&amp;#x27;t actually need a softmax layer at the end, since here they&amp;#x27;re just picking the top token and they can just do that pre-softmax since it won&amp;#x27;t change. It&amp;#x27;s also weird how they talked about this here when the most prominent use of softmax in transformers is actually in the attention component.&lt;p&gt;7. Really shortchanges the feedforward component. It may be simple, but it&amp;#x27;s really important to making the whole thing work.&lt;p&gt;8. Nothing about the residual</text></item></parent_chain><comment><author>eiz</author><text>&amp;gt; 4. Describing positional embeddings as multiplicative. They are generally (and very counterintuitively to me, but nevertheless) additive with token embeddings.&lt;p&gt;Worth noting that rotary position embeddings, used in many recent architectures (LLaMA, GPT-NeoX, ...), are very similar to the original sin&amp;#x2F;cos position embedding in the transformer paper but using complex multiplication instead of addition</text></comment>
<story><title>What are transformer models and how do they work?</title><url>https://txt.cohere.ai/what-are-transformer-models/</url></story><parent_chain><item><author>jaidhyani</author><text>Skimming it, there are a few things about this explanation that rub me just slightly the wrong way.&lt;p&gt;1. Calling the input token sequence a &amp;quot;command&amp;quot;. It probably only makes sense to think of this as a &amp;quot;command&amp;quot; on a model that&amp;#x27;s been fine-tuned to treat it as such.&lt;p&gt;2. Skipping over BPE as part of tokenization - but almost every transformer explainer does this, I guess.&lt;p&gt;3. Describing transformers as using a &amp;quot;word embedding&amp;quot;. I&amp;#x27;m actually not aware of any transformers that use actual word embeddings, except the ones that incidentally fall out of other tokenization approaches sometimes.&lt;p&gt;4. Describing positional embeddings as multiplicative. They are generally (and very counterintuitively to me, but nevertheless) additive with token embeddings.&lt;p&gt;5. &amp;quot;what attention does is it moves the words in a sentence (or piece of text) closer in the word embedding&amp;quot; No, that&amp;#x27;s just incorrect.&lt;p&gt;6. You don&amp;#x27;t actually need a softmax layer at the end, since here they&amp;#x27;re just picking the top token and they can just do that pre-softmax since it won&amp;#x27;t change. It&amp;#x27;s also weird how they talked about this here when the most prominent use of softmax in transformers is actually in the attention component.&lt;p&gt;7. Really shortchanges the feedforward component. It may be simple, but it&amp;#x27;s really important to making the whole thing work.&lt;p&gt;8. Nothing about the residual</text></item></parent_chain><comment><author>VMG</author><text>I have to agree. The article summary says&lt;p&gt;&amp;gt; Transformer block: Guesses the next word. It is formed by an attention block and a feedforward block.&lt;p&gt;But the diagram shows transformer blocks chained in sequence. So the next transformer block in the sequence would only receive a single word as the input? Does not make sense.</text></comment>