<story><title>Mixtral 8x22B</title><url>https://mistral.ai/news/mixtral-8x22b/</url></story><parent_chain><item><author>jjice</author><text>Does anyone have a good layman&amp;#x27;s explanation of the &amp;quot;Mixture-of-Experts&amp;quot; concept? I think I understand the idea of having &amp;quot;sub-experts&amp;quot;, but how do you decide what each specialization is during training? Or is that not how it works at all?</text></item></parent_chain><comment><author>huevosabio</author><text>Ignore the &amp;quot;experts&amp;quot; part, it misleads a lot of people [0]. There is no explicit specialization in the most popular setups, it is achieved implicitly through training. In short: MoEs add multiple MLP sublayers and a routing mechanism after each attention sublayer and let the training procedure learn the MLP parameters and the routing parameters.&lt;p&gt;In a longer, but still rough, form...&lt;p&gt;How these transformers work is roughly:&lt;p&gt;```
x_{l+1} = mlp_l(attention_l(x_l))
```&lt;p&gt;where `x_l` is the hidden representation at layer l, `attention_l` is the attention sublayer at layer l, and `mlp_l` is the multilayer perceptron at sublayer l.&lt;p&gt;This MLP layer is very expensive because it is fully connected (i.e. every input has a weight to every output). So! MoEs instead of creating an even bigger, more expensive MLP to get more capability, they create K MLP sublayers (the &amp;quot;experts&amp;quot;) and a router that decides which MLP sublayers to use. This router spits out an importance score for each MLP &amp;quot;expert&amp;quot; and then you choose the top T MLPs and do an average weighed on importance, so roughly:&lt;p&gt;```
x_{l+1} = \sum_e mlp_{l,e}(attention_l(x_l)) * importance_score_{l, e}
```&lt;p&gt;where the `importance_score_{l, e}` is the score computed by the router at layer l for &amp;quot;expert&amp;quot; e. That is, `importance_score_{l} = attention_l(x_l)`. Note that here we are adding all experts, but in reality we choose the top T, often 2, and use that.&lt;p&gt;[0] some architectures do, in fact, combine domain experts to make a greater whole, but not the currently popular flavor</text></comment>
<story><title>Mixtral 8x22B</title><url>https://mistral.ai/news/mixtral-8x22b/</url></story><parent_chain><item><author>jjice</author><text>Does anyone have a good layman&amp;#x27;s explanation of the &amp;quot;Mixture-of-Experts&amp;quot; concept? I think I understand the idea of having &amp;quot;sub-experts&amp;quot;, but how do you decide what each specialization is during training? Or is that not how it works at all?</text></item></parent_chain><comment><author>hlfshell</author><text>This is a bit of a misnomer. Each expert is a sub network that specializes in sub understanding we can&amp;#x27;t possibly track.&lt;p&gt;During training a routing network is punished if it does not evenly distribute training tokens to the correct experts. This prevents any one or two networks from becoming the primary networks.&lt;p&gt;The result of this is that each token has essentially even probability of being routed to one of the sub models, with the underlying logic of why that model is an expert for that token being beyond our understanding or description.</text></comment>