<story><title>Mixtral 8x7B: A sparse Mixture of Experts language model</title><url>https://arxiv.org/abs/2401.04088</url></story><parent_chain><item><author>coder543</author><text>Mixtral works great at 3-bit quantization. It fits onto a single RTX 3090 and runs at about 50 tokens&amp;#x2F;s. The output quality is not &amp;quot;ruined&amp;quot; at all.&lt;p&gt;For the amount of money you&amp;#x27;re talking about, you could also buy two 3090s (~$750 each on eBay) and have 48GB of VRAM to run with less quantization at full speed.&lt;p&gt;M-series Macs are surprisingly flexible platforms, but they&amp;#x27;re not &amp;quot;the only&amp;quot; consumer platform that can do Mixtral.</text></item><item><author>Casteil</author><text>I&amp;#x27;ve had excellent results with Mixtral too - it&amp;#x27;s genuinely impressive. Only problem is that it&amp;#x27;s a relatively big model that&amp;#x27;s difficult to run with full GPU inference on consumer hardware (vs the 7b&amp;#x2F;13b models people typically use).&lt;p&gt;So far, the main consumer platform capable of running it without &amp;#x27;ruining&amp;#x27; the quality of its output (with high levels of quantization) is the newer Apple Silicon Macs with unified memory - generally &amp;gt;=48GB. It can apparently be done on 32 or 36GB, but there&amp;#x27;s not much headroom.&lt;p&gt;Edit: As coder543 points out, yes - you can run it without more lossy levels of quantization on multi-GPU setups providing those have enough combined vram.</text></item><item><author>vessenes</author><text>This paper details the model that&amp;#x27;s been in the wild for approximately a month now. Mixtral 8x7B is very, very good. It&amp;#x27;s roughly sized at 13B, and ranked much, much higher than competitively sized models by, e.g. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.reddit.com&amp;#x2F;r&amp;#x2F;LocalLLaMA&amp;#x2F;comments&amp;#x2F;1916896&amp;#x2F;llm_comparisontest_confirm_leaderboard_big_news&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.reddit.com&amp;#x2F;r&amp;#x2F;LocalLLaMA&amp;#x2F;comments&amp;#x2F;1916896&amp;#x2F;llm_com...&lt;/a&gt;. Ravenwolf notes that the model slightly outperforms some of its benchmark testing, and this is my experience. It&amp;#x27;s surprisingly good for a model of its size, and a very capable daily driver on a Mac for chat, code input and other uses.&lt;p&gt;Something that has come to light since the release of the weights, and not mentioned in this paper is that it looks like fairly likely that the 8 experts were all seeded by Mistral 7B and subsequently diverged. This has generated a lot of experimentation in the local LLM community with cloning models as a way to cheaply generate experts.&lt;p&gt;It was generally thought likely that training an 8x7B network would be as much work as training 8 7B networks, but this seems not to have been true for Mistral, which is super interesting.&lt;p&gt;There&amp;#x27;s still a lot of rapid innovation happening in this space, with papers like Calm from DeepMind this week, and a lot of the adhoc experimental layer combining happening in the wild, (see, e.g. Goliath-120b), I think we&amp;#x27;re likely to see some pretty interesting architectural improvements this year in the LLM space.&lt;p&gt;Calm seems to point the way to a next step after MoE, and models like Goliath seem to indicate that even a really really lazy version of Calm (no Linear layer combination, just literally alternating layers at full weights) can be very impactful. Overall I think we will see really, really strong models that are performant on consumer hardware in 2024, likely first half of this year.</text></item></parent_chain><comment><author>bekantan</author><text>&amp;gt; The output quality is not &amp;quot;ruined&amp;quot; at all.&lt;p&gt;That was my experience as well - 3-bit version is pretty good.&lt;p&gt;I also tried 2-bit version, which was disappointing.&lt;p&gt;However, there is a new 2-bit approach in the works[1] (merged yesterday) which performs surprisingly well for Mixtral 8x7B Instruct with 2.10 bits per weight (12.3 GB model size).&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;ggerganov&amp;#x2F;llama.cpp&amp;#x2F;pull&amp;#x2F;4773&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;ggerganov&amp;#x2F;llama.cpp&amp;#x2F;pull&amp;#x2F;4773&lt;/a&gt;</text></comment>
<story><title>Mixtral 8x7B: A sparse Mixture of Experts language model</title><url>https://arxiv.org/abs/2401.04088</url></story><parent_chain><item><author>coder543</author><text>Mixtral works great at 3-bit quantization. It fits onto a single RTX 3090 and runs at about 50 tokens&amp;#x2F;s. The output quality is not &amp;quot;ruined&amp;quot; at all.&lt;p&gt;For the amount of money you&amp;#x27;re talking about, you could also buy two 3090s (~$750 each on eBay) and have 48GB of VRAM to run with less quantization at full speed.&lt;p&gt;M-series Macs are surprisingly flexible platforms, but they&amp;#x27;re not &amp;quot;the only&amp;quot; consumer platform that can do Mixtral.</text></item><item><author>Casteil</author><text>I&amp;#x27;ve had excellent results with Mixtral too - it&amp;#x27;s genuinely impressive. Only problem is that it&amp;#x27;s a relatively big model that&amp;#x27;s difficult to run with full GPU inference on consumer hardware (vs the 7b&amp;#x2F;13b models people typically use).&lt;p&gt;So far, the main consumer platform capable of running it without &amp;#x27;ruining&amp;#x27; the quality of its output (with high levels of quantization) is the newer Apple Silicon Macs with unified memory - generally &amp;gt;=48GB. It can apparently be done on 32 or 36GB, but there&amp;#x27;s not much headroom.&lt;p&gt;Edit: As coder543 points out, yes - you can run it without more lossy levels of quantization on multi-GPU setups providing those have enough combined vram.</text></item><item><author>vessenes</author><text>This paper details the model that&amp;#x27;s been in the wild for approximately a month now. Mixtral 8x7B is very, very good. It&amp;#x27;s roughly sized at 13B, and ranked much, much higher than competitively sized models by, e.g. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.reddit.com&amp;#x2F;r&amp;#x2F;LocalLLaMA&amp;#x2F;comments&amp;#x2F;1916896&amp;#x2F;llm_comparisontest_confirm_leaderboard_big_news&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.reddit.com&amp;#x2F;r&amp;#x2F;LocalLLaMA&amp;#x2F;comments&amp;#x2F;1916896&amp;#x2F;llm_com...&lt;/a&gt;. Ravenwolf notes that the model slightly outperforms some of its benchmark testing, and this is my experience. It&amp;#x27;s surprisingly good for a model of its size, and a very capable daily driver on a Mac for chat, code input and other uses.&lt;p&gt;Something that has come to light since the release of the weights, and not mentioned in this paper is that it looks like fairly likely that the 8 experts were all seeded by Mistral 7B and subsequently diverged. This has generated a lot of experimentation in the local LLM community with cloning models as a way to cheaply generate experts.&lt;p&gt;It was generally thought likely that training an 8x7B network would be as much work as training 8 7B networks, but this seems not to have been true for Mistral, which is super interesting.&lt;p&gt;There&amp;#x27;s still a lot of rapid innovation happening in this space, with papers like Calm from DeepMind this week, and a lot of the adhoc experimental layer combining happening in the wild, (see, e.g. Goliath-120b), I think we&amp;#x27;re likely to see some pretty interesting architectural improvements this year in the LLM space.&lt;p&gt;Calm seems to point the way to a next step after MoE, and models like Goliath seem to indicate that even a really really lazy version of Calm (no Linear layer combination, just literally alternating layers at full weights) can be very impactful. Overall I think we will see really, really strong models that are performant on consumer hardware in 2024, likely first half of this year.</text></item></parent_chain><comment><author>Casteil</author><text>Fair enough. I did put &amp;#x27;ruining&amp;#x27; in quotes for a reason - I haven&amp;#x27;t compared output between Q3 and Q4_K_M that I use, but you do generally sacrifice output quality at higher quantization levels.&lt;p&gt;And you&amp;#x27;re right, you can run it on a multi-GPU setup if you&amp;#x27;re so inclined.</text></comment>