<story><title>Mixtral 8x7B: A sparse Mixture of Experts language model</title><url>https://arxiv.org/abs/2401.04088</url></story><parent_chain><item><author>cuuupid</author><text>I’d like to note that this model’s parameter usage is low enough (13b) to run smoothly at high quality on a 3090 while beating GPT-3.5 on humaneval and sporting 32k context.&lt;p&gt;3090s are consumer grade and common on gaming rigs. I’m hoping game devs start experimenting with locally deployed Mixtral in their games. e.g. something like CIV but with each leader powered via LLM</text></item></parent_chain><comment><author>snickell</author><text>You can also run Mixtral, at a decent token rate, on a post-2020 Apple Macbook Pro M1&amp;#x2F;M2&amp;#x2F;M3 with 32GB+ of RAM. 16GB RAM also works, sort of ok, which I suspect is the same quantization a 3090 is using, but I do notice a difference in the quantization. On my M2 Pro, the token rate and intelligence feels like GPT-3.5turbo. This is the first model I&amp;#x27;ve started actually using (vs playing around with for the love of the tech) instead of GPT-3.5.&lt;p&gt;An Apple M2 Pro with 32GB of RAM is in the same price range as a gaming PC with a 3090, but its another example of normal people with moderately high performance systems &amp;quot;accidentally&amp;quot; being able to run a GPT-3.5 comparable model.&lt;p&gt;If you have an Apple meeting these specs and want to play around, LLM Studio is open source and has made it really easy to get started: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;lmstudio.ai&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;lmstudio.ai&amp;#x2F;&lt;/a&gt;&lt;p&gt;I hope to see a LOT more hobby hacking as a result of Mixtral and successors.</text></comment>
<story><title>Mixtral 8x7B: A sparse Mixture of Experts language model</title><url>https://arxiv.org/abs/2401.04088</url></story><parent_chain><item><author>cuuupid</author><text>I’d like to note that this model’s parameter usage is low enough (13b) to run smoothly at high quality on a 3090 while beating GPT-3.5 on humaneval and sporting 32k context.&lt;p&gt;3090s are consumer grade and common on gaming rigs. I’m hoping game devs start experimenting with locally deployed Mixtral in their games. e.g. something like CIV but with each leader powered via LLM</text></item></parent_chain><comment><author>LeoPanthera</author><text>Google tells me that the RTX 3090 is priced between US$1,480 and $1,680.&lt;p&gt;You can buy a whole PC for that, I refuse to believe that a GPU priced that highly is &amp;quot;consumer grade&amp;quot; and &amp;quot;common&amp;quot;.&lt;p&gt;Are there any GPUs that are good for LLMs or other genAI that aren&amp;#x27;t absurdly priced? Or ones specifically designed for AI rather than gaming graphics?</text></comment>