<story><title>Qwen1.5-110B</title><url>https://qwenlm.github.io/blog/qwen1.5-110b/</url></story><parent_chain></parent_chain><comment><author>coder543</author><text>Firstly, I&amp;#x27;ll say that it&amp;#x27;s always exciting to see more weight-available models.&lt;p&gt;However, I don&amp;#x27;t particularly like that benchmark table. I saw the HumanEval score for Llama 3 70B and immediately said &amp;quot;nope, that&amp;#x27;s not right&amp;quot;. It claims Llama 3 70B scored only 45.7. Llama 3 70B Instruct[0] scored 81.7, not even in the same ballpark.&lt;p&gt;It turns out that the Qwen team didn&amp;#x27;t benchmark the chat&amp;#x2F;instruct versions of the model on virtually any of the benchmarks. Why did they only do those benchmarks for the base models?&lt;p&gt;It makes it very hard to draw any useful conclusions from this release, since most people would be using the chat-tuned model for the things those base model benchmarks are measuring.&lt;p&gt;My previous experience with Qwen releases is that the models also have a habit of randomly switching to Chinese for a few words. I wonder if this model is better at responding to English questions with an English response? Maybe we need a benchmark for how well an LLM sticks to responding in the same language as the question, across a range of different languages.&lt;p&gt;[0]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;scontent-atl3-1.xx.fbcdn.net&amp;#x2F;v&amp;#x2F;t39.2365-6&amp;#x2F;438037375_405784438908376_6082258861354187544_n.png?_nc_cat=106&amp;amp;ccb=1-7&amp;amp;_nc_sid=e280be&amp;amp;_nc_ohc=7AIeH58EojUAb7BXK2c&amp;amp;_nc_ht=scontent-atl3-1.xx&amp;amp;oh=00_AfAs3tOUPoHfB2vkPKCZRfAhjDP0aOZH7SrYqgqVFGJWTg&amp;amp;oe=6645F5CA&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;scontent-atl3-1.xx.fbcdn.net&amp;#x2F;v&amp;#x2F;t39.2365-6&amp;#x2F;438037375_...&lt;/a&gt;</text></comment>
<story><title>Qwen1.5-110B</title><url>https://qwenlm.github.io/blog/qwen1.5-110b/</url></story><parent_chain></parent_chain><comment><author>manmal</author><text>Maybe this is the right thread to ask. If you were in the market for a new Mac (eg MacBook Pro), would you go for the 100+GB RAM option for running LLMs locally? Or is the difference between heavily quantized models and their unquantized versions so small, and progress so fast, that it wouldn’t be worth it?</text></comment>