<story><title>Mistral 7B</title><url>https://arxiv.org/abs/2310.06825</url></story><parent_chain></parent_chain><comment><author>anon1253</author><text>It works really really well for chatbots and roleplay applications (at least for me). The fine-tune on the instruct version is rather meh however, and I recommend &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;huggingface.co&amp;#x2F;Open-Orca&amp;#x2F;Mistral-7B-OpenOrca&amp;#x2F;&quot; rel=&quot;nofollow noreferrer&quot;&gt;https:&amp;#x2F;&amp;#x2F;huggingface.co&amp;#x2F;Open-Orca&amp;#x2F;Mistral-7B-OpenOrca&amp;#x2F;&lt;/a&gt; if you plan on using it out-of-the-box. Take note of the prompt template, you&amp;#x27;ll get really undesired results otherwise (basically just garbage). I&amp;#x27;ve been running it on my pet projects with llama.cpp and the inference is blazing fast even with my mediocre 2080 Super</text></comment>
<story><title>Mistral 7B</title><url>https://arxiv.org/abs/2310.06825</url></story><parent_chain></parent_chain><comment><author>joennlae</author><text>Llama1 --&amp;gt; 1.0T
Llama2 --&amp;gt; 2.0T
Mistral --&amp;gt; ??&lt;p&gt;They do not publish how many tokens it is pre-trained on, additionally to sharing no info on datasets used (except for fine-tuning).&lt;p&gt;To my knowledge, no one has trained a larger LLM (&amp;gt;250M) to the capacity limit.
As discussed in the original GPT3 paper (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;twitter.com&amp;#x2F;gneubig&amp;#x2F;status&amp;#x2F;1286731711150280705?s=20&quot; rel=&quot;nofollow noreferrer&quot;&gt;https:&amp;#x2F;&amp;#x2F;twitter.com&amp;#x2F;gneubig&amp;#x2F;status&amp;#x2F;1286731711150280705?s=20&lt;/a&gt;)&lt;p&gt;TinyLlama is trying to do that for 1.1B: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;jzhang38&amp;#x2F;TinyLlama&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;jzhang38&amp;#x2F;TinyLlama&lt;/a&gt;&lt;p&gt;As long as we are not at the capacity limit, we will have a few of these 7B beats 13B (or 7B beats 70B) moments.</text></comment>