<story><title>Falcon LLM – A 40B Model</title><url>https://falconllm.tii.ae/</url></story><parent_chain><item><author>elahieh</author><text>Is there a guide out there for dummies on how to try a ChatGPT like instance of this on a VM cheaply? eg pay $1 or $2 an hour for a point and click experience with the instruct version of this. A docker image perhaps.&lt;p&gt;Reading posts on r&amp;#x2F;LocalLLAMA is people’s trial and error experiences, quite random.</text></item></parent_chain><comment><author>lhl</author><text>For Falcon specifically, this is easy, it&amp;#x27;s embedded here: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;huggingface.co&amp;#x2F;blog&amp;#x2F;falcon#demo&quot; rel=&quot;nofollow noreferrer&quot;&gt;https:&amp;#x2F;&amp;#x2F;huggingface.co&amp;#x2F;blog&amp;#x2F;falcon#demo&lt;/a&gt; or you can access the demo here: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;huggingface.co&amp;#x2F;spaces&amp;#x2F;HuggingFaceH4&amp;#x2F;falcon-chat&quot; rel=&quot;nofollow noreferrer&quot;&gt;https:&amp;#x2F;&amp;#x2F;huggingface.co&amp;#x2F;spaces&amp;#x2F;HuggingFaceH4&amp;#x2F;falcon-chat&lt;/a&gt;&lt;p&gt;I just tested both and it&amp;#x27;s pretty zippy (faster than AMD&amp;#x27;s recent live MI300 demo).&lt;p&gt;For llama-based models, recently I&amp;#x27;ve been using &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;turboderp&amp;#x2F;exllama&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;turboderp&amp;#x2F;exllama&lt;/a&gt; a lot. It has a Dockerfile&amp;#x2F;docker-compose.yml so it should be pretty easy to get going. llama.cpp is the other easy one and the most recent updates put it&amp;#x27;s CUDA support only about 25% slower and generally is a simple `make` with a flag depending on which GPU you support you want and has basically no dependencies.&lt;p&gt;Also, here&amp;#x27;s a Colab notebook that should let shows you run up to 13b quantized models (12G RAM, 80G disk, Tesla T4 16G) for free: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;colab.research.google.com&amp;#x2F;drive&amp;#x2F;1QzFsWru1YLnTVK77itWEASCPnIH7IDPo&quot; rel=&quot;nofollow noreferrer&quot;&gt;https:&amp;#x2F;&amp;#x2F;colab.research.google.com&amp;#x2F;drive&amp;#x2F;1QzFsWru1YLnTVK77itW...&lt;/a&gt; (for Falcon, replace w&amp;#x2F; Koboldcpp or ctransformers)</text></comment>
<story><title>Falcon LLM – A 40B Model</title><url>https://falconllm.tii.ae/</url></story><parent_chain><item><author>elahieh</author><text>Is there a guide out there for dummies on how to try a ChatGPT like instance of this on a VM cheaply? eg pay $1 or $2 an hour for a point and click experience with the instruct version of this. A docker image perhaps.&lt;p&gt;Reading posts on r&amp;#x2F;LocalLLAMA is people’s trial and error experiences, quite random.</text></item></parent_chain><comment><author>joshka</author><text>Take a look at youtube vids for this. Mainly because you&amp;#x27;re going to see people show all the steps when presenting instead of skipping them when talking about what they did. E.g. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=KenORQDCXV0&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=KenORQDCXV0&lt;/a&gt;</text></comment>