<story><title>Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa</title><url>https://github.com/nomic-ai/gpt4all</url></story><parent_chain><item><author>alchemist1e9</author><text>I’ll ask a dumb question. On another of the numerous LLM related posts I was asking if any of the self host-able open model can do code summaries at close to the quality of GPT 3.5 turbo. I was basically told nowhere close yet.&lt;p&gt;Can this potentially do that?&lt;p&gt;Ideally I’d like to have it generate descriptions of large amounts of code but would rather not burn tokens and lose privacy via OpenAI api. But I’d gladly keep a high end GPU burning on such task , even if that was actually slightly more expensive.&lt;p&gt;Edit: To clarify I do this partially now on batches of code via openAI api currently. It’s around 1-3 cents for a typical 400-800 line source code file. And I don’t mean feeding the full code base in as a single input.</text></item><item><author>2bitencryption</author><text>Having spent quite a bit of time playing around with llama.cpp, alpaca.cpp, loras, and the many other llama-based weights lately, here is my impression:&lt;p&gt;The biggest deal with this isn&amp;#x27;t the published lora adapter (which seems limited to llama 7b), but the cleaned training data, which is likely better than the previous data sets used to train the alpaca-inspired loras that have been publicly released so far. [0]&lt;p&gt;If you&amp;#x27;re really limited to running &amp;quot;just&amp;quot; llama 7b, this is great for you. But the biggest value will be when people inevitably release lora adapters for the 13b, 30b, and 65b, based on this training data (assuming it really is better than the previously released adapters).&lt;p&gt;[0] admittedly, this is based off anecdotes and github issues, and not real measurements. but smarter people than I have claimed the currently most popular loras were trained on messy data, and have started an effort to clean that data and retrain. So if the training data in this repo is high quality like the authors claim, it will benefit models of all sizes.</text></item></parent_chain><comment><author>2bitencryption</author><text>Here&amp;#x27;s my experience, having used llama+lora 7b, 13b, and 30b, on both cpu and gpu:&lt;p&gt;On gpu, processing the input prompt, even for huge prompts, is almost instant. Meaning, even if your prompt is huge, it will start generating new tokens &lt;i&gt;after&lt;/i&gt; your prompt very quickly. On a rented A6000 gpu, using llama+lora 30b, you can use huge prompts and it will start giving a new output right away.&lt;p&gt;On cpu (i.e. the project llama.cpu), it takes a very, very long time to process the input prompt, before it begins to generate new tokens.  Meaning, if you provide a huge copy&amp;#x2F;paste of code, it will take a long time to ingest all that input, before it begins outputting new tokens.&lt;p&gt;Once it finally starts outputting new tokens, the rate is surprisingly fast, not much slower than gpu.&lt;p&gt;I wish I knew the reason for this, but I&amp;#x27;m not an expert :) I&amp;#x27;ve just seen this in practice.</text></comment>
<story><title>Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa</title><url>https://github.com/nomic-ai/gpt4all</url></story><parent_chain><item><author>alchemist1e9</author><text>I’ll ask a dumb question. On another of the numerous LLM related posts I was asking if any of the self host-able open model can do code summaries at close to the quality of GPT 3.5 turbo. I was basically told nowhere close yet.&lt;p&gt;Can this potentially do that?&lt;p&gt;Ideally I’d like to have it generate descriptions of large amounts of code but would rather not burn tokens and lose privacy via OpenAI api. But I’d gladly keep a high end GPU burning on such task , even if that was actually slightly more expensive.&lt;p&gt;Edit: To clarify I do this partially now on batches of code via openAI api currently. It’s around 1-3 cents for a typical 400-800 line source code file. And I don’t mean feeding the full code base in as a single input.</text></item><item><author>2bitencryption</author><text>Having spent quite a bit of time playing around with llama.cpp, alpaca.cpp, loras, and the many other llama-based weights lately, here is my impression:&lt;p&gt;The biggest deal with this isn&amp;#x27;t the published lora adapter (which seems limited to llama 7b), but the cleaned training data, which is likely better than the previous data sets used to train the alpaca-inspired loras that have been publicly released so far. [0]&lt;p&gt;If you&amp;#x27;re really limited to running &amp;quot;just&amp;quot; llama 7b, this is great for you. But the biggest value will be when people inevitably release lora adapters for the 13b, 30b, and 65b, based on this training data (assuming it really is better than the previously released adapters).&lt;p&gt;[0] admittedly, this is based off anecdotes and github issues, and not real measurements. but smarter people than I have claimed the currently most popular loras were trained on messy data, and have started an effort to clean that data and retrain. So if the training data in this repo is high quality like the authors claim, it will benefit models of all sizes.</text></item></parent_chain><comment><author>aunty_helen</author><text>We&amp;#x27;re a long long long way off that. So check back in two months.&lt;p&gt;Jokes aside, the limiting factor will be either a technique to pack all of the code into smaller tokens like semantic search (someone else will be able to comment on this as that&amp;#x27;s the limit of my understanding) or GPU memory for input tokens.&lt;p&gt;Buying a &amp;quot;high end GPU&amp;quot; isn&amp;#x27;t buying a 4090 or even two, it&amp;#x27;s 250k on a DGX unit and putting it in a datacentre.
You will probably be able to find a service that would sign a confidentiality agreement and provide you with this service for less than 250k.</text></comment>