<story><title>Running large language models like ChatGPT on a single GPU</title><url>https://github.com/Ying1123/FlexGen</url></story><parent_chain><item><author>muttled</author><text>This is cool! But I wonder if it&amp;#x27;s economical using cloud hardware. The author claims 1.12 tokens&amp;#x2F;second on the 175B parameter model (arguably comparable to GPT-3 Davinci). That&amp;#x27;s about 100k tokens a day on the GCP machine the author used. Someone double check my numbers here, but given the Davinci base cost of $0.02 per 1k tokens and GCP cost for the hardware listed &amp;quot;NVIIDA T4 (16GB) instance on GCP with 208GB of DRAM and 1.5TB of SSD&amp;quot; coming up to about $434 on spot instance pricing, you could simply use the OpenAI API and generate about 723k tokens a day for the same price as running the spot instance (which could go offline at any point due to it being a spot instance).&lt;p&gt;Running the fine-tuned versions of OpenAI models are approximately 6x more expensive per token. If you were running a fine-tuned model on local commodity hardware, the economies would start to tilt in favor of doing something like this if the load was predictable and relatively constant.</text></item></parent_chain><comment><author>ImprobableTruth</author><text>You&amp;#x27;ve made one huge mistake: Davinci&amp;#x27;s $0.02 is not just per 1k tokens &lt;i&gt;generated&lt;/i&gt; but also &lt;i&gt;context tokens consumed&lt;/i&gt;. So if you generate 50 tokens per request with 1k context, the price is actually 20 times as large at $0.40 per 1k tokens generated - much less palatable, costing 3 times as much as the cloud hosted version of this.&lt;p&gt;And that&amp;#x27;s not even taking into account the gigantic markup cloud services have.</text></comment>
<story><title>Running large language models like ChatGPT on a single GPU</title><url>https://github.com/Ying1123/FlexGen</url></story><parent_chain><item><author>muttled</author><text>This is cool! But I wonder if it&amp;#x27;s economical using cloud hardware. The author claims 1.12 tokens&amp;#x2F;second on the 175B parameter model (arguably comparable to GPT-3 Davinci). That&amp;#x27;s about 100k tokens a day on the GCP machine the author used. Someone double check my numbers here, but given the Davinci base cost of $0.02 per 1k tokens and GCP cost for the hardware listed &amp;quot;NVIIDA T4 (16GB) instance on GCP with 208GB of DRAM and 1.5TB of SSD&amp;quot; coming up to about $434 on spot instance pricing, you could simply use the OpenAI API and generate about 723k tokens a day for the same price as running the spot instance (which could go offline at any point due to it being a spot instance).&lt;p&gt;Running the fine-tuned versions of OpenAI models are approximately 6x more expensive per token. If you were running a fine-tuned model on local commodity hardware, the economies would start to tilt in favor of doing something like this if the load was predictable and relatively constant.</text></item></parent_chain><comment><author>cypress66</author><text>This is most likely aimed at people running models locally.&lt;p&gt;And a homelab with 3090s&amp;#x2F;4090s is one or two orders of magnitude cheaper than GCP, if you use them continuously.</text></comment>