<story><title>Cost of self hosting Llama-3 8B-Instruct</title><url>https://blog.lytix.co/posts/self-hosting-llama-3</url></story><parent_chain><item><author>logtrees</author><text>Whoa, so you have code running in AWS making use of your local hardware via what is called a reverse SSH tunnel? I will have to look into how that works, that&amp;#x27;s pretty powerful if so. I have a mac mini that I use for builds and deploys via FTP&amp;#x2F;SFTP and was going to look into setting up &amp;quot;messaging&amp;quot; via that pipeline to access local hardware compute through file messages lol, but reverse SSH tunnel sounds like it&amp;#x27;ll be way better for directly calling executables rather than needing to parse messages from files first.</text></item><item><author>philipkglass</author><text>&lt;i&gt;Instead of using AWS another approach involves self hosting the hardware as well. Even after factoring in energy, this does dramatically lower the price.&lt;/i&gt;&lt;p&gt;&lt;i&gt;Assuming we want to mirror our setup in AWS, we’d need 4x NVidia Tesla T4s. You can buy them for about $700 on eBay.&lt;p&gt;Add in $1,000 to setup the rest of the rig and you have a final price of around:&lt;p&gt;$2,800 + $1,000 = $3,800&lt;/i&gt;&lt;p&gt;This whole exercise assumes that you&amp;#x27;re using the Llama 3 8b model. At full fp16 precision that will fit in one 3090 or 4090 GPU (the int8 version will too, and run faster, with very little degradation.) Especially if you&amp;#x27;re willing to buy GPU hardware from eBay, that will cost significantly less.&lt;p&gt;I have my home workstation with a 4090 exposed as a vLLM service to an AWS environment where I access it via reverse SSH tunnel.</text></item></parent_chain><comment><author>sneak</author><text>Look into Nebula (or Tailscale if you trust third parties).  I have all my workstations and servers on a mesh network that appears as a single &amp;#x2F;24 that is end to end encrypted, mutually authenticated and works through&amp;#x2F;behind NAT.  I can spawn a vhost on any server that reverse proxies an API to any port on any machine.&lt;p&gt;It’s been an absolute gamechanger.</text></comment>
<story><title>Cost of self hosting Llama-3 8B-Instruct</title><url>https://blog.lytix.co/posts/self-hosting-llama-3</url></story><parent_chain><item><author>logtrees</author><text>Whoa, so you have code running in AWS making use of your local hardware via what is called a reverse SSH tunnel? I will have to look into how that works, that&amp;#x27;s pretty powerful if so. I have a mac mini that I use for builds and deploys via FTP&amp;#x2F;SFTP and was going to look into setting up &amp;quot;messaging&amp;quot; via that pipeline to access local hardware compute through file messages lol, but reverse SSH tunnel sounds like it&amp;#x27;ll be way better for directly calling executables rather than needing to parse messages from files first.</text></item><item><author>philipkglass</author><text>&lt;i&gt;Instead of using AWS another approach involves self hosting the hardware as well. Even after factoring in energy, this does dramatically lower the price.&lt;/i&gt;&lt;p&gt;&lt;i&gt;Assuming we want to mirror our setup in AWS, we’d need 4x NVidia Tesla T4s. You can buy them for about $700 on eBay.&lt;p&gt;Add in $1,000 to setup the rest of the rig and you have a final price of around:&lt;p&gt;$2,800 + $1,000 = $3,800&lt;/i&gt;&lt;p&gt;This whole exercise assumes that you&amp;#x27;re using the Llama 3 8b model. At full fp16 precision that will fit in one 3090 or 4090 GPU (the int8 version will too, and run faster, with very little degradation.) Especially if you&amp;#x27;re willing to buy GPU hardware from eBay, that will cost significantly less.&lt;p&gt;I have my home workstation with a 4090 exposed as a vLLM service to an AWS environment where I access it via reverse SSH tunnel.</text></item></parent_chain><comment><author>brrrrrm</author><text>I use my mac mini exactly as described by the parent post but using ollama as the server. Super easy setup and obv chatgpt can guide you through it</text></comment>