<story><title>Serving AI from the Basement – 192GB of VRAM Setup</title><url>https://ahmadosman.com/blog/serving-ai-from-basement/</url></story><parent_chain><item><author>mattnewton</author><text>The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?</text></item><item><author>XMasterrrr</author><text>Hey guys, this is something I have been intending to share here for a while. This setup took me some time to plan and put together, and then some more time to explore the software part of things and the possibilities that came with it.&lt;p&gt;Part of the main reason I built this was data privacy, I do not want to hand over my private data to any company to further train their closed weight models; and given the recent drop in output quality on different platforms (ChatGPT, Claude, etc), I don&amp;#x27;t regret spending the money on this setup.&lt;p&gt;I was also able to do a lot of cool things using this server by leveraging tensor parallelism and batch inference, generating synthetic data, and experimenting with finetuning models using my private data. I am currently building a model from scratch, mainly as a learning project, but I am also finding some cool things while doing so and if I can get around ironing out the kinks, I might release it and write a tutorial from my notes.&lt;p&gt;So I finally had the time this weekend to get my blog up and running, and I am planning on following up this blog post with a series of posts on my learnings and findings. I am also open to topics and ideas to experiment with on this server and write about, so feel free to shoot your shot if you have ideas you want to experiment with and don&amp;#x27;t have the hardware, I am more than willing to do that on your behalf and sharing the findings&lt;p&gt;Please let me know if you have any questions, my PMs are open, and you can also reach me on any of the socials I have posted on my website.</text></item></parent_chain><comment><author>J_Shelby_J</author><text>I’m running two 3090s on a 700w psu. You definitely can get more than that out of 2000w bus.&lt;p&gt;I wrote a blog on reducing the power limits of nvidia gpus. Definitely try it out. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;shelbyjenkins.github.io&amp;#x2F;blog&amp;#x2F;power-limit-nvidia-linux&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;shelbyjenkins.github.io&amp;#x2F;blog&amp;#x2F;power-limit-nvidia-linu...&lt;/a&gt;</text></comment>
<story><title>Serving AI from the Basement – 192GB of VRAM Setup</title><url>https://ahmadosman.com/blog/serving-ai-from-basement/</url></story><parent_chain><item><author>mattnewton</author><text>The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?</text></item><item><author>XMasterrrr</author><text>Hey guys, this is something I have been intending to share here for a while. This setup took me some time to plan and put together, and then some more time to explore the software part of things and the possibilities that came with it.&lt;p&gt;Part of the main reason I built this was data privacy, I do not want to hand over my private data to any company to further train their closed weight models; and given the recent drop in output quality on different platforms (ChatGPT, Claude, etc), I don&amp;#x27;t regret spending the money on this setup.&lt;p&gt;I was also able to do a lot of cool things using this server by leveraging tensor parallelism and batch inference, generating synthetic data, and experimenting with finetuning models using my private data. I am currently building a model from scratch, mainly as a learning project, but I am also finding some cool things while doing so and if I can get around ironing out the kinks, I might release it and write a tutorial from my notes.&lt;p&gt;So I finally had the time this weekend to get my blog up and running, and I am planning on following up this blog post with a series of posts on my learnings and findings. I am also open to topics and ideas to experiment with on this server and write about, so feel free to shoot your shot if you have ideas you want to experiment with and don&amp;#x27;t have the hardware, I am more than willing to do that on your behalf and sharing the findings&lt;p&gt;Please let me know if you have any questions, my PMs are open, and you can also reach me on any of the socials I have posted on my website.</text></item></parent_chain><comment><author>tcdent</author><text>I can&amp;#x27;t believe a group of engineers are so afraid of residential power.&lt;p&gt;It is not expensive, nor is it highly technical. It&amp;#x27;s not like we&amp;#x27;re factoring in latency and crosstalk...&lt;p&gt;Read a quick howto, cruise into Home Depot and grab some legos off the shelf. Far easier to figure out than executing &amp;quot;hello world&amp;quot; without domain expertise.</text></comment>