<story><title>Serving AI from the Basement – 192GB of VRAM Setup</title><url>https://ahmadosman.com/blog/serving-ai-from-basement/</url></story><parent_chain><item><author>tcdent</author><text>I can&amp;#x27;t believe a group of engineers are so afraid of residential power.&lt;p&gt;It is not expensive, nor is it highly technical. It&amp;#x27;s not like we&amp;#x27;re factoring in latency and crosstalk...&lt;p&gt;Read a quick howto, cruise into Home Depot and grab some legos off the shelf. Far easier to figure out than executing &amp;quot;hello world&amp;quot; without domain expertise.</text></item><item><author>mattnewton</author><text>The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?</text></item><item><author>XMasterrrr</author><text>Hey guys, this is something I have been intending to share here for a while. This setup took me some time to plan and put together, and then some more time to explore the software part of things and the possibilities that came with it.&lt;p&gt;Part of the main reason I built this was data privacy, I do not want to hand over my private data to any company to further train their closed weight models; and given the recent drop in output quality on different platforms (ChatGPT, Claude, etc), I don&amp;#x27;t regret spending the money on this setup.&lt;p&gt;I was also able to do a lot of cool things using this server by leveraging tensor parallelism and batch inference, generating synthetic data, and experimenting with finetuning models using my private data. I am currently building a model from scratch, mainly as a learning project, but I am also finding some cool things while doing so and if I can get around ironing out the kinks, I might release it and write a tutorial from my notes.&lt;p&gt;So I finally had the time this weekend to get my blog up and running, and I am planning on following up this blog post with a series of posts on my learnings and findings. I am also open to topics and ideas to experiment with on this server and write about, so feel free to shoot your shot if you have ideas you want to experiment with and don&amp;#x27;t have the hardware, I am more than willing to do that on your behalf and sharing the findings&lt;p&gt;Please let me know if you have any questions, my PMs are open, and you can also reach me on any of the socials I have posted on my website.</text></item></parent_chain><comment><author>gizmo686</author><text>A good engineer knows the difference between safe and dangerous. Setting up an AI computer is safe. Maybe you trip a circut. Maybe you interfere with something else running on your hobby computer. But nothing bad can really happen.&lt;p&gt;Residential electrical is dangerous. Maybe you electrocute yourself. Maybe you cause a fire 5 years down the line. Maybe you cause a fire for the next owner because you didn&amp;#x27;t know to protect the wire with a metal plate so they drill into it.&lt;p&gt;Having said that, 2 4090s will run you aroud $5,000, not counting any of the surrounding system. At that cost point, hireing an electritian would not be that big of an expense relativly speaking.&lt;p&gt;Also, if you are at the point where you need to add a circut for power, you might need to seriously consider cooling, which could potentially be another side quest.</text></comment>
<story><title>Serving AI from the Basement – 192GB of VRAM Setup</title><url>https://ahmadosman.com/blog/serving-ai-from-basement/</url></story><parent_chain><item><author>tcdent</author><text>I can&amp;#x27;t believe a group of engineers are so afraid of residential power.&lt;p&gt;It is not expensive, nor is it highly technical. It&amp;#x27;s not like we&amp;#x27;re factoring in latency and crosstalk...&lt;p&gt;Read a quick howto, cruise into Home Depot and grab some legos off the shelf. Far easier to figure out than executing &amp;quot;hello world&amp;quot; without domain expertise.</text></item><item><author>mattnewton</author><text>The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?</text></item><item><author>XMasterrrr</author><text>Hey guys, this is something I have been intending to share here for a while. This setup took me some time to plan and put together, and then some more time to explore the software part of things and the possibilities that came with it.&lt;p&gt;Part of the main reason I built this was data privacy, I do not want to hand over my private data to any company to further train their closed weight models; and given the recent drop in output quality on different platforms (ChatGPT, Claude, etc), I don&amp;#x27;t regret spending the money on this setup.&lt;p&gt;I was also able to do a lot of cool things using this server by leveraging tensor parallelism and batch inference, generating synthetic data, and experimenting with finetuning models using my private data. I am currently building a model from scratch, mainly as a learning project, but I am also finding some cool things while doing so and if I can get around ironing out the kinks, I might release it and write a tutorial from my notes.&lt;p&gt;So I finally had the time this weekend to get my blog up and running, and I am planning on following up this blog post with a series of posts on my learnings and findings. I am also open to topics and ideas to experiment with on this server and write about, so feel free to shoot your shot if you have ideas you want to experiment with and don&amp;#x27;t have the hardware, I am more than willing to do that on your behalf and sharing the findings&lt;p&gt;Please let me know if you have any questions, my PMs are open, and you can also reach me on any of the socials I have posted on my website.</text></item></parent_chain><comment><author>lolinder</author><text>&amp;gt; I can&amp;#x27;t believe a group of engineers are so afraid of residential power. ... Read a quick howto, cruise into Home Depot and grab some legos off the shelf. Far easier to figure out than executing &amp;quot;hello world&amp;quot; without domain expertise.&lt;p&gt;The instinct to not touch something that you don&amp;#x27;t yet deeply understand is very much an engineer&amp;#x27;s instinct. Any engineer worthy of the title has often spent weeks carefully designing a system to take care of the hundreds of edge cases that weren&amp;#x27;t apparent at a quick glance. Once you&amp;#x27;ve done that once (much less dozens of times) you have a healthy respect for the complexity that usually lurks below the surface, and you&amp;#x27;re loathe to confidently insert yourself confidently into an unfamiliar domain that has a whole engineering discipline dedicated to it. You understand that those engineers are employed full time for a reason.&lt;p&gt;The attitude you describe is one that&amp;#x27;s useful in a lot of cases and may even be correct for this particular application (though I&amp;#x27;m personally leery of it), but if confidently injecting yourself into territory you don&amp;#x27;t know well is what being an &amp;quot;engineer&amp;quot; means to you, that&amp;#x27;s a sad commentary on the state of software engineering today.</text></comment>