<story><title>A working implementation of text-to-3D DreamFusion, powered by Stable Diffusion</title><url>https://github.com/ashawkey/stable-dreamfusion</url></story><parent_chain></parent_chain><comment><author>GistNoesis</author><text>&amp;gt;Stable-Diffusion is a latent diffusion model, which diffuses in a latent space instead of the original image space. Therefore, we need the loss to propagate back from the VAE&amp;#x27;s encoder&lt;p&gt;There is also an alternative way to handle this latent difference with the original paper that should also work :&lt;p&gt;Instead of working in voxel color space, you push the latent to the voxel (Aka instead of having a voxel grid of 3d rgb color, you have a voxel grid of dimlatent latents, (you can also use spherical harmonics if you want as it works just the same in nd) ).&lt;p&gt;Only the color prediction network differ, the density is kept the same.&lt;p&gt;The NERF then directly render to the latent space (so there are less rays to render) which mean you need to decode it with the VAE only for visualization purposes and not in the training loop.</text></comment>
<story><title>A working implementation of text-to-3D DreamFusion, powered by Stable Diffusion</title><url>https://github.com/ashawkey/stable-dreamfusion</url></story><parent_chain></parent_chain><comment><author>nobbis</author><text>Took a week, as predicted: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;twitter.com&amp;#x2F;DvashElad&amp;#x2F;status&amp;#x2F;1575614411834011651&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;twitter.com&amp;#x2F;DvashElad&amp;#x2F;status&amp;#x2F;1575614411834011651&lt;/a&gt;</text></comment>