<story><title>Show HN: Stable Diffusion powered level editor for a 2D game</title><url>https://generalrobots.substack.com/p/dimension-hopper-part-1</url><text>Hey folks, I’ve been working on using control-net to take in a video game level (input as a depth image) and output a beautiful illustration of that level. Play with it here: dimensionhopper.com or read the blog post about what it took to get it to work. Been a super fun project.</text></story><parent_chain><item><author>robobenjie</author><text>While I have folks attention, I want to try training a model to generate monster&amp;#x2F;creature walk animations. Anyone know of a dataset of walk cycle sprite sheets that I could massage and label to see if I can make that work?</text></item></parent_chain><comment><author>mk_stjames</author><text>I have an idea for you to try - instead of training a model to produce subsequent animation frames (which is tough), instead, take a model trained on pixel art sprites in general, and then use a ControlNet with the input to the ControlNet being either a pose model or a higher res 3d model of a generic dummy character made in blender - and then generate output frame by frame, keeping the input prompting the same, but moving the ControlNet input frame by frame.&lt;p&gt;Something like how this posing works: 
&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=CiG_v61cLxI&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=CiG_v61cLxI&lt;/a&gt;&lt;p&gt;To get it down to small pixeled &amp;#x27;sprite&amp;#x27; scale, the right thing may be to actually output &amp;#x27;realistic&amp;#x27; character animation frames this way, and then &amp;#x27;de-res&amp;#x27; them via img2img into pixel art. The whole pipeline could be automated so that your only inputs are a single set of varied walking&amp;#x2F;posing&amp;#x2F;jumping control net poses and the prompts describing the characters.</text></comment>
<story><title>Show HN: Stable Diffusion powered level editor for a 2D game</title><url>https://generalrobots.substack.com/p/dimension-hopper-part-1</url><text>Hey folks, I’ve been working on using control-net to take in a video game level (input as a depth image) and output a beautiful illustration of that level. Play with it here: dimensionhopper.com or read the blog post about what it took to get it to work. Been a super fun project.</text></story><parent_chain><item><author>robobenjie</author><text>While I have folks attention, I want to try training a model to generate monster&amp;#x2F;creature walk animations. Anyone know of a dataset of walk cycle sprite sheets that I could massage and label to see if I can make that work?</text></item></parent_chain><comment><author>doctorpangloss</author><text>There are a lot of sprites to work with. As I&amp;#x27;m sure you&amp;#x27;re aware, there are artists known for making animations, like Pedro Medeiros; spriters-resource.com has material from thousands of games; you can buy the Unity Asset Store, itch.io and stock art pixel art assets; and you can use DevX Tools Pro to extract assets from hundreds of 2D pixel art Unity games. All told, there are maybe 100,000-1m examples of high quality pixel art you can scrape. It is additionally possible that it already exists in the major crawls and needs to be labeled better.&lt;p&gt;A few people have tried training on sprite sheets and emitting them directly, and it did not work.&lt;p&gt;A few people have been working specifically on walking cycles, and it has a lot of limitations.&lt;p&gt;In my specific experience with other bespoke pixel art models, if you ask for a &amp;quot;knight,&amp;quot; you&amp;#x27;re going to get a lot of the same looking knight. Fine-tuning will unlearn other concepts that are not represented in your dataset. LORAs have not been observed to work well for pixel art. You can try the Astropixel model, the highest quality in my opinion, for prototyping.&lt;p&gt;Part of this is you&amp;#x27;re really observing how powerful ControlNet, T2I-Adapters and LORAs are and you may have the expectation that something else you, a layperson, can do will be similarly powerful. Your thing is really cool. But is there some easy trick without doing all this science, for animation? No. Those are really big scientific breakthroughs, and with all the attention on video - maybe 100-1,000 academic and industry teams working on it - there still hasn&amp;#x27;t been something super robust for animation that uses LDMs. The most coherent video is happening in with NeRF, and a layperson isn&amp;#x27;t going to make that coherent with pixel art. Your best bet is to wait. That said, I&amp;#x27;m sure people are going to link here to some great hand-processed LDM videos, and maybe there&amp;#x27;s a pipeline with hand artwork a layperson can do today that would work well.</text></comment>