Orangemix vae completely broken for img2img?
I thought all along that it was normal, but SD Upscale and inpainting for example (not sure about normal img2img, maybe?) results in washed out colors, or in case of upscale an entirely washed out image. Does anyone know how to fix this? When I want to force it to use the orangemix vae in the settings, I get nans exception errors.
It looks really bad and makes editing almost impossible when the vae is not loaded on top of inpainted pictures or upscaled ones.
Other instances of ppl talking about the same issue for more examples
https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/5217
https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/5098
Ok so if you launch with --no-half-vae orangemix vae will work in img2img as well.
@WarriorMama777 Hi could you fix this somehow so you don't need to use no half vae for it to work properly or something? You can replicate the issue by going to img2img in A3 and inpainting over a picture or using sd upscale.
Was also breaking normal img2img, not only inpainting and sd upscale btw
Comparison
probably not related directly with the .vae but more with the inpainting data in a checkpoint.
aom2/aom3
aom2/aom3 inpainting
whole picture + -- no half vae
- abyss inpainting + automatic vae = no absorption + desaturated
- abyss inpainting + orange mix vae = no absorption + desaturated
- abyss + automatic vae =desaturated
- abyss + orange mix vae =oversaturated
- abyss + anything 3.0 vae =oversaturated
= tested 20 vaes, all inconsistent.
whole picture without args
abyss inpainting + automatic vae = oversaturated halo
abyss inpainting + orange mix vae = oversaturated halo
abyss + automatic vae = really desaturated
abyss + orange mix vae =oversaturated
abyss + anything 3.0 vae =oversaturated
abyss inpainting + vae-ft-mse-84000 = light halo, this one kinda succeded.
abyss inpainting + automatic vae = strong halo.
abyss inpainting + none vae = medium halo.
abyss inpainting + orangemix vae = light halo.
abyss inpainting + anything vae = medium halo.
Apply color correction to img2img results to match original colors = color burn most of cases and vaes but:
abyss inpainting + vae-ft-mse-84000 + Apply color correction to img2img results to match original colors = partially succeded with a very thin halo that can be fixed inpainting over it, but still not the way
a full 512- inpainting ema .ckpt + vae-ft-mse-84000 + Apply color correction to img2img results to match original colors
only masked + fill = no artficats, halos, saturations, washed out colors,
only masked + original = no artficats, halos, saturations, washed out colors,
whole picture + original = no artficats, halos, saturations, washed out colors,
whole picture + fill = strong halo.
whole picture + latent noise str 1: = smooth + strong halo
whole picture + latent nothing str 1: = smooth + strong halo
sometimes increasing the mask blur 4 ,6,8,10,12,14 can fix the halos, but blur 4 is like the safe zone.
Thanks for the info.
I've only tried the 2 setups that I wrote about above, and a few other vaes. In my experience, the default settings (sd vae on auto, no changes) img2img, inpainting, and upscale would all be desaturated, and img2img would regularly produce purple artifacts on the images, especially upscaling. After using --no-half-vae and switching to orangemix vae, the saturation seems fine, and I haven't encountered purple artifacts so far, but I only found out about this today so more testing needed.
It's not perfect, but this is the maximum desaturation I'm getting
compared to the default
I will try ticking color correction and see how it does, and check the other methods you mentioned.