How to train a t2i adapter on only color and sketch?
So I'm working on a fun project for my friend and their kids. I want to take their colored drawings and turn them into amazing art that my friends can use during story time. Pretty simple using a GUI interface like invoke/comfyUI/automatic1111, but I wanted to challenge myself and write a super simple version of the full stack myself..
I've already built a jupyter notebook that uses t2i-adapter-sketch-sdxl-1.0, and built a pipeline with an ensemble of expert denoisers to get a really good outcome. However, the problem is that the SDXL t2i adapter for sketch only takes in white on black pictures. The picture that the kids drew loose all their color. So I've been googling up and down trying to figure out how to train a model or how to modify the pipeline to be able to keep the color, until I ran into your models.
So i wanted to ask for your guidance on what would be the best course of action to achieve best results for a combo of color+sketch when using SDXL and T2i adapters via python code.
I figured out an alternative way through the latent space. Thanks anyways