lllyasviel

/

sd-controlnet-canny

like 186

Questions are listed at bottom!

Hello,

Thank you for the work bringing this to the Diffusers library! I wanted to understand if Optimum ONNX export was possible with slight modifications to the repo (i.e. adding a valid model_index.json), or if changes were needed on the 🤗 Optimum codebase as well.

I attempted to create a StableDiffusionControlNetPipeline instance in model_index.json in my personal forked repo: kmcgrath/sd-controlnet-canny-fork.

In python the model is constructed by creating a ControlNetModelobject and then passing this into the StableDiffusionControlNetModel object.

I tried to represent this in model_index.json similarly:

{
  "_class_name": "StableDiffusionControlNetPipeline",
  "_diffusers_version": "0.14.0.dev0",
  "controlnet": [
    "diffusers",
    "ControlNetModel"
  ],
  "feature_extractor": [
    "transformers",
    "CLIPFeatureExtractor"
  ],
  "safety_checker": [
    null,
    null
  ],
  "scheduler": [
    "diffusers",
    "PNDMScheduler"
  ],
  "text_encoder": [
    "transformers",
    "CLIPTextModel"
  ],
  "tokenizer": [
    "transformers",
    "CLIPTokenizer"
  ],
  "unet": [
    "diffusers",
    "UNet2DConditionModel"
  ],
  "vae": [
    "diffusers",
    "AutoencoderKL"
  ]
}

Since the above requires the "feature_extractor", "vae", "unet", etc... models and configs in seperate folders, I copied these models and configs from runwayml/stable-diffusion-v1-5 and placed them into the forked copy of this branch.

I then attempted export:
python -m optimum.exporters.onnx --model sd-controlnet-canny-fork/ --task stable-diffusion .

Optimum was able to run, but it seemed that the extra folder and argument I had created for controlnet was not recognized, instead, optimum seemed to do a basic export of just the runwayml/stable-diffusion-v1-5 models, and the output model_index.json had: "_class_name": "StableDiffusionPipeline" indicating that it did not recognize the "StableDiffusionControlNetPipeline" argument.

Questions

1. Is this supported already in Huggingface Optimum and I just did not get the model_index.json correct?
2. Or is more work required in Optimum to allow for export of StableDiffusionControlNetModel objects?

Tagging @patrickvonplaten : if you have any advice it would be greatly appreciated!

Thank you