Update README.md (#1)

- Update README.md (d7ca456f97abd31fe1f20fc5011293da5879eb51)
- Upload hug_lab_grid.png (3bd29be1e826556184d10de9a87c1cd80c8a030e)
- Update README.md (d173db61274c9a951b429e3e79398d49258123d7)
- Update README.md (cd9b02220b7fabec41132e80f0a7042a66ce2446)
- Update README.md (44dbf89a0d0ca604a04a31b960cd46c7c2cee038)

.gitattributes

@@ -37,3 +37,4 @@ cann-small-couple.png filter=lfs diff=lfs merge=lfs -text
 cann-small-hf-ofice.png filter=lfs diff=lfs merge=lfs -text
 cann-small-megatron.png filter=lfs diff=lfs merge=lfs -text
 cann-small-woman.png filter=lfs diff=lfs merge=lfs -text
+hug_lab_grid.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -10,9 +10,10 @@ tags:
 inference: false
 ---
     
-# SDXL-controlnet: Canny
+# Small SDXL-controlnet: Canny
 
-These are controlnet weights trained on stabilityai/stable-diffusion-xl-base-1.0 with canny conditioning. You can find some example images in the following. 
+These are small controlnet weights trained on stabilityai/stable-diffusion-xl-base-1.0 with canny conditioning. This checkpoint is 7x smaller than the original XL controlnet checkpoint.
+You can find some example images in the following. 
 
 prompt: aerial view, a futuristic research complex in a bright foggy jungle, hard lighting
 ![images_0)](./cann-small-hf-ofice.png)
@@ -46,19 +47,19 @@ import numpy as np
 import cv2
 
 prompt = "aerial view, a futuristic research complex in a bright foggy jungle, hard lighting"
-negative_prompt = 'low quality, bad quality, sketches'
+negative_prompt = "low quality, bad quality, sketches"
 
 image = load_image("https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/sd_controlnet/hf-logo.png")
 
 controlnet_conditioning_scale = 0.5  # recommended for good generalization
 
 controlnet = ControlNetModel.from_pretrained(
-    "diffusers/controlnet-canny-sdxl-1.0",
+    "diffusers/controlnet-canny-sdxl-1.0-small",
     torch_dtype=torch.float16
 )
 vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
 pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
-    "diffusers/controlnet-canny-sdxl-1.0-small",
+    "stabilityai/stable-diffusion-xl-base-1.0",
     controlnet=controlnet,
     vae=vae,
     torch_dtype=torch.float16,
@@ -73,33 +74,35 @@ image = Image.fromarray(image)
 
 images = pipe(
     prompt, negative_prompt=negative_prompt, image=image, controlnet_conditioning_scale=controlnet_conditioning_scale,
-    ).images
+).images
 
 images[0].save(f"hug_lab.png")
 ```
 
-![images_10)](./out_hug_lab_7.png)
+![hug_lab_grid)](./hug_lab_grid.png)
 
 To more details, check out the official documentation of [`StableDiffusionXLControlNetPipeline`](https://huggingface.co/docs/diffusers/main/en/api/pipelines/controlnet_sdxl).
 
+🚨 Please note that this checkpoint is experimental and there's a lot of room for improvement. We encourage the community to build on top of it, improve it, and provide us with feedback. 🚨
+
 ### Training
 
 Our training script was built on top of the official training script that we provide [here](https://github.com/huggingface/diffusers/blob/main/examples/controlnet/README_sdxl.md). 
+You can refer to [this script](https://github.com/patil-suraj/muse-experiments/blob/f71e7e79af24509ddb4e1b295a1d0ef8d8758dc9/ctrlnet/train_controlnet_webdataset.py) for full discolsure.
+
+* This checkpoint does not perform distillation. We just use a smaller ControlNet initialized from the SDXL UNet. We
+encourage the community to try and conduct distillation too. This resource might be of help in [this regard](https://huggingface.co/blog/sd_distillation). 
+* To learn more about how the ControlNet was initialized, refer to [this code block](https://github.com/patil-suraj/muse-experiments/blob/f71e7e79af24509ddb4e1b295a1d0ef8d8758dc9/ctrlnet/train_controlnet_webdataset.py#L1020C1-L1042C36). 
+* It does not have any attention blocks.
+* The model works pretty good on most conditioning images. But for more complex conditionings, the bigger checkpoints might be better. We are still working on improving the quality of this checkpoint and looking for feedback from the community.
+* We recommend playing around with the `controlnet_conditioning_scale` and `guidance_scale` arguments for potentially better
+image generation quality.
 
 #### Training data
-This checkpoint was first trained for 20,000 steps on laion 6a resized to a max minimum dimension of 384. 
-It was then further trained for 20,000 steps on laion 6a resized to a max minimum dimension of 1024 and 
-then filtered to contain only minimum 1024 images. We found the further high resolution finetuning was 
-necessary for image quality.
+The model was trained on 3M images from LAION aesthetic 6 plus subset, with batch size of 256 for 50k steps with constant learning rate of 3e-5.
 
 #### Compute
-one 8xA100 machine
-
-#### Batch size
-Data parallel with a single gpu batch size of 8 for a total batch size of 64.
-
-#### Hyper Parameters
-Constant learning rate of 1e-4 scaled by batch size for total learning rate of 64e-4
+One 8xA100 machine
 
 #### Mixed precision
-fp16
+FP16