DionTimmer radames commited on
Commit
4ca3cf8
1 Parent(s): d70c790

how to diffursers (#3)

Browse files

- how to diffursers (b4fe1a314bf5c99b9c8aacda6cbb3f7c84089519)


Co-authored-by: Radamés Ajna <[email protected]>

Files changed (1) hide show
  1. README.md +65 -0
README.md CHANGED
@@ -18,6 +18,71 @@ pipeline_tag: image-to-image
18
  This repo holds the safetensors & diffusers versions of the QR code conditioned ControlNet for Stable Diffusion v1.5.
19
  The Stable Diffusion 2.1 version is marginally more effective, as it was developed to address my specific needs. However, this 1.5 version model was also trained on the same dataset for those who are using the older version.
20
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
21
  ## Performance and Limitations
22
 
23
  These models perform quite well in most cases, but please note that they are not 100% accurate. In some instances, the QR code shape might not come through as expected. You can increase the ControlNet weight to emphasize the QR code shape. However, be cautious as this might negatively impact the style of your output.**To optimize for scanning, please generate your QR codes with correction mode 'H' (30%).**
 
18
  This repo holds the safetensors & diffusers versions of the QR code conditioned ControlNet for Stable Diffusion v1.5.
19
  The Stable Diffusion 2.1 version is marginally more effective, as it was developed to address my specific needs. However, this 1.5 version model was also trained on the same dataset for those who are using the older version.
20
 
21
+ ## How to use with Diffusers
22
+
23
+
24
+ ```bash
25
+ pip -q install diffusers transformers accelerate torch xformers
26
+ ```
27
+
28
+ ```python
29
+ import torch
30
+ from PIL import Image
31
+ from diffusers import StableDiffusionControlNetImg2ImgPipeline, ControlNetModel, DDIMScheduler
32
+ from diffusers.utils import load_image
33
+
34
+ controlnet = ControlNetModel.from_pretrained("DionTimmer/controlnet_qrcode-control_v1p_sd15",
35
+ torch_dtype=torch.float16)
36
+
37
+ pipe = StableDiffusionControlNetImg2ImgPipeline.from_pretrained(
38
+ "runwayml/stable-diffusion-v1-5",
39
+ controlnet=controlnet,
40
+ safety_checker=None,
41
+ torch_dtype=torch.float16
42
+ )
43
+
44
+ pipe.enable_xformers_memory_efficient_attention()
45
+ pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)
46
+ pipe.enable_model_cpu_offload()
47
+
48
+ def resize_for_condition_image(input_image: Image, resolution: int):
49
+ input_image = input_image.convert("RGB")
50
+ W, H = input_image.size
51
+ k = float(resolution) / min(H, W)
52
+ H *= k
53
+ W *= k
54
+ H = int(round(H / 64.0)) * 64
55
+ W = int(round(W / 64.0)) * 64
56
+ img = input_image.resize((W, H), resample=Image.LANCZOS)
57
+ return img
58
+
59
+
60
+ # play with guidance_scale, controlnet_conditioning_scale and strength to make a valid QR Code Image
61
+
62
+ # qr code image
63
+ source_image = load_image("https://s3.amazonaws.com/moonup/production/uploads/6064e095abd8d3692e3e2ed6/A_RqHaAM6YHBodPLwqtjn.png")
64
+ # initial image, anything
65
+ init_image = load_image("https://s3.amazonaws.com/moonup/production/uploads/noauth/KfMBABpOwIuNolv1pe3qX.jpeg")
66
+ condition_image = resize_for_condition_image(source_image, 768)
67
+ init_image = resize_for_condition_image(init_image, 768)
68
+ generator = torch.manual_seed(123121231)
69
+ image = pipe(prompt="a bilboard in NYC with a qrcode",
70
+ negative_prompt="ugly, disfigured, low quality, blurry, nsfw",
71
+ image=init_image,
72
+ control_image=condition_image,
73
+ width=768,
74
+ height=768,
75
+ guidance_scale=20,
76
+ controlnet_conditioning_scale=1.5,
77
+ generator=generator,
78
+ strength=0.9,
79
+ num_inference_steps=150,
80
+ )
81
+
82
+ image.images[0]
83
+
84
+ ```
85
+
86
  ## Performance and Limitations
87
 
88
  These models perform quite well in most cases, but please note that they are not 100% accurate. In some instances, the QR code shape might not come through as expected. You can increase the ControlNet weight to emphasize the QR code shape. However, be cautious as this might negatively impact the style of your output.**To optimize for scanning, please generate your QR codes with correction mode 'H' (30%).**