h1t commited on
Commit
afc9e0b
1 Parent(s): c490bed

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +79 -0
README.md ADDED
@@ -0,0 +1,79 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: diffusers
3
+ base_model: stabilityai/stable-diffusion-xl-base-1.0
4
+ tags:
5
+ - text-to-image
6
+ license: openrail++
7
+ inference: false
8
+ ---
9
+
10
+ # One More Step
11
+
12
+ One More Step (OMS) module was proposed in [One More Step: A Versatile Plug-and-Play Module for Rectifying Diffusion Schedule Flaws and Enhancing Low-Frequency Controls](https://github.com/mhh0318/OneMoreStep)
13
+ by *Minghui Hu, Jianbin Zheng, Chuanxia Zheng, Tat-Jen Cham et al.*
14
+
15
+
16
+ By **adding one small step** on the top the sampling process, we can address the issues caused by the current schedule flaws of diffusion models **without changing the original model parameters**. This also allows for some control over low-frequency information, such as color.
17
+
18
+ Our model is **versatile** and can be integrated into almost all widely-used Stable Diffusion frameworks. It's compatible with community favorites such as **LoRA, ControlNet, Adapter, and foundational models**.
19
+
20
+
21
+ ## Usage
22
+
23
+ OMS now is supported 🤗 `diffusers` with a customized pipeline [github](https://github.com/mhh0318/OneMoreStep). To run the model (especially with `LCM` variant), first install the latest version of `diffusers` library as well as `accelerate` and `transformers`.
24
+
25
+ ```bash
26
+ pip install --upgrade pip
27
+ pip install --upgrade diffusers transformers accelerate
28
+ ```
29
+
30
+ And then we clone the repo
31
+ ```bash
32
+ git clone https://github.com/mhh0318/OneMoreStep.git
33
+ cd OneMoreStep
34
+ ```
35
+
36
+
37
+ ### SDXL
38
+
39
+ The OMS module can be loaded with SDXL base model `stabilityai/stable-diffusion-xl-base-1.0`.
40
+ And all the SDXL based model and its LoRA can **share the same OMS** `h1t/oms_b_openclip_xl`.
41
+
42
+ Here is an example for SDXL with LCM-LoRA.
43
+ Firstly import the related packages and choose SDXL based backbone and LoRA:
44
+
45
+ ```python
46
+ import torch
47
+ from diffusers import StableDiffusionXLPipeline, LCMScheduler
48
+
49
+ sd_pipe = StableDiffusionXLPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16, variant="fp16", add_watermarker=False).to('cuda')
50
+
51
+ sd_scheduler = LCMScheduler.from_config(sd_pipe.scheduler.config)
52
+ sd_pipe.load_lora_weights('latent-consistency/lcm-lora-sdxl', variant="fp16")
53
+ ```
54
+
55
+ Following import the customized OMS pipeline to wrap the backbone and add OMS for sampling. We have uploaded the `.safetensors` to [HuggingFace Hub](https://huggingface.co/h1t/). There are 2 choices for SDXL backbone currently, one is base OMS module with OpenCLIP text encoder [h1t/oms_b_openclip_xl)](https://huggingface.co/h1t/oms_b_openclip_xl) and the other is large OMS module with two text encoder followed by SDXL architecture [h1t/oms_l_mixclip_xl)](https://huggingface.co/h1t/oms_b_mixclip_xl).
56
+ ```python
57
+ from diffusers_patch import OMSPipeline
58
+
59
+ pipe = OMSPipeline.from_pretrained('h1t/oms_b_openclip_xl', sd_pipeline = sd_pipe, torch_dtype=torch.float16, variant="fp16", trust_remote_code=True, sd_scheduler=sd_scheduler)
60
+ pipe.to('cuda')
61
+ ```
62
+
63
+ After setting a random seed, we can easily generate images with the OMS module.
64
+ ```python
65
+ prompt = 'close-up photography of old man standing in the rain at night, in a street lit by lamps, leica 35mm summilux'
66
+ generator = torch.Generator(device=pipe.device).manual_seed(1024)
67
+
68
+ image = pipe(prompt, guidance_scale=1, num_inference_steps=4, generator=generator)
69
+ image['images'][0]
70
+ ```
71
+
72
+ Or we can offload the OMS module and generate a image only using backbone
73
+ ```python
74
+ image = pipe(prompt, guidance_scale=1, num_inference_steps=4, generator=generator, oms_flag=False)
75
+ image['images'][0]
76
+ ```
77
+
78
+ For more models and more functions like diverse prompt, please refer to [OMS Repo](https://github.com/mhh0318/OneMoreStep).
79
+