Update README.md

#1
by rromb - opened
Files changed (1) hide show
  1. README.md +110 -0
README.md CHANGED
@@ -1,3 +1,113 @@
1
  ---
2
  license: creativeml-openrail-m
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: creativeml-openrail-m
3
  ---
4
+ # SD-XL 1.0-base Model Card
5
+ ![row01](01.png)
6
+
7
+ ## Model
8
+
9
+ ![pipeline](pipeline.png)
10
+
11
+ SDXL consists of a mixture-of-experts pipeline for latent diffusion:
12
+ In a first step, the base model is used to generate (noisy) latents,
13
+ which are then further processed with a refinement model (available here: TODO) specialized for the final denoising steps.
14
+ Note that the base model can be used as a standalone module.
15
+
16
+ Alternatively, we can use a two-step pipeline as follows:
17
+ First, the base model is used to generate latents of the desired output size.
18
+ In the second step, we use a specialized high-resolution model and apply a technique called SDEdit (https://arxiv.org/abs/2108.01073, also known as "img2img")
19
+ to the latents generated in the first step, using the same prompt. Note that this technique is slightly slower than the first one, as it requires more function evaluations.
20
+
21
+ ### Model Description
22
+
23
+ - **Developed by:** Stability AI
24
+ - **Model type:** Diffusion-based text-to-image generative model
25
+ - **License:** [OpenRAIL-M CreativeML](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/LICENSE.md)
26
+ - **Model Description:** This is a model that can be used to generate and modify images based on text prompts. It is a [Latent Diffusion Model](https://arxiv.org/abs/2112.10752) that uses two fixed, pretrained text encoders ([OpenCLIP-ViT/G](https://github.com/mlfoundations/open_clip) and [CLIP-ViT/L](https://github.com/openai/CLIP/tree/main)).
27
+ - **Resources for more information:** [GitHub Repository](https://github.com/Stability-AI/generative-models) [SDXL paper on arXiv](https://arxiv.org/abs/2307.01952).
28
+
29
+ ### Model Sources
30
+
31
+ - **Repository:** https://github.com/Stability-AI/generative-models
32
+ - **Demo:** https://clipdrop.co/stable-diffusion
33
+
34
+
35
+ ## Evaluation
36
+ ![comparison](comparison.png)
37
+ The chart above evaluates user preference for SDXL (with and without refinement) over SDXL 0.9 and Stable Diffusion 1.5 and 2.1.
38
+ The SDXL base model performs significantly better than the previous variants, and the model combined with the refinement module achieves the best overall performance.
39
+
40
+
41
+ ### 🧨 Diffusers
42
+
43
+ Make sure to upgrade diffusers to >= 0.18.0:
44
+ ```
45
+ pip install diffusers --upgrade
46
+ ```
47
+
48
+ In addition make sure to install `transformers`, `safetensors`, `accelerate` as well as the invisible watermark:
49
+ ```
50
+ pip install invisible_watermark transformers accelerate safetensors
51
+ ```
52
+
53
+ You can use the model then as follows
54
+ ```py
55
+ from diffusers import DiffusionPipeline
56
+ import torch
57
+
58
+ pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-0.9", torch_dtype=torch.float16, use_safetensors=True, variant="fp16")
59
+ pipe.to("cuda")
60
+
61
+ # if using torch < 2.0
62
+ # pipe.enable_xformers_memory_efficient_attention()
63
+
64
+ prompt = "An astronaut riding a green horse"
65
+
66
+ images = pipe(prompt=prompt).images[0]
67
+ ```
68
+
69
+ When using `torch >= 2.0`, you can improve the inference speed by 20-30% with torch.compile. Simple wrap the unet with torch compile before running the pipeline:
70
+ ```py
71
+ pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead", fullgraph=True)
72
+ ```
73
+
74
+ If you are limited by GPU VRAM, you can enable *cpu offloading* by calling `pipe.enable_model_cpu_offload`
75
+ instead of `.to("cuda")`:
76
+
77
+ ```diff
78
+ - pipe.to("cuda")
79
+ + pipe.enable_model_cpu_offload()
80
+ ```
81
+
82
+
83
+ ## Uses
84
+
85
+ ### Direct Use
86
+
87
+ The model is intended for research purposes only. Possible research areas and tasks include
88
+
89
+ - Generation of artworks and use in design and other artistic processes.
90
+ - Applications in educational or creative tools.
91
+ - Research on generative models.
92
+ - Safe deployment of models which have the potential to generate harmful content.
93
+ - Probing and understanding the limitations and biases of generative models.
94
+
95
+ Excluded uses are described below.
96
+
97
+ ### Out-of-Scope Use
98
+
99
+ The model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.
100
+
101
+ ## Limitations and Bias
102
+
103
+ ### Limitations
104
+
105
+ - The model does not achieve perfect photorealism
106
+ - The model cannot render legible text
107
+ - The model struggles with more difficult tasks which involve compositionality, such as rendering an image corresponding to “A red cube on top of a blue sphere”
108
+ - Faces and people in general may not be generated properly.
109
+ - The autoencoding part of the model is lossy.
110
+
111
+ ### Bias
112
+ While the capabilities of image generation models are impressive, they can also reinforce or exacerbate social biases.
113
+