|
--- |
|
pipeline_tag: text-to-image |
|
license: other |
|
license_name: stable-cascade-nc-community |
|
license_link: LICENSE |
|
--- |
|
|
|
# Stable Cascade Prior |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
<img src="figures/collage_1.jpg" width="800"> |
|
|
|
This model is built upon the [Würstchen](https://openreview.net/forum?id=gU58d5QeGv) architecture and its main |
|
difference to other models like Stable Diffusion is that it is working at a much smaller latent space. Why is this |
|
important? The smaller the latent space, the **faster** you can run inference and the **cheaper** the training becomes. |
|
How small is the latent space? Stable Diffusion uses a compression factor of 8, resulting in a 1024x1024 image being |
|
encoded to 128x128. Stable Cascade achieves a compression factor of 42, meaning that it is possible to encode a |
|
1024x1024 image to 24x24, while maintaining crisp reconstructions. The text-conditional model is then trained in the |
|
highly compressed latent space. Previous versions of this architecture, achieved a 16x cost reduction over Stable |
|
Diffusion 1.5. <br> <br> |
|
Therefore, this kind of model is well suited for usages where efficiency is important. Furthermore, all known extensions |
|
like finetuning, LoRA, ControlNet, IP-Adapter, LCM etc. are possible with this method as well. |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
Stable Cascade is a diffusion model trained to generate images given a text prompt. |
|
|
|
- **Developed by:** Stability AI |
|
- **Funded by:** Stability AI |
|
- **Model type:** Generative text-to-image model |
|
|
|
### Model Sources |
|
|
|
For research purposes, we recommend our `StableCascade` Github repository (https://github.com/Stability-AI/StableCascade). |
|
|
|
- **Repository:** https://github.com/Stability-AI/StableCascade |
|
- **Paper:** https://openreview.net/forum?id=gU58d5QeGv |
|
|
|
### Model Overview |
|
Stable Cascade consists of three models: Stage A, Stage B and Stage C, representing a cascade to generate images, |
|
hence the name "Stable Cascade". |
|
Stage A & B are used to compress images, similar to what the job of the VAE is in Stable Diffusion. |
|
However, with this setup, a much higher compression of images can be achieved. While the Stable Diffusion models use a |
|
spatial compression factor of 8, encoding an image with resolution of 1024 x 1024 to 128 x 128, Stable Cascade achieves |
|
a compression factor of 42. This encodes a 1024 x 1024 image to 24 x 24, while being able to accurately decode the |
|
image. This comes with the great benefit of cheaper training and inference. Furthermore, Stage C is responsible |
|
for generating the small 24 x 24 latents given a text prompt. The following picture shows this visually. |
|
|
|
<img src="figures/model-overview.jpg" width="600"> |
|
|
|
For this release, we are providing two checkpoints for Stage C, two for Stage B and one for Stage A. Stage C comes with |
|
a 1 billion and 3.6 billion parameter version, but we highly recommend using the 3.6 billion version, as most work was |
|
put into its finetuning. The two versions for Stage B amount to 700 million and 1.5 billion parameters. Both achieve |
|
great results, however the 1.5 billion excels at reconstructing small and fine details. Therefore, you will achieve the |
|
best results if you use the larger variant of each. Lastly, Stage A contains 20 million parameters and is fixed due to |
|
its small size. |
|
|
|
## Evaluation |
|
<img height="300" src="figures/comparison.png"/> |
|
According to our evaluation, Stable Cascade performs best in both prompt alignment and aesthetic quality in almost all |
|
comparisons. The above picture shows the results from a human evaluation using a mix of parti-prompts (link) and |
|
aesthetic prompts. Specifically, Stable Cascade (30 inference steps) was compared against Playground v2 (50 inference |
|
steps), SDXL (50 inference steps), SDXL Turbo (1 inference step) and Würstchen v2 (30 inference steps). |
|
|
|
## Code Example |
|
|
|
**Note:** In order to use the `torch.bfloat16` data type with the `StableCascadeDecoderPipeline` you need to have PyTorch 2.2.0 or higher installed. This also means that using the `StableCascadeCombinedPipeline` with `torch.bfloat16` requires PyTorch 2.2.0 or higher, since it calls the StableCascadeDecoderPipeline internally. |
|
|
|
If it is not possible to install PyTorch 2.2.0 or higher in your environment, the `StableCascadeDecoderPipeline` can be used on its own with the torch.float16 data type. You can download the full precision or bf16 variant weights for the pipeline and cast the weights to torch.float16. |
|
|
|
```shell |
|
pip install diffusers |
|
``` |
|
|
|
```python |
|
import torch |
|
from diffusers import StableCascadeDecoderPipeline, StableCascadePriorPipeline |
|
|
|
prompt = "an image of a shiba inu, donning a spacesuit and helmet" |
|
negative_prompt = "" |
|
|
|
prior = StableCascadePriorPipeline.from_pretrained("stabilityai/stable-cascade-prior", variant="bf16", torch_dtype=torch.bfloat16) |
|
decoder = StableCascadeDecoderPipeline.from_pretrained("stabilityai/stable-cascade", variant="bf16", torch_dtype=torch.float16) |
|
|
|
prior.enable_model_cpu_offload() |
|
prior_output = prior( |
|
prompt=prompt, |
|
height=1024, |
|
width=1024, |
|
negative_prompt=negative_prompt, |
|
guidance_scale=4.0, |
|
num_images_per_prompt=1, |
|
num_inference_steps=20 |
|
) |
|
|
|
decoder.enable_model_cpu_offload() |
|
decoder_output = decoder( |
|
image_embeddings=prior_output.image_embeddings.to(torch.float16), |
|
prompt=prompt, |
|
negative_prompt=negative_prompt, |
|
guidance_scale=0.0, |
|
output_type="pil", |
|
num_inference_steps=10 |
|
).images[0] |
|
decoder_output.save("cascade.png") |
|
``` |
|
|
|
### Using the Lite Version of the Stage B and Stage C models |
|
|
|
```python |
|
import torch |
|
from diffusers import ( |
|
StableCascadeDecoderPipeline, |
|
StableCascadePriorPipeline, |
|
StableCascadeUNet, |
|
) |
|
|
|
prompt = "an image of a shiba inu, donning a spacesuit and helmet" |
|
negative_prompt = "" |
|
|
|
prior_unet = StableCascadeUNet.from_pretrained("stabilityai/stable-cascade-prior", subfolder="prior_lite") |
|
decoder_unet = StableCascadeUNet.from_pretrained("stabilityai/stable-cascade", subfolder="decoder_lite") |
|
|
|
prior = StableCascadePriorPipeline.from_pretrained("stabilityai/stable-cascade-prior", prior=prior_unet) |
|
decoder = StableCascadeDecoderPipeline.from_pretrained("stabilityai/stable-cascade", decoder=decoder_unet) |
|
|
|
prior.enable_model_cpu_offload() |
|
prior_output = prior( |
|
prompt=prompt, |
|
height=1024, |
|
width=1024, |
|
negative_prompt=negative_prompt, |
|
guidance_scale=4.0, |
|
num_images_per_prompt=1, |
|
num_inference_steps=20 |
|
) |
|
|
|
decoder.enable_model_cpu_offload() |
|
decoder_output = decoder( |
|
image_embeddings=prior_output.image_embeddings, |
|
prompt=prompt, |
|
negative_prompt=negative_prompt, |
|
guidance_scale=0.0, |
|
output_type="pil", |
|
num_inference_steps=10 |
|
).images[0] |
|
decoder_output.save("cascade.png") |
|
``` |
|
|
|
### Loading original checkpoints with `from_single_file` |
|
|
|
Loading the original format checkpoints is supported via `from_single_file` method in the StableCascadeUNet. |
|
|
|
```python |
|
import torch |
|
from diffusers import ( |
|
StableCascadeDecoderPipeline, |
|
StableCascadePriorPipeline, |
|
StableCascadeUNet, |
|
) |
|
|
|
prompt = "an image of a shiba inu, donning a spacesuit and helmet" |
|
negative_prompt = "" |
|
|
|
prior_unet = StableCascadeUNet.from_single_file( |
|
"https://huggingface.co/stabilityai/stable-cascade/resolve/main/stage_c_bf16.safetensors", |
|
torch_dtype=torch.bfloat16 |
|
) |
|
decoder_unet = StableCascadeUNet.from_single_file( |
|
"https://huggingface.co/stabilityai/stable-cascade/blob/main/stage_b_bf16.safetensors", |
|
torch_dtype=torch.bfloat16 |
|
) |
|
|
|
prior = StableCascadePriorPipeline.from_pretrained("stabilityai/stable-cascade-prior", prior=prior_unet, torch_dtype=torch.bfloat16) |
|
decoder = StableCascadeDecoderPipeline.from_pretrained("stabilityai/stable-cascade", decoder=decoder_unet, torch_dtype=torch.bfloat16) |
|
|
|
prior.enable_model_cpu_offload() |
|
prior_output = prior( |
|
prompt=prompt, |
|
height=1024, |
|
width=1024, |
|
negative_prompt=negative_prompt, |
|
guidance_scale=4.0, |
|
num_images_per_prompt=1, |
|
num_inference_steps=20 |
|
) |
|
|
|
decoder.enable_model_cpu_offload() |
|
decoder_output = decoder( |
|
image_embeddings=prior_output.image_embeddings, |
|
prompt=prompt, |
|
negative_prompt=negative_prompt, |
|
guidance_scale=0.0, |
|
output_type="pil", |
|
num_inference_steps=10 |
|
).images[0] |
|
decoder_output.save("cascade-single-file.png") |
|
``` |
|
|
|
### Using the `StableCascadeCombinedPipeline` |
|
|
|
```python |
|
from diffusers import StableCascadeCombinedPipeline |
|
|
|
pipe = StableCascadeCombinedPipeline.from_pretrained("stabilityai/stable-cascade", variant="bf16", torch_dtype=torch.bfloat16) |
|
|
|
prompt = "an image of a shiba inu, donning a spacesuit and helmet" |
|
output = pipe( |
|
prompt=prompt, |
|
negative_prompt="", |
|
num_inference_steps=10, |
|
prior_num_inference_steps=20, |
|
prior_guidance_scale=3.0, |
|
width=1024, |
|
height=1024, |
|
) |
|
output.images[0].save("cascade-combined.png") |
|
``` |
|
|
|
## Uses |
|
|
|
### Direct Use |
|
|
|
The model is intended for research purposes for now. Possible research areas and tasks include |
|
|
|
- Research on generative models. |
|
- Safe deployment of models which have the potential to generate harmful content. |
|
- Probing and understanding the limitations and biases of generative models. |
|
- Generation of artworks and use in design and other artistic processes. |
|
- Applications in educational or creative tools. |
|
|
|
Excluded uses are described below. |
|
|
|
### Out-of-Scope Use |
|
|
|
The model was not trained to be factual or true representations of people or events, |
|
and therefore using the model to generate such content is out-of-scope for the abilities of this model. |
|
The model should not be used in any way that violates Stability AI's [Acceptable Use Policy](https://stability.ai/use-policy). |
|
|
|
## Limitations and Bias |
|
|
|
### Limitations |
|
- Faces and people in general may not be generated properly. |
|
- The autoencoding part of the model is lossy. |
|
|
|
|
|
### Recommendations |
|
|
|
The model is intended for research purposes only. |
|
|
|
## How to Get Started with the Model |
|
|
|
Check out https://github.com/Stability-AI/StableCascade |
|
|
