README.md

---
license: other
extra_gated_prompt: >-
  Copyright (c) Stability AI Ltd.


  This License Agreement (as may be amended in accordance with this License
  Agreement, “License”), between you, or your employer or other entity (if you
  are entering into this agreement on behalf of your employer or other entity)
  (“Licensee” or “you”) and Stability AI Ltd. (“Stability AI” or “we”) applies
  to your use of any computer program, algorithm, source code, object code,
  software, models, or model weights that is made available by Stability AI
  under this License (“Software”) and any specifications, manuals,
  documentation, and other written information provided by Stability AI related
  to the Software (“Documentation”). By using the Software, you agree to the
  terms of this License. If you do not agree to this License, then you do not
  have any rights to use the Software or Documentation (collectively, the
  “Software Products”), and you must immediately cease using the Software
  Products. If you are agreeing to be bound by the terms of this License on
  behalf of your employer or other entity, you represent and warrant to
  Stability AI that you have full legal authority to bind your employer or such
  entity to this License. If you do not have the requisite authority, you may
  not accept the License or access the Software Products on behalf of your
  employer or other entity.

  1. LICENSE GRANT

  a. Subject to your compliance with the Documentation and Sections 2, 3, and 5,
  Stability AI grants you a non-exclusive, worldwide, non-transferable,
  non-sublicensable, revocable, royalty free and limited license under Stability
  AI’s copyright interests to use, reproduce, and create derivative works of the
  Software solely for your non-commercial research purposes. The foregoing
  license is personal to you, and you may not assign, sublicense, distribute,
  publish, host, or otherwise make available this Software, derivative works of
  the Software, models or model weights associated with the Software, this
  License, or any other rights or obligations under this License without
  Stability AI’s prior written consent; any such assignment or sublicense
  without Stability AI’s prior written consent will be void and will
  automatically and immediately terminate this License.  For sake of clarity,
  this License does not grant to you the right or ability to extend any license
  to the Software, derivative works of the Software, or associated models or
  model weights to a non-Licensee, nor does this License permit you to create a
  new Licensee, such as by making available a copy of this License.  If you
  would like rights not granted by this License, you may seek permission by
  sending an email to [email protected].

  b. You may make a reasonable number of copies of the Documentation solely for
  your use in connection with the license to the Software granted above.

  c. The grant of rights expressly set forth in this Section 1 (License Grant)
  are the complete grant of rights to you in the Software Products, and no other
  licenses are granted, whether by waiver, estoppel, implication, equity or
  otherwise. Stability AI and its licensors reserve all rights not expressly
  granted by this License.


  2. RESTRICTIONS

  You will not, and will not permit, assist or cause any third party to:

  a. use, modify, copy, reproduce, create derivative works of, or distribute the
  Software Products (or any derivative works thereof, works incorporating the
  Software Products, or any data produced by the Software), in whole or in part,
  for (i) any commercial or production purposes, (ii) military purposes or in
  the service of nuclear technology, (iii) purposes of surveillance, including
  any research or development relating to surveillance, (iv) biometric
  processing, (v) in any manner that infringes, misappropriates, or otherwise
  violates any third-party rights, or (vi) in any manner that violates any
  applicable law and violating any privacy or security laws, rules, regulations,
  directives, or governmental requirements (including the General Data Privacy
  Regulation (Regulation (EU) 2016/679), the California Consumer Privacy Act,
  and any and all laws governing the processing of biometric information), as
  well as all amendments and successor laws to any of the foregoing;

  b. alter or remove copyright and other proprietary notices which appear on or
  in the Software Products;

  c. utilize any equipment, device, software, or other means to circumvent or
  remove any security or protection used by Stability AI in connection with the
  Software, or to circumvent or remove any usage restrictions, or to enable
  functionality disabled by Stability AI; or

  d. offer or impose any terms on the Software Products that alter, restrict, or
  are inconsistent with the terms of this License.

  e. 1) violate any applicable U.S. and non-U.S. export control and trade
  sanctions laws (“Export Laws”); 2) directly or indirectly export, re-export,
  provide, or otherwise transfer Software Products: (a) to any individual,
  entity, or country prohibited by Export Laws; (b) to anyone on U.S. or
  non-U.S. government restricted parties lists; or (c) for any purpose
  prohibited by Export Laws, including nuclear, chemical or biological weapons,
  or missile technology applications; 3) use or download Software Products if
  you or they are: (a) located in a comprehensively sanctioned jurisdiction, (b)
  currently listed on any U.S. or non-U.S. restricted parties list, or (c) for
  any purpose prohibited by Export Laws; and (4) will not disguise your location
  through IP proxying or other methods.


  3. ATTRIBUTION

  Together with any copies of the Software Products (as well as derivative works
  thereof or works incorporating the Software Products) that you distribute, you
  must provide (i) a copy of this License, and (ii) the following attribution
  notice: “SDXL 0.9 is licensed under the SDXL Research License, Copyright (c)
  Stability AI Ltd. All Rights Reserved.”


  4. DISCLAIMERS

  THE SOFTWARE PRODUCTS ARE PROVIDED “AS IS” AND “WITH ALL FAULTS” WITH NO
  WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. STABILITY AI EXPRESSLY DISCLAIMS ALL
  REPRESENTATIONS AND WARRANTIES, EXPRESS OR IMPLIED, WHETHER BY STATUTE,
  CUSTOM, USAGE OR OTHERWISE AS TO ANY MATTERS RELATED TO THE SOFTWARE PRODUCTS,
  INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY,
  FITNESS FOR A PARTICULAR PURPOSE, TITLE, SATISFACTORY QUALITY, OR
  NON-INFRINGEMENT. STABILITY AI MAKES NO WARRANTIES OR REPRESENTATIONS THAT THE
  SOFTWARE PRODUCTS WILL BE ERROR FREE OR FREE OF VIRUSES OR OTHER HARMFUL
  COMPONENTS, OR PRODUCE ANY PARTICULAR RESULTS.


  5. LIMITATION OF LIABILITY

  TO THE FULLEST EXTENT PERMITTED BY LAW, IN NO EVENT WILL STABILITY AI BE
  LIABLE TO YOU (A) UNDER ANY THEORY OF LIABILITY, WHETHER BASED IN CONTRACT,
  TORT, NEGLIGENCE, STRICT LIABILITY, WARRANTY, OR OTHERWISE UNDER THIS LICENSE,
  OR (B) FOR ANY INDIRECT, CONSEQUENTIAL, EXEMPLARY, INCIDENTAL, PUNITIVE OR
  SPECIAL DAMAGES OR LOST PROFITS, EVEN IF STABILITY AI HAS BEEN ADVISED OF THE
  POSSIBILITY OF SUCH DAMAGES. THE SOFTWARE PRODUCTS, THEIR CONSTITUENT
  COMPONENTS, AND ANY OUTPUT (COLLECTIVELY, “SOFTWARE MATERIALS”) ARE NOT
  DESIGNED OR INTENDED FOR USE IN ANY APPLICATION OR SITUATION WHERE FAILURE OR
  FAULT OF THE SOFTWARE MATERIALS COULD REASONABLY BE ANTICIPATED TO LEAD TO
  SERIOUS INJURY OF ANY PERSON, INCLUDING POTENTIAL DISCRIMINATION OR VIOLATION
  OF AN INDIVIDUAL’S PRIVACY RIGHTS, OR TO SEVERE PHYSICAL, PROPERTY, OR
  ENVIRONMENTAL DAMAGE (EACH, A “HIGH-RISK USE”). IF YOU ELECT TO USE ANY OF THE
  SOFTWARE MATERIALS FOR A HIGH-RISK USE, YOU DO SO AT YOUR OWN RISK. YOU AGREE
  TO DESIGN AND IMPLEMENT APPROPRIATE DECISION-MAKING AND RISK-MITIGATION
  PROCEDURES AND POLICIES IN CONNECTION WITH A HIGH-RISK USE SUCH THAT EVEN IF
  THERE IS A FAILURE OR FAULT IN ANY OF THE SOFTWARE MATERIALS, THE SAFETY OF
  PERSONS OR PROPERTY AFFECTED BY THE ACTIVITY STAYS AT A LEVEL THAT IS
  REASONABLE, APPROPRIATE, AND LAWFUL FOR THE FIELD OF THE HIGH-RISK USE.


  6. INDEMNIFICATION

  You will indemnify, defend and hold harmless Stability AI and our subsidiaries
  and affiliates, and each of our respective shareholders, directors, officers,
  employees, agents, successors, and assigns (collectively, the “Stability AI
  Parties”) from and against any losses, liabilities, damages, fines, penalties,
  and expenses (including reasonable attorneys’ fees) incurred by any Stability
  AI Party in connection with any claim, demand, allegation, lawsuit,
  proceeding, or investigation (collectively, “Claims”) arising out of or
  related to: (a) your access to or use of the Software Products (as well as any
  results or data generated from such access or use), including any High-Risk
  Use (defined below); (b) your violation of this License; or (c) your
  violation, misappropriation or infringement of any rights of another
  (including intellectual property or other proprietary rights and privacy
  rights). You will promptly notify the Stability AI Parties of any such Claims,
  and cooperate with Stability AI Parties in defending such Claims. You will
  also grant the Stability AI Parties sole control of the defense or settlement,
  at Stability AI’s sole option, of any Claims. This indemnity is in addition
  to, and not in lieu of, any other indemnities or remedies set forth in a
  written agreement between you and Stability AI or the other Stability AI
  Parties.


  7. TERMINATION; SURVIVAL

  a. This License will automatically terminate upon any breach by you of the
  terms of this License.

  b. We may terminate this License, in whole or in part, at any time upon notice
  (including electronic) to you.

  c. The following sections survive termination of this License: 2
  (Restrictions), 3 (Attribution), 4 (Disclaimers), 5 (Limitation on Liability),
  6 (Indemnification) 7 (Termination; Survival), 8 (Third Party Materials), 9
  (Trademarks), 10 (Applicable Law; Dispute Resolution), and 11 (Miscellaneous).


  8. THIRD PARTY MATERIALS

  The Software Products may contain third-party software or other components
  (including free and open source software) (all of the foregoing, “Third Party
  Materials”), which are subject to the license terms of the respective
  third-party licensors. Your dealings or correspondence with third parties and
  your use of or interaction with any Third Party Materials are solely between
  you and the third party. Stability AI does not control or endorse, and makes
  no representations or warranties regarding, any Third Party Materials, and
  your access to and use of such Third Party Materials are at your own risk.


  9. TRADEMARKS

  Licensee has not been granted any trademark license as part of this License
  and may not use any name or mark associated with Stability AI without the
  prior written permission of Stability AI, except to the extent necessary to
  make the reference required by the “ATTRIBUTION” section of this Agreement.


  10. APPLICABLE LAW; DISPUTE RESOLUTION

  This License will be governed and construed under the laws of the State of
  California without regard to conflicts of law provisions. Any suit or
  proceeding arising out of or relating to this License will be brought in the
  federal or state courts, as applicable, in San Mateo County, California, and
  each party irrevocably submits to the jurisdiction and venue of such courts.


  11. MISCELLANEOUS

  If any provision or part of a provision of this License is unlawful, void or
  unenforceable, that provision or part of the provision is deemed severed from
  this License, and will not affect the validity and enforceability of any
  remaining provisions. The failure of Stability AI to exercise or enforce any
  right or provision of this License will not operate as a waiver of such right
  or provision. This License does not confer any third-party beneficiary rights
  upon any other person or entity. This License, together with the
  Documentation, contains the entire understanding between you and Stability AI
  regarding the subject matter of this License, and supersedes all other written
  or oral agreements and understandings between you and Stability AI regarding
  such subject matter. No change or addition to any provision of this License
  will be binding unless it is in writing and signed by an authorized
  representative of both you and Stability AI.
extra_gated_heading: Researcher Early Access
extra_gated_description: SDXL 0.9 Research License Agreement
extra_gated_button_content: Submit application
extra_gated_fields:
  Organization: text
  Nature of research: text
  Personal researcher link (CV, website, github): text
  Other Comments: text
  I accept the above license agreement, and will use the Software non-commercially and for research purposes only: checkbox
tags:
- image-to-image
- stable-diffusion
---

# SD-XL 0.9-refiner Model Card
![row01](01.png)

This model card focuses on the model associated with the SD-XL 0.9-refiner model, available [here](https://github.com/Stability-AI/generative-models/). 
The refiner has been trained to denoise small noise levels of high quality data and as such is not expected to work as a pure text-to-image model; instead, it should only be used as an image-to-image model.
## Model

![pipeline](pipeline.png)

SDXL consists of a two-step pipeline for latent diffusion: 
First, we use a base model to generate latents of the desired output size. 
In the second step, we use a specialized high-resolution model and apply a technique called SDEdit (https://arxiv.org/abs/2108.01073, also known as "img2img") 
to the latents generated in the first step, using the same prompt.



### Model Description

- **Developed by:** Stability AI
- **Model type:** Diffusion-based text-to-image generative model
- **License:** [SDXL 0.9 Research License](https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-0.9/blob/main/LICENSE.md)
- **Model Description:** This is a model that can be used to generate and modify images based on text prompts. It is a [Latent Diffusion Model](https://arxiv.org/abs/2112.10752) that uses a pretrained text encoder ([OpenCLIP-ViT/G](https://github.com/mlfoundations/open_clip)).
- **Resources for more information:** [GitHub Repository](https://github.com/Stability-AI/generative-models) [SDXL paper on arXiv](https://arxiv.org/abs/2307.01952).

### Model Sources

- **Repository:** https://github.com/Stability-AI/generative-models
- **Demo [optional]:** https://clipdrop.co/stable-diffusion

### 🧨 Diffusers 

Make sure to upgrade diffusers to >= 0.18.0:

```
pip install diffusers --upgrade
```

In addition make sure to install `transformers`, `safetensors`, `accelerate` as well as the invisible watermark:
```
pip install transformers accelerate safetensors invisible_watermark
```

You should use the refiner in combination with [`stabilityai/stable-diffusion-xl-base-0.9`](https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9) as follows

```py
from diffusers import DiffusionPipeline
import torch

pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-0.9", torch_dtype=torch.float16, use_safetensors=True, variant="fp16")
pipe.to("cuda")

# if using torch < 2.0
# pipe.enable_xformers_memory_efficient_attention()

prompt = "An astronaut riding a green horse"

image = pipe(prompt=prompt, output_type="latent").images

pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-refiner-0.9", torch_dtype=torch.float16, use_safetensors=True, variant="fp16")
pipe.to("cuda")

# if using torch < 2.0
# pipe.enable_xformers_memory_efficient_attention()

images = pipe(prompt=prompt, image=image).images
```

When using `torch >= 2.0`, you can improve the inference speed by 20-30% with torch.compile. Simple wrap the unet with torch compile before running the pipeline:
```py
pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead", fullgraph=True)
```

If you are limited by GPU VRAM, you can enable *cpu offloading* by calling `pipe.enable_model_cpu_offload`
instead of `.to("cuda")`:

```diff
- pipe.to("cuda")
+ pipe.enable_model_cpu_offload()
```

## Uses

### Direct Use

The model is intended for research purposes only. Possible research areas and tasks include

- Generation of artworks and use in design and other artistic processes.
- Applications in educational or creative tools.
- Research on generative models.
- Safe deployment of models which have the potential to generate harmful content.
- Probing and understanding the limitations and biases of generative models.

Excluded uses are described below.

### Out-of-Scope Use

The model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.

## Limitations and Bias

### Limitations

- The model does not achieve perfect photorealism
- The model cannot render legible text
- The model struggles with more difficult tasks which involve compositionality, such as rendering an image corresponding to “A red cube on top of a blue sphere”
- Faces and people in general may not be generated properly.
- The autoencoding part of the model is lossy.

### Bias
While the capabilities of image generation models are impressive, they can also reinforce or exacerbate social biases.

## Evaluation
![comparison](comparison.png)
The chart above evaluates user preference for SDXL (with and without refinement) over Stable Diffusion 1.5 and 2.1. 
The SDXL base model performs significantly better than the previous variants, and the model combined with the refinement module achieves the best overall performance.