|
--- |
|
license: other |
|
license_name: sdxl-0.9-research-license |
|
license_link: LICENSE.md |
|
extra_gated_prompt: >- |
|
Copyright (c) Stability AI Ltd. |
|
|
|
|
|
This License Agreement (as may be amended in accordance with this License |
|
Agreement, “License”), between you, or your employer or other entity (if you |
|
are entering into this agreement on behalf of your employer or other entity) |
|
(“Licensee” or “you”) and Stability AI Ltd. (“Stability AI” or “we”) applies |
|
to your use of any computer program, algorithm, source code, object code, |
|
software, models, or model weights that is made available by Stability AI |
|
under this License (“Software”) and any specifications, manuals, |
|
documentation, and other written information provided by Stability AI related |
|
to the Software (“Documentation”). By using the Software, you agree to the |
|
terms of this License. If you do not agree to this License, then you do not |
|
have any rights to use the Software or Documentation (collectively, the |
|
“Software Products”), and you must immediately cease using the Software |
|
Products. If you are agreeing to be bound by the terms of this License on |
|
behalf of your employer or other entity, you represent and warrant to |
|
Stability AI that you have full legal authority to bind your employer or such |
|
entity to this License. If you do not have the requisite authority, you may |
|
not accept the License or access the Software Products on behalf of your |
|
employer or other entity. |
|
|
|
1. LICENSE GRANT |
|
|
|
a. Subject to your compliance with the Documentation and Sections 2, 3, and 5, |
|
Stability AI grants you a non-exclusive, worldwide, non-transferable, |
|
non-sublicensable, revocable, royalty free and limited license under Stability |
|
AI’s copyright interests to use, reproduce, and create derivative works of the |
|
Software solely for your non-commercial research purposes. The foregoing |
|
license is personal to you, and you may not assign, sublicense, distribute, |
|
publish, host, or otherwise make available this Software, derivative works of |
|
the Software, models or model weights associated with the Software, this |
|
License, or any other rights or obligations under this License without |
|
Stability AI’s prior written consent; any such assignment or sublicense |
|
without Stability AI’s prior written consent will be void and will |
|
automatically and immediately terminate this License. For sake of clarity, |
|
this License does not grant to you the right or ability to extend any license |
|
to the Software, derivative works of the Software, or associated models or |
|
model weights to a non-Licensee, nor does this License permit you to create a |
|
new Licensee, such as by making available a copy of this License. If you |
|
would like rights not granted by this License, you may seek permission by |
|
sending an email to [email protected]. |
|
|
|
b. You may make a reasonable number of copies of the Documentation solely for |
|
your use in connection with the license to the Software granted above. |
|
|
|
c. The grant of rights expressly set forth in this Section 1 (License Grant) |
|
are the complete grant of rights to you in the Software Products, and no other |
|
licenses are granted, whether by waiver, estoppel, implication, equity or |
|
otherwise. Stability AI and its licensors reserve all rights not expressly |
|
granted by this License. |
|
|
|
|
|
2. RESTRICTIONS |
|
|
|
You will not, and will not permit, assist or cause any third party to: |
|
|
|
a. use, modify, copy, reproduce, create derivative works of, or distribute the |
|
Software Products (or any derivative works thereof, works incorporating the |
|
Software Products, or any data produced by the Software), in whole or in part, |
|
for (i) any commercial or production purposes, (ii) military purposes or in |
|
the service of nuclear technology, (iii) purposes of surveillance, including |
|
any research or development relating to surveillance, (iv) biometric |
|
processing, (v) in any manner that infringes, misappropriates, or otherwise |
|
violates any third-party rights, or (vi) in any manner that violates any |
|
applicable law and violating any privacy or security laws, rules, regulations, |
|
directives, or governmental requirements (including the General Data Privacy |
|
Regulation (Regulation (EU) 2016/679), the California Consumer Privacy Act, |
|
and any and all laws governing the processing of biometric information), as |
|
well as all amendments and successor laws to any of the foregoing; |
|
|
|
b. alter or remove copyright and other proprietary notices which appear on or |
|
in the Software Products; |
|
|
|
c. utilize any equipment, device, software, or other means to circumvent or |
|
remove any security or protection used by Stability AI in connection with the |
|
Software, or to circumvent or remove any usage restrictions, or to enable |
|
functionality disabled by Stability AI; or |
|
|
|
d. offer or impose any terms on the Software Products that alter, restrict, or |
|
are inconsistent with the terms of this License. |
|
|
|
e. 1) violate any applicable U.S. and non-U.S. export control and trade |
|
sanctions laws (“Export Laws”); 2) directly or indirectly export, re-export, |
|
provide, or otherwise transfer Software Products: (a) to any individual, |
|
entity, or country prohibited by Export Laws; (b) to anyone on U.S. or |
|
non-U.S. government restricted parties lists; or (c) for any purpose |
|
prohibited by Export Laws, including nuclear, chemical or biological weapons, |
|
or missile technology applications; 3) use or download Software Products if |
|
you or they are: (a) located in a comprehensively sanctioned jurisdiction, (b) |
|
currently listed on any U.S. or non-U.S. restricted parties list, or (c) for |
|
any purpose prohibited by Export Laws; and (4) will not disguise your location |
|
through IP proxying or other methods. |
|
|
|
|
|
3. ATTRIBUTION |
|
|
|
Together with any copies of the Software Products (as well as derivative works |
|
thereof or works incorporating the Software Products) that you distribute, you |
|
must provide (i) a copy of this License, and (ii) the following attribution |
|
notice: “SDXL 0.9 is licensed under the SDXL Research License, Copyright (c) |
|
Stability AI Ltd. All Rights Reserved.” |
|
|
|
|
|
4. DISCLAIMERS |
|
|
|
THE SOFTWARE PRODUCTS ARE PROVIDED “AS IS” AND “WITH ALL FAULTS” WITH NO |
|
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. STABILITY AI EXPRESSLY DISCLAIMS ALL |
|
REPRESENTATIONS AND WARRANTIES, EXPRESS OR IMPLIED, WHETHER BY STATUTE, |
|
CUSTOM, USAGE OR OTHERWISE AS TO ANY MATTERS RELATED TO THE SOFTWARE PRODUCTS, |
|
INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, |
|
FITNESS FOR A PARTICULAR PURPOSE, TITLE, SATISFACTORY QUALITY, OR |
|
NON-INFRINGEMENT. STABILITY AI MAKES NO WARRANTIES OR REPRESENTATIONS THAT THE |
|
SOFTWARE PRODUCTS WILL BE ERROR FREE OR FREE OF VIRUSES OR OTHER HARMFUL |
|
COMPONENTS, OR PRODUCE ANY PARTICULAR RESULTS. |
|
|
|
|
|
5. LIMITATION OF LIABILITY |
|
|
|
TO THE FULLEST EXTENT PERMITTED BY LAW, IN NO EVENT WILL STABILITY AI BE |
|
LIABLE TO YOU (A) UNDER ANY THEORY OF LIABILITY, WHETHER BASED IN CONTRACT, |
|
TORT, NEGLIGENCE, STRICT LIABILITY, WARRANTY, OR OTHERWISE UNDER THIS LICENSE, |
|
OR (B) FOR ANY INDIRECT, CONSEQUENTIAL, EXEMPLARY, INCIDENTAL, PUNITIVE OR |
|
SPECIAL DAMAGES OR LOST PROFITS, EVEN IF STABILITY AI HAS BEEN ADVISED OF THE |
|
POSSIBILITY OF SUCH DAMAGES. THE SOFTWARE PRODUCTS, THEIR CONSTITUENT |
|
COMPONENTS, AND ANY OUTPUT (COLLECTIVELY, “SOFTWARE MATERIALS”) ARE NOT |
|
DESIGNED OR INTENDED FOR USE IN ANY APPLICATION OR SITUATION WHERE FAILURE OR |
|
FAULT OF THE SOFTWARE MATERIALS COULD REASONABLY BE ANTICIPATED TO LEAD TO |
|
SERIOUS INJURY OF ANY PERSON, INCLUDING POTENTIAL DISCRIMINATION OR VIOLATION |
|
OF AN INDIVIDUAL’S PRIVACY RIGHTS, OR TO SEVERE PHYSICAL, PROPERTY, OR |
|
ENVIRONMENTAL DAMAGE (EACH, A “HIGH-RISK USE”). IF YOU ELECT TO USE ANY OF THE |
|
SOFTWARE MATERIALS FOR A HIGH-RISK USE, YOU DO SO AT YOUR OWN RISK. YOU AGREE |
|
TO DESIGN AND IMPLEMENT APPROPRIATE DECISION-MAKING AND RISK-MITIGATION |
|
PROCEDURES AND POLICIES IN CONNECTION WITH A HIGH-RISK USE SUCH THAT EVEN IF |
|
THERE IS A FAILURE OR FAULT IN ANY OF THE SOFTWARE MATERIALS, THE SAFETY OF |
|
PERSONS OR PROPERTY AFFECTED BY THE ACTIVITY STAYS AT A LEVEL THAT IS |
|
REASONABLE, APPROPRIATE, AND LAWFUL FOR THE FIELD OF THE HIGH-RISK USE. |
|
|
|
|
|
6. INDEMNIFICATION |
|
|
|
You will indemnify, defend and hold harmless Stability AI and our subsidiaries |
|
and affiliates, and each of our respective shareholders, directors, officers, |
|
employees, agents, successors, and assigns (collectively, the “Stability AI |
|
Parties”) from and against any losses, liabilities, damages, fines, penalties, |
|
and expenses (including reasonable attorneys’ fees) incurred by any Stability |
|
AI Party in connection with any claim, demand, allegation, lawsuit, |
|
proceeding, or investigation (collectively, “Claims”) arising out of or |
|
related to: (a) your access to or use of the Software Products (as well as any |
|
results or data generated from such access or use), including any High-Risk |
|
Use (defined below); (b) your violation of this License; or (c) your |
|
violation, misappropriation or infringement of any rights of another |
|
(including intellectual property or other proprietary rights and privacy |
|
rights). You will promptly notify the Stability AI Parties of any such Claims, |
|
and cooperate with Stability AI Parties in defending such Claims. You will |
|
also grant the Stability AI Parties sole control of the defense or settlement, |
|
at Stability AI’s sole option, of any Claims. This indemnity is in addition |
|
to, and not in lieu of, any other indemnities or remedies set forth in a |
|
written agreement between you and Stability AI or the other Stability AI |
|
Parties. |
|
|
|
|
|
7. TERMINATION; SURVIVAL |
|
|
|
a. This License will automatically terminate upon any breach by you of the |
|
terms of this License. |
|
|
|
b. We may terminate this License, in whole or in part, at any time upon notice |
|
(including electronic) to you. |
|
|
|
c. The following sections survive termination of this License: 2 |
|
(Restrictions), 3 (Attribution), 4 (Disclaimers), 5 (Limitation on Liability), |
|
6 (Indemnification) 7 (Termination; Survival), 8 (Third Party Materials), 9 |
|
(Trademarks), 10 (Applicable Law; Dispute Resolution), and 11 (Miscellaneous). |
|
|
|
|
|
8. THIRD PARTY MATERIALS |
|
|
|
The Software Products may contain third-party software or other components |
|
(including free and open source software) (all of the foregoing, “Third Party |
|
Materials”), which are subject to the license terms of the respective |
|
third-party licensors. Your dealings or correspondence with third parties and |
|
your use of or interaction with any Third Party Materials are solely between |
|
you and the third party. Stability AI does not control or endorse, and makes |
|
no representations or warranties regarding, any Third Party Materials, and |
|
your access to and use of such Third Party Materials are at your own risk. |
|
|
|
|
|
9. TRADEMARKS |
|
|
|
Licensee has not been granted any trademark license as part of this License |
|
and may not use any name or mark associated with Stability AI without the |
|
prior written permission of Stability AI, except to the extent necessary to |
|
make the reference required by the “ATTRIBUTION” section of this Agreement. |
|
|
|
|
|
10. APPLICABLE LAW; DISPUTE RESOLUTION |
|
|
|
This License will be governed and construed under the laws of the State of |
|
California without regard to conflicts of law provisions. Any suit or |
|
proceeding arising out of or relating to this License will be brought in the |
|
federal or state courts, as applicable, in San Mateo County, California, and |
|
each party irrevocably submits to the jurisdiction and venue of such courts. |
|
|
|
|
|
11. MISCELLANEOUS |
|
|
|
If any provision or part of a provision of this License is unlawful, void or |
|
unenforceable, that provision or part of the provision is deemed severed from |
|
this License, and will not affect the validity and enforceability of any |
|
remaining provisions. The failure of Stability AI to exercise or enforce any |
|
right or provision of this License will not operate as a waiver of such right |
|
or provision. This License does not confer any third-party beneficiary rights |
|
upon any other person or entity. This License, together with the |
|
Documentation, contains the entire understanding between you and Stability AI |
|
regarding the subject matter of this License, and supersedes all other written |
|
or oral agreements and understandings between you and Stability AI regarding |
|
such subject matter. No change or addition to any provision of this License |
|
will be binding unless it is in writing and signed by an authorized |
|
representative of both you and Stability AI. |
|
extra_gated_heading: Researcher Early Access |
|
extra_gated_description: SDXL 0.9 Research License Agreement |
|
extra_gated_button_content: Submit application |
|
extra_gated_fields: |
|
Organization: text |
|
Nature of research: text |
|
Personal researcher link (CV, website, github): text |
|
Other Comments: text |
|
I accept the above license agreement, and will use the Software non-commercially and for research purposes only: checkbox |
|
tags: |
|
- text-to-image |
|
- stable-diffusion |
|
--- |
|
|
|
# SD-XL 0.9-base Model Card |
|
 |
|
|
|
## Model |
|
|
|
 |
|
|
|
SDXL consists of a two-step pipeline for latent diffusion: |
|
First, we use a base model to generate latents of the desired output size. |
|
In the second step, we use a specialized high-resolution model and apply a technique called SDEdit (https://arxiv.org/abs/2108.01073, also known as "img2img") |
|
to the latents generated in the first step, using the same prompt. |
|
|
|
### Model Description |
|
|
|
- **Developed by:** Stability AI |
|
- **Model type:** Diffusion-based text-to-image generative model |
|
- **License:** [SDXL 0.9 Research License](https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9/blob/main/LICENSE.md) |
|
- **Model Description:** This is a model that can be used to generate and modify images based on text prompts. It is a [Latent Diffusion Model](https://arxiv.org/abs/2112.10752) that uses two fixed, pretrained text encoders ([OpenCLIP-ViT/G](https://github.com/mlfoundations/open_clip) and [CLIP-ViT/L](https://github.com/openai/CLIP/tree/main)). |
|
- **Resources for more information:** [GitHub Repository](https://github.com/Stability-AI/generative-models) [SDXL paper on arXiv](https://arxiv.org/abs/2307.01952). |
|
|
|
### Model Sources |
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
- **Repository:** https://github.com/Stability-AI/generative-models |
|
- **Demo [optional]:** https://clipdrop.co/stable-diffusion |
|
|
|
### 🧨 Diffusers |
|
|
|
Make sure to upgrade diffusers to >= 0.18.0: |
|
``` |
|
pip install diffusers --upgrade |
|
``` |
|
|
|
In addition make sure to install `transformers`, `safetensors`, `accelerate` as well as the invisible watermark: |
|
``` |
|
pip install invisible_watermark transformers accelerate safetensors |
|
``` |
|
|
|
You can use the model then as follows |
|
```py |
|
from diffusers import DiffusionPipeline |
|
import torch |
|
|
|
pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-0.9", torch_dtype=torch.float16, use_safetensors=True, variant="fp16") |
|
pipe.to("cuda") |
|
|
|
# if using torch < 2.0 |
|
# pipe.enable_xformers_memory_efficient_attention() |
|
|
|
prompt = "An astronaut riding a green horse" |
|
|
|
images = pipe(prompt=prompt).images[0] |
|
``` |
|
|
|
When using `torch >= 2.0`, you can improve the inference speed by 20-30% with torch.compile. Simple wrap the unet with torch compile before running the pipeline: |
|
```py |
|
pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead", fullgraph=True) |
|
``` |
|
|
|
If you are limited by GPU VRAM, you can enable *cpu offloading* by calling `pipe.enable_model_cpu_offload` |
|
instead of `.to("cuda")`: |
|
|
|
```diff |
|
- pipe.to("cuda") |
|
+ pipe.enable_model_cpu_offload() |
|
``` |
|
|
|
## Uses |
|
|
|
### Direct Use |
|
|
|
The model is intended for research purposes only. Possible research areas and tasks include |
|
|
|
- Generation of artworks and use in design and other artistic processes. |
|
- Applications in educational or creative tools. |
|
- Research on generative models. |
|
- Safe deployment of models which have the potential to generate harmful content. |
|
- Probing and understanding the limitations and biases of generative models. |
|
|
|
Excluded uses are described below. |
|
|
|
### Out-of-Scope Use |
|
|
|
The model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model. |
|
|
|
## Limitations and Bias |
|
|
|
### Limitations |
|
|
|
- The model does not achieve perfect photorealism |
|
- The model cannot render legible text |
|
- The model struggles with more difficult tasks which involve compositionality, such as rendering an image corresponding to “A red cube on top of a blue sphere” |
|
- Faces and people in general may not be generated properly. |
|
- The autoencoding part of the model is lossy. |
|
|
|
### Bias |
|
While the capabilities of image generation models are impressive, they can also reinforce or exacerbate social biases. |
|
|
|
## Evaluation |
|
 |
|
The chart above evaluates user preference for SDXL (with and without refinement) over Stable Diffusion 1.5 and 2.1. |
|
The SDXL base model performs significantly better than the previous variants, and the model combined with the refinement module achieves the best overall performance. |
