|
--- |
|
license: mit |
|
pipeline_tag: text-to-image |
|
tags: |
|
- diffusion |
|
- efficient |
|
- quantization |
|
- StableDiffusionXLPipeline |
|
- Diffusers |
|
base_model: |
|
- stabilityai/sdxl-turbo |
|
--- |
|
|
|
# MixDQ Model Card |
|
|
|
## Model Description |
|
|
|
MixDQ is a mixed precision quantization methods that compress the memory and computational usage of text-to-image diffusion models while preserving genration quality. |
|
It supports few-step diffusion models (e.g., SDXL-turbo, LCM-lora) to construct both fast and tiny diffusion models. Efficient CUDA kernel implemention is provided for practical resource savings. |
|
|
|
<img src="https://github.com/A-suozhang/MyPicBed/raw/master/img/mixdq_model_card_0.jpg" width="600"> |
|
|
|
|
|
## Model Sources |
|
|
|
for more information, please refer to: |
|
|
|
- Project Page: [https://a-suozhang.xyz/mixdq.github.io/](https://a-suozhang.xyz/mixdq.github.io/). |
|
- Arxiv paper: [https://arxiv.org/abs/2405.17873](https://arxiv.org/abs/2405.17873) |
|
- Github Repository: [https://github.com/A-suozhang/MixDQ](https://github.com/A-suozhang/MixDQ) |
|
|
|
## Evaluation |
|
|
|
We evaluate the MixDQ model using various metrics, including FID (fidelity), CLIPScore (image-text alignment), and ImageReward (human preference). MixDQ can achieve W8A8 quantization without performance loss. The differences between images generated by MixDQ and those generated by FP16 models are negligible. |
|
|
|
| Method | FID (↓) | ClipScore | ImageReward | |
|
|------------|---------|-----------|-------------| |
|
| FP16 | 17.15 | 0.2722 | 0.8631 | |
|
| MixDQ-W8A8 | 17.03 | 0.2703 | 0.8415 | |
|
| MixDQ-W5A8 | 17.23 | 0.2697 | 0.8307 | |
|
|
|
## Usage |
|
|
|
|
|
install the prerequisite for Mixdq: |
|
```shell |
|
# The Python versions required to run mixdq: 3.8, 3.9, 3.10 |
|
pip install -i https://pypi.org/simple/ mixdq-extension |
|
``` |
|
|
|
run the pipeline: |
|
```python |
|
pipe = DiffusionPipeline.from_pretrained( |
|
"stabilityai/sdxl-turbo", custom_pipeline="nics-efc/MixDQ", |
|
torch_dtype=torch.float16, variant="fp16" |
|
) |
|
|
|
# quant the UNet |
|
pipe.quantize_unet( |
|
w_bit = 8, |
|
a_bit = 8, |
|
bos=True, |
|
) |
|
|
|
# The set_cuda_graph func is optional and used for acceleration |
|
pipe.set_cuda_graph( |
|
run_pipeline = True, |
|
) |
|
|
|
# test the memory and the lantency of the pipeline or the UNet |
|
pipe.run_for_test( |
|
device="cuda", |
|
output_type="pil", |
|
run_pipeline=True, |
|
path="pipeline_test.png", |
|
profile=True |
|
) |
|
''' |
|
After execution is finished, there will be a report under log/sdxl folder in formats of json. |
|
This report can be opened by tensorboard for users to examine profiling results: |
|
tensorboard --logdir=./log |
|
''' |
|
|
|
# run the pipeline |
|
pipe = pipe.to("cuda") |
|
prompts = "A black Honda motorcycle parked in front of a garage." |
|
image = pipe(prompts, num_inference_steps=1, guidance_scale=0.0).images[0] |
|
image.save('mixdq_pipeline.png') |
|
``` |
|
|
|
|
|
|
|
Performance tested on NVIDIA 4080: |
|
|
|
| UNet Latency (ms) | No CUDA Graph | With CUDA Graph | |
|
|-------------------|---------------|-----------------| |
|
| FP16 version | 44.6 | 36.1 | |
|
| Quantized version | 59.1 | 24.9 | |
|
| Speedup | 0.75 | 1.45 | |
