Hugging Face's logo

Disty0
/
sotediffusion-v2

Text-to-Image
Diffusers
Safetensors
English
StableCascadeDecoderPipeline
anime
Model card Files Community
1
sotediffusion-v2 / README.md
Disty0's picture
Disty0
Update README.md
bfdd545 verified
preview code
|
raw
history blame contribute delete
12 kB
metadata
pipeline_tag: text-to-image
license: other
license_name: faipl-1.0-sd
license_link: LICENSE
datasets: Disty0/sotediffusion-v1-text_only
base_model: Disty0/sotediffusion-wuerstchen3
tags:
  - text-to-image
  - anime
language: en
library_name: diffusers
prior:
  - Disty0/sotediffusion-v2-prior

SoteDiffusion V2

An Anime finetune of Würstchen V3 / Stable Cascade.

Release Notes

  • This release is sponsored by fal.ai/grants
  • Trained on 12M text & image paris including WD tags and natural language captions for a single epoch on 8xH100 80GB SXM5 GPUs.
  • Trained with Full FP32 and MAE Loss.

////////////////////////////////////////////////////////////////////////////////////////////////////////////////

ComfyUI

Use these arguments when starting ComfyUI: --fp16-vae --fp16-unet

Download the Stage C to unet folder: https://huggingface.co/Disty0/sotediffusion-v2/resolve/main/sotediffusion-v2-stage_c.safetensors
Download the Stage C Text Encoder to clip folder: https://huggingface.co/Disty0/sotediffusion-v2/resolve/main/sotediffusion-v2-stage_c_text_encoder.safetensors
Download the Stage B to unet folder: https://huggingface.co/Disty0/sotediffusion-v2/resolve/main/sotediffusion-v2-stage_b.safetensors
Download the Stage A to vae folder: https://huggingface.co/Disty0/sotediffusion-v2/resolve/main/stage_a_ft_hq.safetensors

Download the workflow and load it: https://huggingface.co/Disty0/sotediffusion-v2/resolve/main/comfyui_workflow.json?download=true

Stage C sampler: DPMPP 2M or DPMPP 2M SDE with SGM Uniform scheduler
Stage C steps: 28
Stage C CFG: 6.0

Stage B sampler: LCM with Exponential scheduler
Stage B steps: 14
Stage B CFG: 1.0

SD.Next

URL: https://github.com/vladmandic/automatic/

Go to Models -> Huggingface and type Disty0/sotediffusion-v2 into the model name and press download.
Load Disty0/sotediffusion-v2 after the download process is complete.

Prompt: 

your prompt goes here
very aesthetic, best quality, newest,

(New lines act the same way as BREAK in SD.Next)

Negative Prompt: 

very displeasing, displeasing, worst quality, bad quality, low quality, realistic, monochrome, comic, sketch, oldest, early, artist name, signature, blurry, simple background, upside down,

Parameters:
Sampler: Default

Steps: 28
Refiner Steps: 14

CFG: 5.0 to 6.0
Secondary CFG: 1.0 to 1.5

Resolution: 1280x1280, 1024x1536, 1024x2048, 2048x1152
Anything works as long as it's a multiply of 128.

Diffusers 

pip install git+https://github.com/huggingface/diffusers

import torch
import diffusers

device = "cuda"
dtype = torch.float16
model_path = "Disty0/sotediffusion-v2"
pipe = diffusers.AutoPipelineForText2Image.from_pretrained(model_path, torch_dtype=dtype)

# de-dupe
pipe.decoder_pipe.text_encoder = pipe.text_encoder = None # nothing uses this
del pipe.decoder_pipe.text_encoder
del pipe.prior_prior
del pipe.prior_text_encoder
del pipe.prior_tokenizer
del pipe.prior_scheduler
del pipe.prior_feature_extractor
del pipe.prior_image_encoder

pipe = pipe.to(device, dtype=dtype)
pipe.prior_pipe = pipe.prior_pipe.to(device, dtype=dtype)


def encode_prompt(
    prior_pipe,
    device,
    num_images_per_prompt,
    prompt=""
    ):

    if prompt == "":
        text_inputs = prior_pipe.tokenizer(
            prompt,
            padding="max_length",
            max_length=77,
            truncation=False,
            return_tensors="pt",
        )
        input_ids = text_inputs.input_ids
        attention_mask=None
    else:   
        text_inputs = prior_pipe.tokenizer(
            prompt,
            padding="longest",
            truncation=False,
            return_tensors="pt",
        )
        chunk = []
        padding = []
        max_len = 75
        start_token = text_inputs.input_ids[:,0].unsqueeze(0)
        end_token = text_inputs.input_ids[:,-1].unsqueeze(0)
        raw_input_ids = text_inputs.input_ids[:,1:-1]
        prompt_len = len(raw_input_ids[0])
        last_lenght = prompt_len % max_len
        
        for i in range(int((prompt_len - last_lenght) / max_len)):
            chunk.append(torch.cat([start_token, raw_input_ids[:,i*max_len:(i+1)*max_len], end_token], dim=1))
        for i in range(max_len - last_lenght):
            padding.append(text_inputs.input_ids[:,-1])
        
        last_chunk = torch.cat([raw_input_ids[:,prompt_len-last_lenght:], torch.tensor([padding])], dim=1)
        chunk.append(torch.cat([start_token, last_chunk, end_token], dim=1))
        input_ids = torch.cat(chunk, dim=0)
        attention_mask = torch.ones(input_ids.shape, device=device, dtype=torch.int64)
        attention_mask[-1,last_lenght+1:] = 0

    text_encoder_output = prior_pipe.text_encoder(
        input_ids.to(device), attention_mask=attention_mask, output_hidden_states=True
    )

    prompt_embeds = text_encoder_output.hidden_states[-1].reshape(1,-1,1280)
    prompt_embeds = prompt_embeds.to(dtype=prior_pipe.text_encoder.dtype, device=device)
    prompt_embeds = prompt_embeds.repeat_interleave(num_images_per_prompt, dim=0)

    prompt_embeds_pooled = text_encoder_output.text_embeds[0].unsqueeze(0).unsqueeze(1)
    prompt_embeds_pooled = prompt_embeds_pooled.to(dtype=prior_pipe.text_encoder.dtype, device=device)
    prompt_embeds_pooled = prompt_embeds_pooled.repeat_interleave(num_images_per_prompt, dim=0)

    return prompt_embeds, prompt_embeds_pooled


prompt = "1girl, solo, looking at viewer, open mouth, blue eyes, medium breasts, blonde hair, gloves, dress, bow, hair between eyes, bare shoulders, upper body, hair bow, indoors, elbow gloves, hand on own chest, bridal gauntlets, candlestand, smile, rim lighting, from side, castle interior, looking side,"
quality_prompt = "very aesthetic, best quality, newest"
negative_prompt = "very displeasing, displeasing, worst quality, bad quality, low quality, realistic, monochrome, comic, sketch, oldest, early, artist name, signature, blurry, simple background, upside down,"
num_images_per_prompt=1

# Encode prompts and quality prompts eperately, long prompt support and don't use attention masks for empty prompts:
# pipe, device, num_images_per_prompt, prompt
empty_prompt_embeds, _ = encode_prompt(pipe.prior_pipe, device, num_images_per_prompt, prompt="")

prompt_embeds, prompt_embeds_pooled = encode_prompt(pipe.prior_pipe, device, num_images_per_prompt, prompt=prompt)
quality_prompt_embeds, _ = encode_prompt(pipe.prior_pipe, device, num_images_per_prompt, prompt=quality_prompt)
prompt_embeds = torch.cat([prompt_embeds, quality_prompt_embeds], dim=1)

negative_prompt_embeds, negative_prompt_embeds_pooled = encode_prompt(pipe.prior_pipe, device, num_images_per_prompt, prompt=negative_prompt)

while prompt_embeds.shape[1] < negative_prompt_embeds.shape[1]:
    prompt_embeds = torch.cat([prompt_embeds, empty_prompt_embeds], dim=1)

while negative_prompt_embeds.shape[1] < prompt_embeds.shape[1]:
    negative_prompt_embeds = torch.cat([negative_prompt_embeds, empty_prompt_embeds], dim=1)

output = pipe(
    width=1024,
    height=1536,
    decoder_guidance_scale=1.0,
    prior_guidance_scale=5.0,
    prior_num_inference_steps=28,
    num_inference_steps=14,
    output_type="pil",
    prompt=prompt + " " + quality_prompt,
    negative_prompt=negative_prompt,
    prompt_embeds=prompt_embeds,
    prompt_embeds_pooled=prompt_embeds_pooled,
    negative_prompt_embeds=negative_prompt_embeds,
    negative_prompt_embeds_pooled=negative_prompt_embeds_pooled,
    num_images_per_prompt=num_images_per_prompt,
).images[0]

display(output)

Training:

Stage C

Base model: Disty0/sotediffusion-wuerstchen3
GPU used: 7x Nvidia H100 80GB SXM5

parameter value
amp no
weights fp32
save weights fp32
resolution 1024x1024
effective batch size 84
unet learning rate 2e-6
te learning rate 1e-7
optimizer AdamW 8bit
images 6M * 2 captions per image
epochs 1

Stage B

Base model: Disty0/sotediffusion-wuerstchen3-decoder
GPU used: 1x Nvidia H100 80GB SXM5

parameter value
amp no
weights fp32
save weights fp32
resolution 1024x1024
effective batch size 8
unet learning rate 8e-6
te learning rate none
optimizer AdamW
images 120K
epochs 6

WD Tags:

Model is trained with this tag order: 

aesthetic tags, quality tags, date tags, custom tags, rating tags, character, series, rest of the tags

Date:

tag date
newest 2022 to 2024
recent 2019 to 2021
mid 2015 to 2018
early 2011 to 2014
oldest 2005 to 2010

Aesthetic Tags:

Model used: shadowlilac/aesthetic-shadow-v2

score greater than tag count
0.90 extremely aesthetic 125.451
0.80 very aesthetic 887.382
0.70 aesthetic 1.049.857
0.50 slightly aesthetic 1.643.091
0.40 not displeasing 569.543
0.30 not aesthetic 445.188
0.20 slightly displeasing 341.424
0.10 displeasing 237.660
rest of them very displeasing 328.712

Quality Tags:

Model used: https://huggingface.co/hakurei/waifu-diffusion-v1-4/blob/main/models/aes-B32-v0.pth

score greater than tag count
0.980 best quality 1.270.447
0.900 high quality 498.244
0.750 great quality 351.006
0.500 medium quality 366.448
0.250 normal quality 368.380
0.125 bad quality 279.050
0.025 low quality 538.958
rest of them worst quality 1.955.966

Rating Tags:

tag count
general 1.416.451
sensitive 3.447.664
nsfw 427.459
explicit nsfw 336.925

Custom Tags:

dataset name custom tag
image boards date,
text The text says "text",
characters character, series
pixiv art by Display_Name,
visual novel cg Full_VN_Name (short_3_letter_name), visual novel cg,
anime wallpaper date, anime wallpaper,

Limitations and Bias

Bias

  • This model is intended for anime illustrations.
    Realistic capabilites are not tested at all.

Limitations

  • Can fall back to realistic.
    Add "realistic" tag to the negatives when this happens.
  • Far shot eyes and hands can be bad.
  • Still has a lot more room for more training.

License

SoteDiffusion models falls under Fair AI Public License 1.0-SD license, which is compatible with Stable Diffusion models’ license. Key points:

  1. Modification Sharing: If you modify SoteDiffusion models, you must share both your changes and the original license.
  2. Source Code Accessibility: If your modified version is network-accessible, provide a way (like a download link) for others to get the source code. This applies to derived models too.
  3. Distribution Terms: Any distribution must be under this license or another with similar rules.
  4. Compliance: Non-compliance must be fixed within 30 days to avoid license termination, emphasizing transparency and adherence to open-source values.

Notes: Anything not covered by Fair AI license is inherited from Stability AI Non-Commercial license which is named as LICENSE_INHERIT.