Diffusers and Model keys conflict
I understand I may be jumping the gun a little bit but if I run the example script
import torch
from diffusers import StableDiffusion3Pipeline
pipe = StableDiffusion3Pipeline.from_pretrained("stabilityai/stable-diffusion-3-medium-diffusers", torch_dtype=torch.float16)
pipe.to("cuda")
image = pipe(
prompt="a photo of a cat holding a sign that says hello world",
negative_prompt="",
num_inference_steps=28,
height=1024,
width=1024,
guidance_scale=7.0,
).images[0]
image.save("sd3_hello_world.png")
I get a stack of errors, seems to be conflicts with the model.
A get loads of these style warnings, far to many to count...
/Volumes/SSD2TB/AI/Diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py:2044: UserWarning: for pos_embed.proj.weight: copying from a non-meta parameter in the checkpoint to a meta parameter in the current model, which is a no-op. (Did you mean to pass `assign=True` to assign items in the state dictionary to their corresponding key in the module instead of copying them in place?)
and terminates with
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for SD3Transformer2DModel:
Missing key(s) in state_dict: "pos_embed.pos_embed", "transformer_blocks.14.attn.to_add_out.weight", "transformer_blocks.14.attn.to_add_out.bias", "transformer_blocks.14.ff.net.0.proj.weight", "transformer_blocks.14.ff.net.0.proj.bias", "transformer_blocks.14.ff.net.2.weight", "transformer_blocks.14.ff.net.2.bias", "transformer_blocks.14.ff_context.net.0.proj.
listed what looks like all the keys
Have the same :)
Please make sure to pass low_cpu_mem_usage=False and device_map=None if you want to randomly initialize those weights or else make sure your checkpoint file is correct. have this error
The model is the one downloaded by Diffusers from HF, so the example needs fixing at a minimum.
Making a local copy and deleting
transformers/diffusion_pytorch_model.safetensors.index.json
transformers/diffusion_pytorch_model-00001-of-00002.safetensors
transformers/diffusion_pytorch_model-00002-of-00002.safetensors
Also works. So removing those from the HF Repo would likely work too
Which I see has been already done
Example code working now, closing the issue