How can I fine-tune stable diffusion 3 medium on a dataset having images along with description?
How can I fine-tune stable diffusion 3 medium on dataset having images along with description?
Just create a jsonl file called "metadata.jsonl" under the images folder, and the specific contents of this file are about the file_name of the image and text of the description,like {"file_name ": "xx.png", "text": "xxx"}. Then you should use the argument of dataset_name rather than instance_data_dir.
while running the finetuning for sd3mediumdiffusers with dreambooth+lora i'm running into File "/h/env/lib/python3.11/site-packages/torch/amp/grad_scaler.py", line 260, in unscale_grads
raise ValueError("Attempting to unscale FP16 gradients.")
ValueError: Attempting to unscale FP16 gradients.
Any suggestions how to fix ?
Maybe you can use command python xxx.py to train your model rather than accelerate launch xxx.py.
Thanks, could you please clarify that If a dataset contains different captions/prompts for the different images, do we need to provide --caption_column="text" along with dataset_name ? when tried it does not work .
Yes and I forgeted to say that.
can you please tell if while training for different images and captions , instance_prompt parameter is necessary? and for 20 images what shall be the best parameters and how to inference on trained model using dreembooth+ LoRa with use of refiner?
Codes have told that"No caption column provided, defaulting to instance_prompt for all images. If your dataset " "contains captions/prompts for the images, make sure to specify the " "column as --caption_column"
So it decides to you if there are custom captions.
There are no universally optimal parameters as they need to align with specific features of your datasets.
To test your model, two methods are available: one involves adding the argument 'output_kohya_format' referring to 'train_dreambooth_lora_sdxl.py', while the other allows direct conversion to 'webui' format in the scripts directory. Subsequently, test on comfyui or webui.