# Riffsuion Fine-Tune

This is a Fine-Tuned version of **Rifussion**, trained on **bass** samples extracted from the **NSynth** dataset.
The porpuse of this work is to evaluate the performance of the model to generate bass audio samples.

## Notes

* This is the way I found to achieve this goal, if you have a better idea for doing this, please share it with me.

## Quickstart Guide

Clone the **Riffusion** repository and install the requirements.txt file from: [Riffusion Github](https://github.com/riffusion/riffusion) 
```python
import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained("DaveLoay/Riffusion_FT_Bass_512_4000", torch_dtype=torch.float16).to(device)
prompt = "Your desired prompt"
image = pipe(prompt).images[0]
```

After that, you would have been generated an spectrogram saved on image. So if you want to convert this image into an audio file, you could use the **spectrogram_image_converter** mehtod contained in the **Rifussion** repo.

```python
from riffusion.spectrogram_image_converter import SpectrogramImageConverter
from riffusion.spectrogram_params import SpectrogramParams

params = SpectrogramParams()
converter = SpectrogramImageConverter(params)
audio = converter.audio_from_spectrogram_image(image)
```

## Fine Tuning

For the Fine-Tuning process, I used the bass samples from the test split in the NSynth dataset, which you can check out here: [NSynth Dataset](https://magenta.tensorflow.org/nsynth) 

You can find the pre-processed files in my repo, here: [DaveLoay/NSynth_Bass_Captions](DaveLoay/NSynth_Bass_Captions)

And as mention in the official **Rifussion** HF repo, I used the **train_text_to_image** script contained in the **Diffusers** repo, which you can check out here: [Diffusers Repo](https://github.com/huggingface/diffusers/tree/main/examples/text_to_image)

After configuring all dependencies, I used the following code to train the model:

```bash
  accelerate launch --mixed_precision="fp16"  train_text_to_image.py \
    --pretrained_model_name_or_path=riffusion/riffusion-model-v1 \
    --dataset_name=DaveLoay/NSynth_Bass_Captions \
    --resolution=512 \
    --use_ema \
    --train_batch_size=3 \
    --gradient_accumulation_steps=4 \
    --gradient_checkpointing \
    --max_train_steps=4000 \
    --learning_rate=1e-05 \
    --max_grad_norm=1 \
    --lr_scheduler="constant" --lr_warmup_steps=0 \
    --output_dir="Riffusion_FT_Bass_512_4000" \
    --push_to_hub
```

## Hardware

The hardware I used to fine-tune this model is:

* NVIDIA A100 40 GB vRAM hosted in Google Colab Pro

It took about 3 hours to complete the training process, and used about ~26 GB of vRAM.

## Credits

You can check the original repositories here:

[Riffusion](https://www.riffusion.com/)

[NSynth Dataset](https://magenta.tensorflow.org/nsynth)

[Diffusers](https://huggingface.co/docs/diffusers/index)



{"language": ["en"], "datasets": ["DaveLoay/NSynth_Bass_Captions"]}
diffusers tensorboard en dataset:DaveLoay/NSynth_Bass_Captions endpoints_compatible diffusers:StableDiffusionPipeline region:us
TAGS
#diffusers #tensorboard #en #dataset-DaveLoay/NSynth_Bass_Captions #endpoints_compatible #diffusers-StableDiffusionPipeline #region-us 

# Riffsuion Fine-Tune

This is a Fine-Tuned version of Rifussion, trained on bass samples extracted from the NSynth dataset.
The porpuse of this work is to evaluate the performance of the model to generate bass audio samples.

## Notes

* This is the way I found to achieve this goal, if you have a better idea for doing this, please share it with me.

## Quickstart Guide

Clone the Riffusion repository and install the URL file from: Riffusion Github 


After that, you would have been generated an spectrogram saved on image. So if you want to convert this image into an audio file, you could use the spectrogram_image_converter mehtod contained in the Rifussion repo.



## Fine Tuning

For the Fine-Tuning process, I used the bass samples from the test split in the NSynth dataset, which you can check out here: NSynth Dataset 

You can find the pre-processed files in my repo, here: DaveLoay/NSynth_Bass_Captions

And as mention in the official Rifussion HF repo, I used the train_text_to_image script contained in the Diffusers repo, which you can check out here: Diffusers Repo

After configuring all dependencies, I used the following code to train the model:



## Hardware

The hardware I used to fine-tune this model is:

* NVIDIA A100 40 GB vRAM hosted in Google Colab Pro

It took about 3 hours to complete the training process, and used about ~26 GB of vRAM.

## Credits

You can check the original repositories here:

Riffusion

NSynth Dataset

Diffusers



# Riffsuion Fine-Tune

This is a Fine-Tuned version of Rifussion, trained on bass samples extracted from the NSynth dataset.
The porpuse of this work is to evaluate the performance of the model to generate bass audio samples. ## Notes

* This is the way I found to achieve this goal, if you have a better idea for doing this, please share it with me. ## Quickstart Guide

Clone the Riffusion repository and install the URL file from: Riffusion Github 


After that, you would have been generated an spectrogram saved on image. So if you want to convert this image into an audio file, you could use the spectrogram_image_converter mehtod contained in the Rifussion repo. ## Fine Tuning

For the Fine-Tuning process, I used the bass samples from the test split in the NSynth dataset, which you can check out here: NSynth Dataset 

You can find the pre-processed files in my repo, here: DaveLoay/NSynth_Bass_Captions

And as mention in the official Rifussion HF repo, I used the train_text_to_image script contained in the Diffusers repo, which you can check out here: Diffusers Repo

After configuring all dependencies, I used the following code to train the model: ## Hardware

The hardware I used to fine-tune this model is:

* NVIDIA A100 40 GB vRAM hosted in Google Colab Pro

It took about 3 hours to complete the training process, and used about ~26 GB of vRAM. ## Credits

You can check the original repositories here:

Riffusion

NSynth Dataset

Diffusers
TAGS
#diffusers #tensorboard #en #dataset-DaveLoay/NSynth_Bass_Captions #endpoints_compatible #diffusers-StableDiffusionPipeline #region-us 
 # Riffsuion Fine-Tune

This is a Fine-Tuned version of Rifussion, trained on bass samples extracted from the NSynth dataset.
The porpuse of this work is to evaluate the performance of the model to generate bass audio samples. ## Notes

* This is the way I found to achieve this goal, if you have a better idea for doing this, please share it with me. ## Quickstart Guide

Clone the Riffusion repository and install the URL file from: Riffusion Github 


After that, you would have been generated an spectrogram saved on image. So if you want to convert this image into an audio file, you could use the spectrogram_image_converter mehtod contained in the Rifussion repo. ## Fine Tuning

For the Fine-Tuning process, I used the bass samples from the test split in the NSynth dataset, which you can check out here: NSynth Dataset 

You can find the pre-processed files in my repo, here: DaveLoay/NSynth_Bass_Captions

And as mention in the official Rifussion HF repo, I used the train_text_to_image script contained in the Diffusers repo, which you can check out here: Diffusers Repo

After configuring all dependencies, I used the following code to train the model: ## Hardware

The hardware I used to fine-tune this model is:

* NVIDIA A100 40 GB vRAM hosted in Google Colab Pro

It took about 3 hours to complete the training process, and used about ~26 GB of vRAM. ## Credits

You can check the original repositories here:

Riffusion

NSynth Dataset

Diffusers