size_categories:1M<n<10M license:mit arxiv:2303.17395 arxiv:2301.11325 region:us
---
license: mit
size_categories:
- 1M<n<10M
---
# Project Links
[Github](https://github.com/declare-lab/tango)

[Web](https://tango-web.github.io/)

[Huggingface Space](https://huggingface.co/spaces/declare-lab/tango)

# Dataset Description
This dataset was used to Pre-train [Tango-Full-FT-Audiocaps](https://huggingface.co/declare-lab/tango-full-ft-audiocaps). **TangoPromptBank** is a diverse corpus consisting of textual prompts and audio samples sourced from WavCaps [1], AudioCaps [9], ESC [2], UrbanSound [3], MusicCaps [4], GTZAN [5], and Musical Instruments [6] dataset. The dataset statistics are reported in Table 1. All audio clips longer than 10 seconds were segmented into partitions of successive 10 seconds or shorter. We also resampled all audio clips to 16KHz.

The WavCaps dataset consists of ChatGPT-generated captions for the FreeSound [7], BBC Sound Effects [8] (SFX), and the AudioSet strongly labeled subset. The Urban Sound and ESC50 datasets contain various environmental sounds. The Musical Instruments dataset contains sounds of guitar, drum, violin, and piano instruments. The GTZAN dataset contains sounds of different musical genres -- classical, jazz, etc. These four datasets -- Urban Sound, ESC50, Musical Instruments, GTZAN are audio classification datasets. We use the classification label (e.g., *piano*) and a more natural prompt (*sound of piano*) to create two different training instances for each audio sample from these datasets.

[1]: [WavCaps](https://arxiv.org/abs/2303.17395) [2]: [ESC](http://dl.acm.org/citation.cfm?doid=2733373.2806390)
[3]: [UrbanSound](https://dl.acm.org/doi/10.1145/2647868.2655045)
[4]: [MusicCaps](https://arxiv.org/abs/2301.11325)
[5]: [GTZAN](https://ieeexplore.ieee.org/document/1021072)
[6]: [Musical Instruments Dataset](https://www.kaggle.com/datasets/soumendraprasad/musical-instruments-sound-dataset)
[7]: [FreeSound](https://freesound.org/)
[8]: [BBC Sound Effects](https://sound-effects.bbcrewind.co.uk) [9]: [AudioCaps](https://aclanthology.org/N19-1011/)

# Dataset Statistics

| Dataset                 | Count |
|-------------------------|-------|
| AudioSet Strong         | 108K  |
| AudioCaps               | 45K   |
| Freesound               | 680K  |
| BBC                     | 374K  |
| Urban Sound             | 17K   |
| Musical Instrument      | 10K   |
| MusicCaps               | 10K   |
| Gtzan Music Genre       | 6K    |
| ESC50                   | 4K    |
| **Total**               | **1.2M**  |



# Baseline Results using TangoPromptBank for Pre-training

| **Model** | **Datasets** | **Dataset Size** | **#Params** | **FD ↓** | **KL ↓** |
| --- | --- | --- | --- | --- | --- |
| [**Tango-Full-FT-Audiocaps**](https://huggingface.co/declare-lab/tango-full-ft-audiocaps) | AS+AC+7 others | 1.2M | 866M | **18.93** | **1.12** |

# Citation

Please consider citing the following article if you found our work useful:

```bibtex
@article{ghosal2023tango,
  title={Text-to-Audio Generation using Instruction Tuned LLM and Latent Diffusion Model},
  author={Ghosal, Deepanway and Majumder, Navonil and Mehrish, Ambuj and Poria, Soujanya},
  journal={arXiv preprint arXiv:2304.13731},
  year={2023}
}
```