|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
tags: |
|
- audio-captioning |
|
- audiocaps |
|
- clotho |
|
- dcase-challenge |
|
- icassp-24 |
|
--- |
|
## Summary |
|
This repo contains the config & pretrained weights of the model described in the following paper: |
|
- **Improving Audio Captioning Models with Fine-grained Audio Features, Text Embedding Supervision, and LLM Mix-up Augmentation** |
|
Shih-Lun Wu, Xuankai Chang, Gordon Wichern, Jee-weon Jung, François Germain, Jonathan Le Roux, and Shinji Watanabe |
|
Int. Conf. on Acoustics, Speech, and Signal Processing (**ICASSP**) 2024 |
|
[[arXiv page](https://arxiv.org/abs/2309.17352)] |
|
## GitHub Repository |
|
To use this model, please refer to our code published at: |
|
- https://github.com/slSeanWU/beats-conformer-bart-audio-captioner |
|
## Training Data |
|
- Pretrain |
|
- **AudioCaps**: https://github.com/cdjkim/audiocaps/tree/master |
|
- **ChatGPT mix-ups from Clotho**: https://huggingface.co/datasets/slseanwu/clotho-chatgpt-mixup-50K |
|
- Finetune |
|
- **Clotho (V2)**: https://zenodo.org/records/4783391 |
|
## BibTex |
|
If you find our model useful, please consider citing our paper. Thanks! |
|
``` |
|
@inproceedings{wu2024improving, |
|
title={Improving Audio Captioning Models with Fine-grained Audio Features, Text Embedding Supervision, and LLM Mix-up Augmentation}, |
|
author={Wu, Shih-Lun and Chang, Xuankai and Wichern, Gordon and Jung, Jee-weon and Germain, Fran{\c{c}}ois and Le Roux, Jonathan and Watanabe, Shinji}, |
|
booktitle={Proc. Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP)}, |
|
year={2024} |
|
} |
|
``` |
