size_categories:1K<n<10K language:en license:mit music text-to-music music-to-text art arxiv:2307.16372 region:us
---
license: mit
language:
- en
tags:
- music
- text-to-music
- music-to-text
- art
pretty_name: LP-MusicCaps-MC
size_categories:
- 1K<n<10K
---

======================================

**!important**: Be careful when using `caption_attribute_prediction` (We don't recommend to use)!

======================================

# Dataset Card for LP-MusicCaps-MC

## Dataset Description

- **Repository:** [LP-MusicCaps repository](https://github.com/seungheondoh/lp-music-caps)
- **Paper:** [ArXiv](https://arxiv.org/abs/2307.16372)

## Dataset Summary

**LP-MusicCaps** is a Large Language Model based Pseudo Music Caption dataset for `text-to-music` and `music-to-text` tasks. We construct the music-to-caption pairs with tag-to-caption generation (using three existing multi-label tag datasets and four task instructions). The data sources are MusicCaps, Magnatagtune, and Million Song Dataset ECALS subset.

- [LP-MusicCaps MSD](https://huggingface.co/datasets/seungheondoh/LP-MusicCaps-MSD): 0.5M Audio with 2.2M Caption
- [LP-MusicCaps MTT](https://huggingface.co/datasets/seungheondoh/LP-MusicCaps-MTT): 22k Audio with 88k Caption
- **LP-MusicCaps MC (This Repo)**: 5521 Audio with 22084 Caption. We utilize 13,219 unique aspects used by 10 musicians in the [MusicCaps dataset](https://huggingface.co/datasets/google/MusicCaps) to perform tag-to-caption generation through LLM.


## Data Instances

Each instance in LP-MusicCaps MC (This Repo) represents multiple image-text pair information with meta-attributes:

```
{
  'fname': '[-0Gj8-vB1q4]-[30-40]',
  'ytid': '-0Gj8-vB1q4',
  'aspect_list': ['low quality',
    'sustained strings melody',
    'soft female vocal',
    'mellow piano melody',
    'sad',
    'soulful',
    'ballad'
  ],
  'caption_ground_truth': 'The low quality recording features a ballad song that contains sustained strings, mellow piano melody and soft female vocal singing over it. It sounds sad and soulful, like something you would hear at Sunday services.',
  'caption_writing': 'This heartfelt ballad showcases a soulful and sad low-quality sustained strings melody intertwined with a mellow piano melody, and a soft female vocal, resulting in an emotionally charged and sonically rich experience for listeners.',
  'caption_summary': 'A melancholic and soulful ballad with low-quality sustained strings, a mellow piano melody, and soft female vocals.',
  'caption_paraphrase': 'A melancholic ballad of soulful sadness featuring a low quality sustained strings melody complemented by a soft, mellow piano melody accompanied by a plaintive, soothing female vocal.',
  'caption_attribute_prediction': 'This soulful ballad features a sustained strings melody that tugs at your heartstrings, accompanied by a mellow piano melody and gentle percussion. The soft, emotionally-charged female vocal delivers poetic and poignant lyrics that speak to the sadness and pain of lost love. The addition of a beautiful string arrangement adds to the melodic depth of the song, making it a truly moving listening experience. With its slow tempo, this track exudes a mellow and introspective vibe, perfect for those moments when you need a moment to sit and reflect on the past.',
  'pseudo_attribute': ['emotional lyrics',
    'slow tempo',
    'gentle percussion',
    'string arrangement'
  ],
  'is_crawled': True,
  'author_id': 4,
  'start_s': 30,
  'end_s': 40,
  'audioset_positive_labels': '/m/0140xf,/m/02cjck,/m/04rlf',
  'is_balanced_subset': False,
  'is_audioset_eval': True
}
```

## Pseudo Caption Example:

Input Tags: 

*"video game theme, no singer, instrumental, analog sounding, small keyboard, beatboxing, playful, cheerful, groovy"*

Output Pseudo Captions

*"instrumental track has a joyful and playful vibe, perfect for a video game theme. With no singer, the analog-sounding music features a small keyboard and beatboxing, creating a groovy and cheerful atmosphere"*

[More Information for pseudo caption generation](https://github.com/seungheondoh/lp-music-caps/blob/main/lpmc/llm_captioning/generate.py)
 

## Data Fields

| Name                         | Type            | Description                                                         |
|------------------------------|-----------------|---------------------------------------------------------------------|
| fname                        | string          | File name of the data                                               |
| ytid                         | string          | YouTube ID of the data                                              |
| aspect_list                  | list of strings | List of unique aspects used by musicians in the MusicCaps dataset   |
| caption_ground_truth         | string          | Ground truth caption for the data                                   |
| caption_writing              | string          | Pseudo Caption generated through a writing instruction              |
| caption_summary              | string          | Pseudo Caption generated through a summary instruction              |
| caption_paraphrase           | string          | Pseudo Caption generated through a paraphrase instruction           |
| caption_attribute_prediction | string          | Pseudo Caption generated through a attribute_prediction instruction |
| pseudo_attribute             | list of strings | List of pseudo-attributes using in caption_attribute_prediction     |
| is_crawled                   | boolean         | Indicates whether the data is crawled or not                        |
| author_id                    | int64           | ID of the author                                                    |
| start_s                      | int64           | Start time in seconds                                               |
| end_s                        | int64           | End time in seconds                                                 |
| audioset_positive_labels     | string          | Positive labels from the AudioSet dataset                           |
| is_balanced_subset           | boolean         | Indicates whether the data is part of a balanced subset             |
| is_audioset_eval             | boolean         | Indicates whether the data is for AudioSet evaluation               |

## Considerations for Using the Data

The LP-MusicCaps dataset is recommended to be used for research purposes. Due to the wrong labeling issue, we recommend not using caption_attribute_prediction and pseudo_attribute unless it is specifically for large-scale pretraining. Additionally, the field "is_crawled" indicates the samples used in the reference paper mentioned below.

## Discussion of Biases

It will be described in a paper to be released soon.

## Other Known Limitations

It will be described in a paper to be released soon.