size_categories:10M<n<100M language:en license:cc-by-4.0 music arxiv:2306.13512 doi:10.57967/hf/1190 region:us
---
license: cc-by-4.0
language:
- en
tags:
- music
size_categories:
- 10M<n<100M
dataset_info:
  features:
  - name: video_url_youtube
    dtype: string
  - name: video_title_youtube
    dtype: string
  - name: track_name_spotify
    dtype: string
  - name: video_duration_youtube_sec
    dtype: float64
  - name: preview_url_spotify
    dtype: string
  - name: video_view_count_youtube
    dtype: float64
  - name: video_thumbnail_url_youtube
    dtype: string
  - name: search_query_youtube
    dtype: string
  - name: video_description_youtube
    dtype: string
  - name: track_id_spotify
    dtype: string
  - name: album_id_spotify
    dtype: string
  - name: artist_id_spotify
    sequence: string
  - name: track_duration_spotify_ms
    dtype: int64
  - name: primary_artist_name_spotify
    dtype: string
  - name: track_release_date_spotify
    dtype: string
  - name: explicit_content_spotify
    dtype: bool
  - name: similarity_duration
    dtype: float64
  - name: similarity_query_video_title
    dtype: float64
  - name: similarity_query_description
    dtype: float64
  - name: similarity_audio
    dtype: float64
  - name: audio_embedding_spotify
    sequence: float32
  - name: audio_embedding_youtube
    sequence: float32
  splits:
  - name: train
    num_bytes: 73263841657.0
    num_examples: 15296232
  download_size: 88490703682
  dataset_size: 73263841657.0
---

### Getting Started

You can download the dataset using HuggingFace:
```python
from datasets import load_dataset
ds = load_dataset("DISCOX/DISCO-10M")
```

## Dataset Structure

The dataset contains the following features:
```json
{
 'video_url_youtube',
 'video_title_youtube',
 'track_name_spotify',
 'video_duration_youtube_sec',
 'preview_url_spotify',
 'video_view_count_youtube',
 'video_thumbnail_url_youtube',
 'search_query_youtube',
 'video_description_youtube',
 'track_id_spotify',
 'album_id_spotify',
 'artist_id_spotify',
 'track_duration_spotify_ms',
 'primary_artist_name_spotify',
 'track_release_date_spotify',
 'explicit_content_spotify',
 'similarity_duration',
 'similarity_query_video_title',
 'similarity_query_description',
 'similarity_audio',
 'audio_embedding_spotify',
 'audio_embedding_youtube',
}
```

## What is DISCO-10M?

DISCO-10M is a music dataset created to democratize research on large-scale machine learning models for music.

The dataset contains no music due to copyright laws.
The audio embedding features were computed using [Laion-CLAP](https://github.com/LAION-AI/CLAP), and can be used instead of the raw audio for many down-stream tasks.
In case the raw audio is needed, it can be downloaded from the provided Spotify preview URL or via the YouTube link.
DISCO-10M was created by collecting a list of 400,000 artist IDs and 2.6M track IDs from Spotify, and collecting YouTube video links that match the track duration,
artist name, and track names. These matches were computed using the following three similarity metrics:

- Duration similarity: ` 1 - abs(track_duration_spotify - video_duration_youtube) / max(track_duration_spotify, video_duration_youtube) `
- Text similarity is calculated using the cosine similarity between the embedding of the search query and the embedding of the video title, as well as the search query embedding and the video description embedding. Embeddings are computed using [Sentence Bert](https://huggingface.co/sentence-transformers).
- Audio similarity is calculated using the cosine similarity between the Spotify preview snippet audio embedding and the YouTube audio embedding.

For DISCO-10M we only keep samples that return true for: ` duration_similarity > 0.25 and (description_similarity > 0.65 or title_similarity > 0.65) and audio_similarity > 0.4 `

We offer three subsets based on DISCO-10M:

- [DISCO-10K-random](https://huggingface.co/datasets/DISCOX/DISCO-10K-random): a small subset of random samples from the entire dataset.
- [DISCO-200K-random](https://huggingface.co/datasets/DISCOX/DISCO-200K-random): a subset of random samples, useful for a light-weight and representative analysis of the entire dataset.
- [DISCO-200K-high-quality](https://huggingface.co/datasets/DISCOX/DISCO-200K-high-quality): a subset of samples which were filtered more strictly to ensure a higher quality match between Spotify tracks and YouTube videos.

To cite our work, please refer to our paper [here](https://arxiv.org/abs/2306.13512).
<!--
### Curation Rationale

[More Information Needed]

### Source Data

[More Information Needed]
-->