<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# FMA: A Dataset For Music Analysis\n",
    "\n",
    "<NAME>, <NAME>, <NAME>, <NAME>, EPFL LTS2.\n",
    "\n",
    "## Generation / Collection / Creation\n",
    "\n",
    "From `raw_*.csv`, this notebook generates:\n",
    "* `tracks.csv`: per-track / album / artist metadata.\n",
    "* `genres.csv`: genre hierarchy.\n",
    "\n",
    "A companion script, [creation.py](creation.py):\n",
    "1. Query the API and store metadata in `raw_tracks.csv`, `raw_albums.csv`, `raw_artists.csv` and `raw_genres.csv`.\n",
    "2. Download the audio for each track.\n",
    "3. Trim the audio to 30s clips.\n",
    "4. Normalize the permissions and modification / access times.\n",
    "5. Create the `.zip` archives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import os\n",
    "import ast\n",
    "import pickle\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import IPython.display as ipd\n",
    "import utils\n",
    "import creation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "AUDIO_DIR = os.environ.get('AUDIO_DIR')\n",
    "BASE_DIR = os.path.abspath(os.path.dirname(AUDIO_DIR))\n",
    "FMA_FULL = os.path.join(BASE_DIR, 'fma_full')\n",
    "FMA_LARGE = os.path.join(BASE_DIR, 'fma_large')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 1 Retrieve metadata and audio from FMA\n",
    "\n",
    "1. Crawl the tracks, albums and artists metadata through their [API](https://freemusicarchive.org/api).\n",
    "2. Download original `.mp3` by HTTPS for each track id (only if it does not exist already).\n",
    "\n",
    "Todo:\n",
    "* Scrap curators.\n",
    "* Download images (`track_image_file`, `album_image_file`, `artist_image_file`). Beware the quality.\n",
    "* Verify checksum for some random tracks.\n",
    "\n",
    "Examples:\n",
    "* To add new tracks: iterate from largest known track id to the most recent only.\n",
    "* To update user data: get them all again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# ./creation.py metadata\n",
    "# ./creation.py data /path/to/fma/fma_full\n",
    "# ./creation.py clips /path/to/fma\n",
    "\n",
    "#!cat creation.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# converters={'genres': ast.literal_eval}\n",
    "tracks = pd.read_csv('raw_tracks.csv', index_col=0)\n",
    "albums = pd.read_csv('raw_albums.csv', index_col=0)\n",
    "artists = pd.read_csv('raw_artists.csv', index_col=0)\n",
    "genres = pd.read_csv('raw_genres.csv', index_col=0)\n",
    "\n",
    "not_found = pickle.load(open('not_found.pickle', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def get_fs_tids(audio_dir):\n",
    "    tids = []\n",
    "    for _, dirnames, files in os.walk(audio_dir):\n",
    "        if dirnames == []:\n",
    "            tids.extend(int(file[:-4]) for file in files)\n",
    "    return tids\n",
    "\n",
    "audio_tids = get_fs_tids(FMA_FULL)\n",
    "clips_tids = get_fs_tids(FMA_LARGE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "tmp = len(tracks), len(not_found['tracks']), tracks.index.max()\n",
    "print('tracks: {} collected ({} not found, {} max id)'.format(*tmp))\n",
    "tmp = len(albums), len(not_found['albums']), len(tracks['album_id'].unique())\n",
    "print('albums: {} collected ({} not found, {} in tracks)'.format(*tmp))\n",
    "tmp = len(artists), len(not_found['artists']), len(tracks['artist_id'].unique())\n",
    "print('artists: {} collected ({} not found, {} in tracks)'.format(*tmp))\n",
    "print('genres: {} collected'.format(len(genres)))\n",
    "print('audio: {} collected ({} not found)'.format(len(audio_tids), len(not_found['audio'])))\n",
    "print('clips: {} collected ({} not found)'.format(len(clips_tids), len(not_found['clips'])))\n",
    "assert sum(tracks.index.isin(audio_tids)) + len(not_found['audio']) == len(tracks)\n",
    "assert sum(tracks.index.isin(clips_tids)) + len(not_found['clips']) == sum(tracks.index.isin(audio_tids))\n",
    "assert len(clips_tids) + len(not_found['clips']) + len(not_found['audio']) == len(tracks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "N = 5\n",
    "ipd.display(tracks.head(N))\n",
    "ipd.display(albums.head(N))\n",
    "ipd.display(artists.head(N))\n",
    "ipd.display(genres.head(N))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true<commit_msg>creation: show number of additional audio<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# FMA: A Dataset For Music Analysis\n",
    "\n",
    "<NAME>, <NAME>, <NAME>, <NAME>, EPFL LTS2.\n",
    "\n",
    "## Generation / Collection / Creation\n",
    "\n",
    "From `raw_*.csv`, this notebook generates:\n",
    "* `tracks.csv`: per-track / album / artist metadata.\n",
    "* `genres.csv`: genre hierarchy.\n",
    "\n",
    "A companion script, [creation.py](creation.py):\n",
    "1. Query the API and store metadata in `raw_tracks.csv`, `raw_albums.csv`, `raw_artists.csv` and `raw_genres.csv`.\n",
    "2. Download the audio for each track.\n",
    "3. Trim the audio to 30s clips.\n",
    "4. Normalize the permissions and modification / access times.\n",
    "5. Create the `.zip` archives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import os\n",
    "import ast\n",
    "import pickle\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import IPython.display as ipd\n",
    "import utils\n",
    "import creation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "AUDIO_DIR = os.environ.get('AUDIO_DIR')\n",
    "BASE_DIR = os.path.abspath(os.path.dirname(AUDIO_DIR))\n",
    "FMA_FULL = os.path.join(BASE_DIR, 'fma_full')\n",
    "FMA_LARGE = os.path.join(BASE_DIR, 'fma_large')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 1 Retrieve metadata and audio from FMA\n",
    "\n",
    "1. Crawl the tracks, albums and artists metadata through their [API](https://freemusicarchive.org/api).\n",
    "2. Download original `.mp3` by HTTPS for each track id (only if it does not exist already).\n",
    "\n",
    "Todo:\n",
    "* Scrap curators.\n",
    "* Download images (`track_image_file`, `album_image_file`, `artist_image_file`). Beware the quality.\n",
    "* Verify checksum for some random tracks.\n",
    "\n",
    "Examples:\n",
    "* To add new tracks: iterate from largest known track id to the most recent only.\n",
    "* To update user data: get them all again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# ./creation.py metadata\n",
    "# ./creation.py data /path/to/fma/fma_full\n",
    "# ./creation.py clips /path/to/fma\n",
    "\n",
    "#!cat creation.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# converters={'genres': ast.literal_eval}\n",
    "tracks = pd.read_csv('raw_tracks.csv', index_col=0)\n",
    "albums = pd.read_csv('raw_albums.csv', index_col=0)\n",
    "artists = pd.read_csv('raw_artists.csv', index_col=0)\n",
    "genres = pd.read_csv('raw_genres.csv', index_col=0)\n",
    "\n",
    "not_found = pickle.load(open('not_found.pickle', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def get_fs_tids(audio_dir):\n",
    "    tids = []\n",
    "    for _, dirnames, files in os.walk(audio_dir):\n",
    "        if dirnames == []:\n",
    "            tids.extend(int(file[:-4]) for file in files)\n",
    "    return tids\n",
    "\n",
    "audio_tids = get_fs_tids(FMA_FULL)\n",
    "clips_tids = get_fs_tids(FMA_LARGE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('tracks: {} collected ({} not found, {} max id)'.format(\n",
    "    len(tracks), len(not_found['tracks']), tracks.index.max()))\n",
    "print('albums: {} collected ({} not found, {} in tracks)'.format(\n",
    "    len(albums), len(not_found['albums']), len(tracks['album_id'].unique())))\n",
    "print('artists: {} collected ({} not found, {} in tracks)'.format(\n",
    "    len(artists), len(not_found['artists']), len(tracks['artist_id'].unique())))\n",
    "print('genres: {} collected'.format(len(genres)))\n",
    "print('audio: {} collected ({} not found, {} not in tracks)'.format(\n",
    "    len(audio_tids), len(not_found['audio']), len(set(audio_tids).difference(tracks.index))))\n",
    "print('clips: {} collected ({} not found, {} not in tracks)'.format(\n",
    "    len(clips_tids), len(not_found['clips']), len(set(clips_tids).difference(tracks.index))))\n",
    "assert sum(tracks.index.isin(audio_tids)) + len(not_found['audio']) == len(tracks)\n",
    "assert sum(tracks.index.isin(clips_tids)) + len(not_found['clips']) == sum(tracks.index.isin(audio_tids))\n",
    "assert len(clips_tids) + len(not_found['clips']) + len(not_found['audio']) == len(tracks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "N = 5\n",
    "ipd.display(tracks.head(N))\n",
    "ipd.display(albums.head(N))\n",
    "ipd.display(artists.head(N))\n",
    "ipd.display(genres.head(N))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
{'fraction_non_alphanumeric': 0.21021439776886874, 'fraction_numerical': 0.0033118354540700714, 'mean_word_length': 3.0010460251046025, 'pattern_counts': {'":': 141, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '566733', 'n_tokens_mistral': 4418, 'n_tokens_neox': 4335, 'n_words': 853}