<jupyter_start><jupyter_text>GTZAN: convert to HDF5 format The GTZAN dataset comes in a ZIP archive wich contains one folder per musical genre. Each folder contains all the clips who belong to the genre. While easily browsable, this format is not appropriate for data analysis. We thus first convert the dataset to HDF5 before any pre-processing step.  Dataset definition * 10 genres: blues, classical, country, disco, hiphop, jazz, metal, pop, reggae and rock* 100 clips per musical genre* each clip lasts 30 seconds* audio sampled at 22050 Hz<jupyter_code>class gtzan:
    Ngenres = 10
    Nclips = 100
    sr = 22050
    # Size ranges from 660000 to 675808 samples per clip.
    # We will truncate all clips to not bias the classifier in any way.
    Nsamples = 660000

print('Clip duration: {0:.2f} seconds ({1} samples at {2} Hz)'.format(
        float(gtzan.Nsamples)/gtzan.sr, gtzan.Nsamples, gtzan.sr))<jupyter_output><empty_output><jupyter_text>Setup<jupyter_code>import os, time
import numpy as np
import librosa
import h5py
import IPython.display

print('Software versions:')
for pkg in [np, librosa]:
    print('  {}: {}'.format(pkg.__name__, pkg.__version__))<jupyter_output><empty_output><jupyter_text>Data loading<jupyter_code>folder = os.path.join('data', 'genres')
genres = os.listdir(folder)
assert len(genres) == gtzan.Ngenres<jupyter_output><empty_output><jupyter_text>Define some helper functions.<jupyter_code>def genpath(genre, clip):
    """Usage: genpath('rock', 0)"""
    return os.path.join(folder, genre, '{0}.{1:0>5d}.au'.format(genre, clip))

def read(genre, clip):
    """Usage: read('rock', 0)"""

    # Load audio file.
    path = genpath(genre, clip)
    y, sr = librosa.load(path, sr=None, mono=False)  # No resampling.

    # Sanity checks.
    if sr != gtzan.sr:
        raise ValueError('{}: wrong sampling rate of {}'.format(path, sr))
    if y.size < gtzan.Nsamples:
        raise ValueError('{}: too short, {} samples'.format(path, y.size))

    # Truncate.
    return y[:gtzan.Nsamples]<jupyter_output><empty_output><jupyter_text>HDF5 data store:* One dataset per musical genre.* Each dataset contains the whole audio data of the genre.* Each column of a dataset is an entire clip.* There is as many rows as samples per clip.<jupyter_code>filename = os.path.join('data', 'gtzan.hdf5')

# Remove existing HDF5 file without warning if non-existent.
try:
    os.remove(filename)
except OSError:
    pass

# Create HDF5 file and datasets.
f = h5py.File(filename, 'w')
dsize = (gtzan.Nsamples, gtzan.Nclips)
for genre in genres:
    _ = f.create_dataset(genre, dsize, dtype='float32')<jupyter_output><empty_output><jupyter_text>Fill the datasets with actual audio data.<jupyter_code>tstart = time.time()
for genre in genres:
    for clip in range(gtzan.Nclips):
        f[genre][:,clip] = read(genre, clip)
print('Elapsed time: {:.0f} seconds'.format(time.time() - tstart))<jupyter_output><empty_output><jupyter_text>Store the GTZAN metadata along with the audio data.<jupyter_code>for var in vars(gtzan):
    if not var.startswith("__"):
        f.attrs[var] = vars(gtzan)[var]

# Display HDF5 attributes.
print('Attributes:')
for attr in f.attrs:
    print('  {} = {}'.format(attr, f.attrs[attr]))<jupyter_output><empty_output><jupyter_text>Sanity check * Check that the content stored in HDF5 is similar to the audio file.* Test on 10 randomly chosen clips.<jupyter_code>for _ in range(10):
    genre = int(np.floor(gtzan.Ngenres * np.random.uniform()))
    genre = genres[genre]
    clip = int(np.floor(gtzan.Nclips * np.random.uniform()))
    assert np.alltrue(f[genre][:,clip] == read(genre, clip))<jupyter_output><empty_output><jupyter_text>Listen to a clip From the HDF5 data store.<jupyter_code>genre, clip = 'pop', 20
#IPython.display.Audio(f[genre][:,clip], rate=gtzan.sr)<jupyter_output><empty_output><jupyter_text>Via librosa.<jupyter_code>IPython.display.Audio(read(genre, clip), rate=gtzan.sr)<jupyter_output><empty_output><jupyter_text>Directly from the file. It does unfortunately not support .au audio files.<jupyter_code>#IPython.display.Audio(os.path.abspath(genpath(genre, clip)))<jupyter_output><empty_output><jupyter_text>Close HDF5 data store<jupyter_code>f.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12734962406015038, 'fraction_numerical': 0.01668233082706767, 'mean_word_length': 5.072753209700428, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 50, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '658115', 'license': 'MIT', 'n_tokens_mistral': 1555, 'n_tokens_neox': 1472, 'n_words': 397, 'path': 'gtzan.ipynb', 'repo_name': 'mdeff/dlaudio', 'size': '4256'}