<jupyter_start><jupyter_text>Audio t-SNEThis notebook will show you how to create a t-SNE plot of a group of audio clips. Along the way, we'll cover a few basic audio processing and machine learning tasks.We will make two separate t-SNE plots. The first is clustering a group of many audio files in a single directory. The second one takes only a single audio track (a song) as it's input, segments it into many audio chunks (and saves them to a directory) and clusters the resulting chunks.This notebook requires numpy, matplotlib, scikit-learn, and [librosa](https://github.com/librosa/librosa) to run. To install librosa, run the following command in the terminal:    pip install librosaAfter verifying you have the required libraries, verify the following import commands work.<jupyter_code>%matplotlib inline
from matplotlib import pyplot as plt
import matplotlib.cm as cm
import fnmatch
import os
import numpy as np
import librosa
import matplotlib.pyplot as plt
import librosa.display
from sklearn.manifold import TSNE
import json<jupyter_output><empty_output><jupyter_text>First we need to scan some directory of audio files and collect all their paths into a single list. This notebook is using a free sample pack called "Vintage drum machines" which can be downloaded, along with all the other data needed for ml4a-guides, by running the script `download.sh` in the `data` folder, or downloaded and unzipped manually from [http://ivcloud.de/index.php/s/QyDXk1EDYDTVYkF](http://ivcloud.de/index.php/s/QyDXk1EDYDTVYkF).Once you've done that, or changed the `path` variable to another directory of audio samples on your computer, you can proceed with the next code block to load all the filepaths into memory.<jupyter_code>path = '../data/Vintage Drum Machines'

files = []
for root, dirnames, filenames in os.walk(path):
    for filename in fnmatch.filter(filenames, '*.wav'):
        files.append(os.path.join(root, filename))

print("found %d .wav files in %s"%(len(files),path))<jupyter_output>found 6153 .wav files in ../data/Vintage Drum Machines<jupyter_text>In the next block, we're going to create a function which extracts a feature vector from an audio file. We are using [librosa](https://github.com/librosa/librosa/), a python library for audio analysis and music information retrieval, to handle the feature extraction.The function we're creating `get_features` will take a waveform `y` at a sample rate `sr`, and extract features for only the first second of the audio. It is possible to use longer samples, but because we are interested in clustering according to sonic similarity, we choose to focus on short samples with relatively homogenous content over their durations. Longer samples have different sections and would require somewhat more sophisticated feature extraction.The feature extraction will calculate the first 13 [mel-frequency cepstral coefficients](https://en.wikipedia.org/wiki/Mel-frequency_cepstrum) of the audio file, as well as their first- and second-order derivatives, and concatenate them into a single 39-element feature vector. The feature vector is also [standardized](https://en.wikipedia.org/wiki/Standard_score) so that each feature has equal variance.<jupyter_code>def get_features(y, sr):
    y = y[0:sr]  # analyze just first second
    S = librosa.feature.melspectrogram(y, sr=sr, n_mels=128)
    log_S = librosa.amplitude_to_db(S, ref=np.max)
    mfcc = librosa.feature.mfcc(S=log_S, n_mfcc=13)
    delta_mfcc = librosa.feature.delta(mfcc, mode='nearest')
    delta2_mfcc = librosa.feature.delta(mfcc, order=2, mode='nearest')
    feature_vector = np.concatenate((np.mean(mfcc,1), np.mean(delta_mfcc,1), np.mean(delta2_mfcc,1)))
    feature_vector = (feature_vector-np.mean(feature_vector)) / np.std(feature_vector)
    return feature_vector<jupyter_output><empty_output><jupyter_text>Now we will iterate through all the files, and get their feature vectors, placing them into a new list `feature_vectors`. We also make a new array `sound_paths` to index the feature vectors to the correct paths, in case some of the files are empty or corrupted (as they are in the `Vintage Drum Machines` sample pack), or otherwise something goes wrong in analyzing them.<jupyter_code>feature_vectors = []
sound_paths = []
for i,f in enumerate(files):
    if i % 100 == 0:
        print("get %d of %d = %s"%(i+1, len(files), f))
    try:
        y, sr = librosa.load(f)
        if len(y) < 2:
            print("error loading %s" % f)
            continue
        feat = get_features(y, sr)
        feature_vectors.append(feat)
        sound_paths.append(f)
    except:
        print("error loading %s" % f)
        
print("calculated %d feature vectors"%len(feature_vectors))<jupyter_output>get 1 of 6153 = ../data/Vintage Drum Machines/Ace Tone Rhythm Ace/CLAVE.wav
get 101 of 6153 = ../data/Vintage Drum Machines/Akai  XR-10/Snaredrums/Snaredrum-09.wav
get 201 of 6153 = ../data/Vintage Drum Machines/Akai XR10/XR10Rimshot01.wav
get 301 of 6153 = ../data/Vintage Drum Machines/Alesis DM5/DM5Hat_C07.wav
get 401 of 6153 = ../data/Vintage Drum Machines/Alesis DM5/DM5Kick89.wav
get 501 of 6153 = ../data/Vintage Drum Machines/Alesis DM5/DM5Perc094.wav
get 601 of 6153 = ../data/Vintage Drum Machines/Alesis DM5/DM5Snare072.wav
get 701 of 6153 = ../data/Vintage Drum Machines/Alesis HR-16/Agogo Bell.wav
get 801 of 6153 = ../data/Vintage Drum Machines/Alesis HR16B/HR16B 038 Bottles.wav
error loading ../data/Vintage Drum Machines/Alesis SR15/sr16_hats22/SR16 Hat_Closed.wav
error loading ../data/Vintage Drum Machines/Alesis SR15/sr16_hats22/SR16 Hat_Small.wav
error loading ../data/Vintage Drum Machines/Alesis SR15/sr16_hats22/SR16 Hat_Tight.wav
error loading ../data/Vintage Drum Machines[...]<jupyter_text>Now we can run t-SNE over the feature vectors to get a 2-dimensional embedding of our audio files. We use scikit-learn's TSNE function, and additionally normalize the results so that they are between 0 and 1.<jupyter_code>model = TSNE(n_components=2, learning_rate=150, perplexity=30, verbose=2, angle=0.1).fit_transform(feature_vectors)<jupyter_output>[t-SNE] Computing pairwise distances...
[t-SNE] Computing 91 nearest neighbors...
[t-SNE] Computed conditional probabilities for sample 1000 / 6091
[t-SNE] Computed conditional probabilities for sample 2000 / 6091
[t-SNE] Computed conditional probabilities for sample 3000 / 6091
[t-SNE] Computed conditional probabilities for sample 4000 / 6091
[t-SNE] Computed conditional probabilities for sample 5000 / 6091
[t-SNE] Computed conditional probabilities for sample 6000 / 6091
[t-SNE] Computed conditional probabilities for sample 6091 / 6091
[t-SNE] Mean sigma: 0.121914
[t-SNE] Iteration 25: error = 1.7862769, gradient norm = 0.0000158
[t-SNE] Iteration 25: gradient norm 0.000016. Finished.
[t-SNE] Iteration 50: error = 1.7862240, gradient norm = 0.0006299
[t-SNE] Iteration 50: gradient norm 0.000630. Finished.
[t-SNE] KL divergence after 50 iterations with early exaggeration: 1.786224
[t-SNE] Iteration 75: error = 1.7804098, gradient norm = 0.0110668
[t-SNE] Iteration 100: error = 1.76655[...]<jupyter_text>Let's plot our t-SNE points.  We can use matplotlib to quickly scatter them and see their distribution.<jupyter_code>x_axis=model[:,0]
y_axis=model[:,1]

plt.figure(figsize = (10,10))
plt.scatter(x_axis, y_axis)
plt.show()<jupyter_output><empty_output><jupyter_text>We see our t-SNE plot of our audio files, but it's not particularly interesting! Since we are dealing with audio files, there's no easy way to compare neighboring audio samples to each other. We can use some other, more interactive environment to view the results of the t-SNE. One way we can do this is by saving the results to a JSON file which stores the filepaths and t-SNE assignments of all the audio files. We can then load this JSON file in another environment. One example of this is provided in the "AudioTSNEViewer" application in [ml4a-ofx](https://github.com/ml4a/ml4a-ofx/tree/master/apps). This is an [openFrameworks](http://openframeworks.cc) application which loads all the audio clips into an interactive 2d grid (using the t-SNE layout) and lets you play each sample by hovering your mouse over it.  In any case, to save the t-SNE to a JSON file, we first normalize the coordinates to between 0 and 1 and save them, along with the full filepaths.<jupyter_code>tsne_path = "../data/example-audio-tSNE.json"

x_norm = (x_axis - np.min(x_axis)) / (np.max(x_axis) - np.min(x_axis))
y_norm = (y_axis - np.min(y_axis)) / (np.max(y_axis) - np.min(y_axis))

data = [{"path":os.path.abspath(f), "point":[x, y]} for f, x, y in zip(sound_paths, x_norm, y_norm)]
with open(tsne_path, 'w') as outfile:
    json.dump(data, outfile, cls=MyEncoder)

print("saved %s to disk!" % tsne_path)<jupyter_output>saved ../data/example-audio-tSNE.json to disk!<jupyter_text>Now what about if we want to do this same analysis, but instead of analyzing a directory of individual audio clips, we want to cut up a _single audio file_ into many chunks and cluster those instead. We can add a few extra lines of code to the above in order to do this. First let's select a piece of audio. Find any song on your computer that you want to do this analysis to and set a path to it.<jupyter_code>source_audio = '/Users/gene/Downloads/bohemianrhapsody.mp3'<jupyter_output><empty_output><jupyter_text>What we will now do is use librosa to calculate the _onsets_ of our audio file. Onsets are the timestamps to the beginning of discrete sonic events in our audio. We set the hop_length (number of samples for each frame) to 512.<jupyter_code>hop_length = 512
y, sr = librosa.load(source_audio)
onsets = librosa.onset.onset_detect(y=y, sr=sr, hop_length=hop_length)<jupyter_output><empty_output><jupyter_text>How do we interpret these numbers?  Initially our original audio, at a sample rate `sr` is divided up into bins which each cointain 512 samples (specified by `hop_length` in the onset detection function). The onset numbers are an index to each of these bins. So for example, if the first onset is 20, this corresponds to sample 20 * 512 = 10,240. Given a sample rate of 22050, this corresponds to 10250/22050 = 0.46 seconds into the track.We can view the onsets as vertical lines on top of the waveform using matplotlib and the following code.<jupyter_code>times = [hop_length * onset / sr for onset in onsets]

plt.figure(figsize=(16,4))
plt.subplot(1, 1, 1)
librosa.display.waveplot(y, sr=sr)
plt.vlines(times, -1, 1, color='r', alpha=0.9, label='Onsets')
plt.title('Wavefile with %d onsets plotted' % len(times))<jupyter_output><empty_output><jupyter_text>Now what we will do is, we will go through each of the detected onsets, and crop the original audio to the interval from that onset until the next one. We will create a new folder, `../data/audio_clips`, into which we will then save all of the individual audio clips, and extract the same feature vector we described above for our new audio sample.<jupyter_code># where to save our new clips to
path_save_intervals = "/Users/gene/Downloads/bohemian_segments/"

# make new directory to save them 
if not os.path.isdir(path_save_intervals):
    os.mkdir(path_save_intervals)

# grab each interval, extract a feature vector, and save the new clip to our above path
feature_vectors = []
for i in range(len(onsets)-1):
    idx_y1 = onsets[i  ] * hop_length  # first sample of the interval
    idx_y2 = onsets[i+1] * hop_length  # last sample of the interval
    y_interval = y[idx_y1:idx_y2]
    features = get_features(y_interval, sr)   # get feature vector for the audio clip between y1 and y2
    file_path = '%s/onset_%d.wav' % (path_save_intervals, i)   # where to save our new audio clip
    feature_vectors.append({"file":file_path, "features":features})   # append to a feature vector
    librosa.output.write_wav(file_path, y_interval, sr)   # save to disk
    if i % 50 == 0:
        print "analyzed %d/%d = %s"%(i+1, len(onsets)-1, file_path)

# save results to this json file
tsne_path = "../data/example-audio-tSNE-onsets.json"

# feature_vectors has both the features and file paths in it. let's pull out just the feature vectors
features_matrix = [f["features"] for f in feature_vectors]

# calculate a t-SNE and normalize it
model = TSNE(n_components=2, learning_rate=150, perplexity=30, verbose=2, angle=0.1).fit_transform(features_matrix)
x_axis, y_axis = model[:,0], model[:,1] # normalize t-SNE
x_norm = (x_axis - np.min(x_axis)) / (np.max(x_axis) - np.min(x_axis))
y_norm = (y_axis - np.min(y_axis)) / (np.max(y_axis) - np.min(y_axis))

data = [{"path":os.path.abspath(f['file']), "point":[float(x), float(y)]} for f, x, y in zip(feature_vectors, x_norm, y_norm)]
with open(tsne_path, 'w') as outfile:
    json.dump(data, outfile)

print("saved %s to disk!" % tsne_path)<jupyter_output>[t-SNE] Computing pairwise distances...
[t-SNE] Computing 91 nearest neighbors...
[t-SNE] Computed conditional probabilities for sample 784 / 784
[t-SNE] Mean sigma: 0.231958
[t-SNE] Iteration 25: error = 2.3879926, gradient norm = 0.0265828
[t-SNE] Iteration 50: error = 1.9338583, gradient norm = 0.0146562
[t-SNE] Iteration 75: error = 1.2881635, gradient norm = 0.0044903
[t-SNE] Iteration 100: error = 1.2138703, gradient norm = 0.0037113
[t-SNE] KL divergence after 100 iterations with early exaggeration: 1.213870
[t-SNE] Iteration 125: error = 1.1115109, gradient norm = 0.0026617
[t-SNE] Iteration 150: error = 1.0837917, gradient norm = 0.0024060
[t-SNE] Iteration 175: error = 1.0771828, gradient norm = 0.0023484
[t-SNE] Iteration 200: error = 1.0754088, gradient norm = 0.0023334
[t-SNE] Iteration 225: error = 1.0749190, gradient norm = 0.0023294
[t-SNE] Iteration 250: error = 1.0747885, gradient norm = 0.0023282
[t-SNE] Iteration 275: error = 1.0747490, gradient norm = 0.0023279
[t-[...]<jupyter_text>Let's plot the results on another scatter plot. One nice thing we can do is color the points according to their order in the original track.<jupyter_code>colors = cm.rainbow(np.linspace(0, 1, len(x_axis)))
plt.figure(figsize = (8,6))
plt.scatter(x_axis, y_axis, color=colors)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0969812385385809, 'fraction_numerical': 0.04506982649174778, 'mean_word_length': 4.613222486144101, 'pattern_counts': {'":': 6, '<': 45, '<?xml version=': 0, '>': 44, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '321278', 'license': 'MIT', 'n_tokens_mistral': 5147, 'n_tokens_neox': 4538, 'n_words': 1693, 'path': 'examples/info_retrieval/audio-tsne.ipynb', 'repo_name': 'yo252yo/ml4a', 'size': '14178'}