<jupyter_start><jupyter_text>Music vs. Speech Classification with Deep LearningThis tutorial shows how different Convolutional Neural Network architectures are used for the task of discriminating a piece of audio whether it is music or speech (binary classification).The data set used is the [Music Speech data set](http://marsyas.info/downloads/datasets.html) compiled by <NAME>. It consists of 128 tracks, each 30 seconds long. Each class (music/speech) has 64 examples. The tracks are all 22050Hz Mono 16-bit audio files in .wav format.  Table of ContentsThis tutorial contains:* Loading and preprocessing of audio files* Loading class files from CSV and using Label Encoder* Audio Preprocessing: Generating log Mel spectrograms* Standardization of Data* Convolutional Neural Networks: single, stacked, parallel* ReLU Activation* Dropout* Train/Test set splitYou can execute the following code blocks by pressing SHIFT+Enter consecutively.  Download Data**On the lab machine this data is already prepared, no need to download.**The data set can be downloaded from [here](http://opihi.cs.uvic.ca/sound/music_speech.tar.gz). Please unpack it (it is .tar.gz compressed)Set the path to the unpacked folder in the next box:<jupyter_code># if you use the iDSDL Lab machine
DATA_PATH = '/home/ffg/idsdlmteacher1/Code/DeepLearningTutorial_2019/data/Music_speech'
# if you downloaded the dataset into the tutorial folder (otherwise adjust the DATA_PATH please)
# DATA_PATH = './Music_speech'
# if you want to use Multi-GPU set this before the import of keras or tensorflow:
# import os
# os.environ["CUDA_VISIBLE_DEVICES"] = "2"
# General Imports
import argparse
import csv
import datetime
import glob
import librosa  # for audio loading and preprocessing
import math
import numpy as np
import os
import pandas as pd  # for reading CSV files and easier metadata preparation
import sys
import time

# Deep Learning
from keras.backend import floatx  # abstract default float type for Keras (e.g. float32)
from keras.models import Sequential, Model
from keras.layers import Input, Conv2D, MaxPooling2D, Dense, Dropout, Activation, Flatten, merge
from keras.layers.merge import concatenate
from keras.layers.normalization import BatchNormalization

from sklearn import preprocessing
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.model_selection import StratifiedShuffleSplit<jupyter_output>Using TensorFlow backend.<jupyter_text>Load the MetadataThe tab-separated file contains pairs of filename TAB class.<jupyter_code>csv_file = os.path.join(DATA_PATH,'filelist_wclasses.txt')
metadata = pd.read_csv(csv_file, index_col=0, sep='\t', header=None)
metadata.head(10)
# create two lists: one with filenames and one with associated classes
filelist = metadata.index.tolist()
classes = metadata[1].values.tolist()<jupyter_output><empty_output><jupyter_text>Encode Labels to NumbersString labels need to be encoded as numbers. We use the LabelEncoder from the scikit-learn package.<jupyter_code>classes[0:5]
from sklearn.preprocessing import LabelEncoder

labelencoder = LabelEncoder()
labelencoder.fit(classes)
print(len(labelencoder.classes_), "classes:", ", ".join(list(labelencoder.classes_)))

classes_num = labelencoder.transform(classes)
# check first few classes (speech)
classes_num[0:5]
# check first few classes (music)
classes_num[-5:]<jupyter_output><empty_output><jupyter_text>Note: In order to correctly re-transform any predicted numbers into strings, we keep the labelencoder for later.  Load the Audio FilesThe audio is stored as .WAV files. We can load them directly. Otherwise tools like ffmpeg can help to decode. librosa.load supports the use of ffmpeg.Audio files are usually preprocessed the following way before using them as input to a CNN:* Conversion to Mono* Cut a segment (or multiple, to create more instances)* Compute a Spectrograme using FFT* Convert to Mel scale (a psycho-acoustically motivated perceptual scale, also useful for dimension reduction)* Convert to Log (or Decibel) scaleThe FFT parameters as well as the desired number of output frames and Mel bands can be set below:<jupyter_code># SETTINGS
AUDIO_PATH = DATA_PATH

# some FFT parameters
fft_window_size = 512
fft_overlap = 0.5  # 50% overlap between successive frames

# desired spectrogram parameters as input for the CNN
frames = 80        # x axis
n_mel_bands = 40   # y axis
list_spectrograms = [] # spectrograms are put into a list first

# FFT parameters
frames = frames - 2  # for librosa, which otherwise creates 2 frames more than needed
hop_size = int(fft_window_size*(1-fft_overlap))
segment_size = fft_window_size + (frames-1) * hop_size  # segment size for desired # of frames

for filename in filelist:
    print(".", end='')
    
    # 1) load audio as waveform using librosa
    audiofile = os.path.join(AUDIO_PATH, filename)
    # auto-convert to mono. to preserve the native sampling rate of the file use sr=None.
    wave_data, samplerate = librosa.load(audiofile, mono=True, sr=None)
    
    # 2) use only a segment; choose start position:
    #pos = 0  # beginning
    pos = int(wave_data.shape[0]/2 - segment_size/2)  # center minus half segment size
    wav_segment = wave_data[pos:pos+segment_size]

    # 3) FFT + Mel-transform in a single step  
    spectrogram = librosa.feature.melspectrogram(y=wav_segment, sr=samplerate, 
                                                 n_fft=fft_window_size, hop_length=hop_size,
                                                 n_mels=n_mel_bands, fmax=None)
    # Note: one may want to limit the maximum frequency in the output spectrogram using the fmax parameter
    
    # 4) Log 10 transform
    spectrogram = np.log10(spectrogram)

    list_spectrograms.append(spectrogram)
        
print("\nRead", len(filelist), "audio files")
len(list_spectrograms)
wave_data.shape
wav_segment.shape
segment_size
spectrogram.shape
# duration of the segment in seconds
wav_segment.shape[0] / samplerate
print("An audio segment is", round(float(segment_size) / samplerate, 2), "seconds long")<jupyter_output>An audio segment is 0.92 seconds long<jupyter_text>Note: For simplicity of this tutorial, here we load only 1 single segment of ~ 1 second length from each audio file.In a real setting, one would create training instances of as many audio segments as possible to be fed to a Neural Network. Show Waveform and Spectrogram<jupyter_code># you can skip this if you do not have matplotlib installed

import matplotlib.pyplot as plt
%matplotlib inline 

# show 1 sec wave segment
plt.plot(wav_segment)
filename
# show Mel spectrogram
fig = plt.imshow(spectrogram, origin='lower')
fig.set_cmap('jet')
fig.axes.get_xaxis().set_visible(False)
fig.axes.get_yaxis().set_visible(False)<jupyter_output><empty_output><jupyter_text>Make 1 big array of list of spectrograms<jupyter_code># dtype: convert the input data to the right data type used by Keras Deep Learning (GPU) (float32)
dtype = floatx()
dtype
# a list of many 40x80 spectrograms is made into 1 big array
data = np.array(list_spectrograms, dtype=dtype)
data_shape = data.shape
data_shape<jupyter_output><empty_output><jupyter_text>StandardizationAlways standardize the data before feeding it into the Neural Network!We use Zero-mean Unit-variance standardization (also known as Z-score normalization).Here, we use attribute-wise standardization, i.e. each pixel is standardized individually, as opposed to computing a single mean and single standard deviation of all values.('Flat' standardization would also be possible, but we have seen benefits of attribut-wise standardization in our experiments).We use the StandardScaler from the scikit-learn package for our purpose.As it works typically on vector data, we have to vectorize (i.e. reshape) our matrices first. After standardization we have to reshape back to the original format.<jupyter_code># vectorize
N, ydim, xdim = data.shape
data = data.reshape(N, xdim*ydim)
data.shape
# standardize
scaler = preprocessing.StandardScaler()
data = scaler.fit_transform(data)
# show mean and standard deviation: two vectors with same length as data.shape[1]
scaler.mean_, scaler.scale_
# reshape back to the original format
data = data.reshape(data_shape)<jupyter_output><empty_output><jupyter_text>Creating Train & Test Set We split the original full data set into two parts: Train Set (75%) and Test Set (25%).Here we compare Random Split vs. Stratified Split:<jupyter_code>testset_size = 0.25 # % portion of whole data set to keep for testing, i.e. 75% is used for training

# Normal (random) split of data set into 2 parts
# from sklearn.model_selection import train_test_split

train_set, test_set, train_classes, test_classes = train_test_split(data, classes_num, test_size=testset_size, random_state=0)
train_classes
test_classes
# The two classes may be unbalanced
print("Class Counts: Class 0:", sum(train_classes==0), "Class 1:", sum(train_classes))
# better: Stratified Split retains the class balance in both sets
# from sklearn.model_selection import StratifiedShuffleSplit

splitter = StratifiedShuffleSplit(n_splits=1, test_size=testset_size, random_state=0)
splits = splitter.split(data, classes_num)

for train_index, test_index in splits:
    print("TRAIN INDEX:", train_index)
    print("TEST INDEX:", test_index)
    train_set = data[train_index]
    test_set = data[test_index]
    train_classes = classes_num[train_index]
    test_classes = classes_num[test_index]
# Note: this for loop is only executed once, if n_iter==1 resp. n_splits==1
print(train_set.shape)
print(test_set.shape)
# Note: we will reshape the data later back to matrix form 
print("Class Counts: Class 0:", sum(train_classes==0), "Class 1:", sum(train_classes))<jupyter_output>Class Counts: Class 0: 48 Class 1: 48<jupyter_text>Convolutional Neural NetworksA Convolutional Neural Network (ConvNet or CNN) is a type of (deep) Neural Network that is well-suited for 2D axes data, such as images or spectrograms, as it is optimized for learning from spatial proximity. Its core elements are 2D filter kernels which essentially learn the weights of the Neural Network, and downscaling functions such as Max Pooling.A CNN can have one or more Convolution layers, each of them having an arbitrary number of N filters (which define the depth of the CNN layer), following typically by a pooling step, which aggregates neighboring pixels together and thus reduces the image resolution by retaining only the maximum values of neighboring pixels.  Preparing the Data Adding the channelAs CNNs were initially made for image data, we need to add a dimension for the color channel to the data. RGB images typically have a 3rd dimension with the color. Spectrograms, however, are considered like greyscale images, as in the previous tutorial.Likewise we need to add an extra dimension for compatibility with the CNN implementation.<jupyter_code># verify image data format (Tensorflow: channels_last) 
#keras.backend.image_data_format()
train_set.shape
test_set.shape
n_channels = 1
N, height, width = train_set.shape
train_set = train_set.reshape(N, height, width, n_channels)  # Tensorflow ordering: channel last
N, height, width = test_set.shape
test_set = test_set.reshape(N, height, width, n_channels)  # height and width must be the same shape as train_set
train_set.shape
test_set.shape
# we store the new shape of the images in the 'input_shape' variable.
# take all dimensions except the 0th one (which is the number of images)
input_shape = train_set.shape[1:]  
input_shape<jupyter_output><empty_output><jupyter_text>Creating Neural Network Models in Keras Sequential ModelsIn Keras, one can choose between a Sequential model and a Graph model. Sequential models are the standard case. Graph models are for parallel networks.  Creating a Single Layer and a Two Layer CNN Try: (comment/uncomment code in the following code block)* 1 Layer* 2 Layer* more conv_filters* Dropout<jupyter_code>#np.random.seed(0) # make results repeatable

model = Sequential()

#conv_filters = 16   # number of convolution filters (= CNN depth)
conv_filters = 32   # number of convolution filters (= CNN depth)

# Layer 1
model.add(Conv2D(conv_filters, (3, 3), input_shape=input_shape))
model.add(MaxPooling2D(pool_size=(2, 2))) 
model.add(Dropout(0.25)) 

# Layer 2
#model.add(Conv2D(conv_filters, (3, 3)))
#model.add(MaxPooling2D(pool_size=(2, 2))) 

# After Convolution, we have a 16*x*y matrix output
# In order to feed this to a Full(Dense) layer, we need to flatten all data
# Note: Keras does automatic shape inference, i.e. it knows how many (flat) input units the next layer will need,
# so no parameter is needed for the Flatten() layer.
model.add(Flatten()) 

# Full layer
model.add(Dense(256, activation='sigmoid')) 

# Output layer
# For binary/2-class problems use ONE sigmoid unit, 
# for multi-class/multi-label problems use n output units and activation='softmax!'
model.add(Dense(1,activation='sigmoid'))<jupyter_output><empty_output><jupyter_text>If you get OverflowError: Range exceeds valid bounds in the above box, check the correct Theano vs. Tensorflow ordering in the box before and your keras.json configuration file.<jupyter_code>model.summary()<jupyter_output>_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 38, 78, 32)        320       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 19, 39, 32)        0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 19, 39, 32)        0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 23712)             0         
_________________________________________________________________
dense_1 (Dense)              (None, 256)               6070528   
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 257       
=================================================================
Total para[...]<jupyter_text>Training the CNN<jupyter_code># Define a loss function 
loss = 'binary_crossentropy'  # 'categorical_crossentropy' for multi-class problems

# Optimizer = Stochastic Gradient Descent
optimizer = 'sgd' 

# Compiling the model
model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])
# TRAINING the model
epochs = 15
history = model.fit(train_set, train_classes, batch_size=32, epochs=epochs)<jupyter_output>Epoch 1/15
96/96 [==============================] - 1s 8ms/step - loss: 0.7308 - acc: 0.5000
Epoch 2/15
96/96 [==============================] - 0s 1ms/step - loss: 0.5760 - acc: 0.6875
Epoch 3/15
96/96 [==============================] - 0s 1ms/step - loss: 0.5170 - acc: 0.7500
Epoch 4/15
96/96 [==============================] - 0s 1ms/step - loss: 0.4852 - acc: 0.7812
Epoch 5/15
96/96 [==============================] - 0s 1ms/step - loss: 0.4653 - acc: 0.7917
Epoch 6/15
96/96 [==============================] - 0s 996us/step - loss: 0.4423 - acc: 0.7812
Epoch 7/15
96/96 [==============================] - 0s 998us/step - loss: 0.4239 - acc: 0.8021
Epoch 8/15
96/96 [==============================] - 0s 990us/step - loss: 0.4117 - acc: 0.8125
Epoch 9/15
96/96 [==============================] - 0s 1ms/step - loss: 0.3988 - acc: 0.8333
Epoch 10/15
96/96 [==============================] - 0s 1ms/step - loss: 0.3889 - acc: 0.8438
Epoch 11/15
96/96 [==============================] - 0s 995us/s[...]<jupyter_text>Accuracy goes up pretty quickly for 1 layer on Train set! Also on Test set?  Verifying Accuracy on Test Set<jupyter_code># always execute this, and then a box of accuracy_score below to print the result
test_pred = model.predict_classes(test_set)
# 1 layer
accuracy_score(test_classes, test_pred)
# 2 layer
accuracy_score(test_classes, test_pred)
# 2 layer + 32 convolution filters
accuracy_score(test_classes, test_pred)
# 2 layer + 32 convolution filters + Dropout
accuracy_score(test_classes, test_pred)<jupyter_output><empty_output><jupyter_text>Additional Parameters & TechniquesTry out more parameters and techniques: (comment/uncomment code blocks below)* Adding ReLU activation* Adding Batch normalization* Adding Dropout<jupyter_code>model = Sequential()

conv_filters = 16   # number of convolution filters (= CNN depth)
filter_size = (3,3)
pool_size = (2,2)

# Layer 1
model.add(Conv2D(conv_filters, filter_size, padding='valid', input_shape=input_shape))
#model.add(BatchNormalization())
#model.add(Activation('relu')) 
model.add(MaxPooling2D(pool_size=pool_size)) 
#model.add(Dropout(0.3))

# Layer 2
model.add(Conv2D(conv_filters, filter_size, padding='valid', input_shape=input_shape))
#model.add(BatchNormalization())
#model.add(Activation('relu')) 
model.add(MaxPooling2D(pool_size=pool_size)) 
#model.add(Dropout(0.1))

# In order to feed this to a Full(Dense) layer, we need to flatten all data
model.add(Flatten()) 

# Full layer
model.add(Dense(256))  
#model.add(Activation('relu'))
#model.add(Dropout(0.1))

# Output layer
# For binary/2-class problems use ONE sigmoid unit, 
# for multi-class/multi-label problems use n output units and activation='softmax!'
model.add(Dense(1,activation='sigmoid'))
# Compiling and training the model
model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])
epochs = 15
history = model.fit(train_set, train_classes, batch_size=32, epochs=epochs)
# Verifying Accuracy on Test Set
test_pred = model.predict_classes(test_set)
accuracy_score(test_classes, test_pred)<jupyter_output><empty_output><jupyter_text>MIREX 2015 Winning ModelThe following model was the winnner of the [MIREX 2015 Music/Speech Classification](https://www.music-ir.org/mirex/wiki/2015:Music/Speech_Classification_and_Detection_Results) benchmarking competition.(It was trained for 150 epochs on another data set though.)<jupyter_code># MIREX 2015 model
model = Sequential()

conv_filters = 15   # number of convolution filters (= CNN depth)

# Layer 1
model.add(Conv2D(conv_filters, (12, 8), padding='valid', input_shape=input_shape))
#model.add(BatchNormalization())
model.add(Activation('relu')) 
#model.add(Activation('sigmoid'))  # used in original model
model.add(MaxPooling2D(pool_size=(2, 1))) 
#model.add(Dropout(0.3))

# In order to feed this to a Full(Dense) layer, we need to flatten all data
model.add(Flatten()) 

# Full layer
model.add(Dense(200, activation='sigmoid'))  
#model.add(Activation('relu'))
#model.add(Dropout(0.2))  # was not in original model

# Output layer
# For binary/2-class problems use ONE sigmoid unit, 
# for multi-class/multi-label problems use n output units and activation='softmax!'
model.add(Dense(1,activation='sigmoid'))
# Compiling and training the model
model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])
epochs = 30
history = model.fit(train_set, train_classes, batch_size=40, epochs=epochs)
# Verifying Accuracy on Test Set
test_pred = model.predict_classes(test_set)
accuracy_score(test_classes, test_pred)<jupyter_output><empty_output><jupyter_text>Parallel CNNs: Graph ModelThese are graph models with more than 1 pipeline of Convolution layers: They use **2 parallel pipelines of Convolution and Pooling**, processing the same input data in different ways.This makes use of domain knowledge, where it is known in the Music IR domain that time and frequencies comprise different information in a Machine Learning task, which can be beneficial to process separately (depending on the use case). This is a major difference to processing images.It has been discovered, that **CNNs for music work better**, when they have one filter that is detecting frequencies in the vertical axis, and another filter that is focused on the time axis, i.e. detecting rhythm. Consequently, this is realized in a parallel CNN, where 2 layers are not stacked after each other, but first run independently in parallel with their output being merged later.Note that this might not hold true for using deeper networks.To create parallel CNNs we need a "graph-based" model. In Keras this is realized via the **functional API** of the **Model() class**.We use it to create two CNN layers that run in parallel to each other and are merged subsequently.In the functional API, you pass the name of the previous layer in (brackets) after defining the next layer.<jupyter_code># SET DESIRED PARAMETERS
n_filters = 64 #16
activation = 'relu'  # 'sigmoid'  # or other

# MODEL DEFINITION

# Input "layer" only specifies the input shape
input = Input(input_shape)

# CNN layers
# The functional API allows to specify the predecessor in (brackets) after the new Layer function call

# Pipeline 1
conv_layer1 = Conv2D(n_filters, (10, 2), activation=activation)(input)       # vertical filter
maxpool1 = MaxPooling2D(pool_size=(1,2))(conv_layer1) # horizontal pooling

# Pipeline 2 (starting with same input)
conv_layer2 = Conv2D(n_filters, (2, 10), activation=activation)(input)       # horizontal filter
maxpool2 = MaxPooling2D(pool_size=(2,1))(conv_layer2) # vertical pooling

# we have to flatten the Pooling output of both pipelines in order to be concatenated
flat1 = Flatten()(maxpool1)
flat2 = Flatten()(maxpool2)

# Then we can merge (concatenate) the 2
merged = concatenate([flat1, flat2])

# we add a Dense layer after the merging
full = Dense(256, activation=activation)(merged)

# and the Output layer
output_layer = Dense(1, activation='sigmoid')(full)

# finally create the model: define input and output layer
model = Model(inputs=input, outputs=output_layer)
model.summary()
# Define a loss function 
loss = 'binary_crossentropy'  # 'categorical_crossentropy' for multi-class problems

# Optimizer = Stochastic Gradient Descent
optimizer = 'sgd' 

# Compiling the model
model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])
# TRAINING the model
epochs = 15
history = model.fit(train_set, train_classes, batch_size=32, epochs=epochs)<jupyter_output>Epoch 1/15
96/96 [==============================] - 3s 27ms/step - loss: 0.5803 - acc: 0.6354
Epoch 2/15
96/96 [==============================] - 1s 7ms/step - loss: 0.3932 - acc: 0.8229
Epoch 3/15
96/96 [==============================] - 1s 7ms/step - loss: 0.2994 - acc: 0.8958
Epoch 4/15
96/96 [==============================] - 1s 7ms/step - loss: 0.3044 - acc: 0.8646
Epoch 5/15
96/96 [==============================] - 1s 6ms/step - loss: 0.2254 - acc: 0.9375
Epoch 6/15
96/96 [==============================] - 1s 6ms/step - loss: 0.2059 - acc: 0.9271
Epoch 7/15
96/96 [==============================] - 1s 6ms/step - loss: 0.1835 - acc: 0.9583
Epoch 8/15
96/96 [==============================] - 1s 6ms/step - loss: 0.1613 - acc: 0.9479
Epoch 9/15
96/96 [==============================] - 1s 6ms/step - loss: 0.1226 - acc: 0.9896
Epoch 10/15
96/96 [==============================] - 1s 6ms/step - loss: 0.1114 - acc: 0.9896
Epoch 11/15
96/96 [==============================] - 1s 7ms/step - l[...]<jupyter_text>Verifying Accuracy on Test SetNote: The functional API, i.e. Model() does not have a convenience method `.predict_classes()`. We therefore do 'raw' predictions with `predict()`, which returns values between 0 and 1, and then round to the nearest value (0 or 1).<jupyter_code>test_pred = model.predict(test_set)
#test_pred[0:35,0]
test_pred = np.round(test_pred)
accuracy_score(test_classes, test_pred)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11724605157301944, 'fraction_numerical': 0.03129101918109836, 'mean_word_length': 5.145719489981785, 'pattern_counts': {'":': 0, '<': 64, '<?xml version=': 0, '>': 64, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '120268', 'license': 'CNRI-Python', 'n_tokens_mistral': 7619, 'n_tokens_neox': 6902, 'n_words': 2520, 'path': 'Music_speech_classification.ipynb', 'repo_name': 'audiofeature/DeepLearningTutorial_2019', 'size': '23628'}