<jupyter_start><jupyter_text>In this project, we will use machine learning to complete a single word speech recognition task.  Here, we use one second recordings of the digits zero - nine from the Google Speech Command database https://ai.googleblog.com/2017/08/launching-speech-commands-dataset.html, meaning that every recording used belongs to one of ten classes (zero, one, two, three, four, five, six, seven, eight, or nine).  To perform the recognition, we have two frameworks as shown below:1. We can directly feed the audio signal into a neural network for audio classification.  Models that read in the raw signal with no feature extraction are called end to end models.2. We can extract some short time feature from each frame of the audio, giving the spectrogram or another acoustic representation.  We can then take these combination of these features accross time as an image and perform image classification on them.In this project, you will explore both methods.You may (but are not required) to change any of the code given here.  You can just add your new code to the bottom of the file.**Before you run anything, go to the toolbar at the top and select: Runtime > Change runtime type > GPU.** Mount your google drive to give google colab access to files stored in your google drive<jupyter_code>from google.colab import drive
drive.mount('/content/gdrive/')<jupyter_output>Mounted at /content/gdrive/<jupyter_text>Change directory to the place where the google ai speech commands dataset is stored.  You can find the path by typing "!ls" to see what files and directories are currently available.  Then type "!ls " to see what files are in the next subdirectory.  Keep doing that, separating the subdirectories with slashes "/" until you get to the speech_commands_v0.01 folder as shown below.  Then put that filepath in the os.chdir command.<jupyter_code>import os

os.chdir('/content/gdrive/My Drive/speech_commands_v0.01')
!ls<jupyter_output>_background_noise_  happy    README.md		     train_list.txt
bed		    house    right		     tree
bird		    left     seven		     two
cat		    LICENSE  sheila		     up
dog		    marvin   six		     validation_list.txt
down		    nine     stop		     wow
eight		    no	     testing_digit_list.txt  yes
five		    off      testing_list.txt	     zero
four		    on	     three
go		    one      train_digit_list.txt<jupyter_text>Load in the text file that lists the audio files to be used.  We'll only use the subset of files listed in this text file for this project.<jupyter_code>text_file = open("train_digit_list.txt", "r")
training_list = text_file.read().splitlines()<jupyter_output><empty_output><jupyter_text>Read in all of the audio files.  Store the raw audio to a list called all_wav_list.  Store any features extracted to a listt called all_feat_list.  This part may take a while depending on what features you extract.  You can go watch netflix or work on your homework or something and come back when it's done>  You have to redo this every time you run the code which can be kind of a pain.  If you prefer, you can store the list after you've generated it and then redownload it instead of recreating it: Import the pickle libraryimport pickle Create a listl = [1,2,3,4]  create a liststore the list as a file    with open("test.txt", "wb") as fp:   Pickling        pickle.dump(l, fp) open the file and read the list from it     with open("test.txt", "rb") as fp:    Unpickling        b = pickle.load(fp)<jupyter_code>import librosa.feature as lf
import scipy.io.wavfile as wav
import numpy as np
from tqdm.notebook import tnrange

#initialize lists
all_wav_list = []
all_feat_list = []
all_labels = []

#loop through all audio files listed in the text file
for i in tnrange(len(training_list), desc='Load in files'):
  #load in the given audio file
  fs, audio = wav.read(training_list[i])

  
  z=np.zeros((fs,))
  #if an audio file is less than a second, add zeros to it to make it a second
  if audio.size<=fs:
    z[:audio.size]=audio
  # if an audio file is longer than a second, clip it to a second
  elif audio.size>fs:
    z=audio[:fs]
  feat = lf.melspectrogram(z.astype('float'), sr =fs)
  #here, we use the melspectrogram as a feature. You can use other features like
  #LPCs, mfccs, or whatever you find.  The Librosa library has more features,
  #and you can explore other libraries
  all_wav_list.append(z.astype('float'))
  all_feat_list.append(feat.reshape(1, feat.shape[0], feat.shape[1]))

# get labels from the file name (ie which word is in the audio file)
  which_word=training_list[i].split('/')[0]
  all_labels.append(which_word)<jupyter_output><empty_output><jupyter_text>Let's listen to one of the audio files to verify that it loaded in correctly<jupyter_code>import IPython
IPython.display.Audio(training_list[0])<jupyter_output><empty_output><jupyter_text>And verify that the sizes of the lists look correct<jupyter_code>print(len(all_wav_list))
print(len(all_feat_list))
print(len(all_labels))
print(all_feat_list[0].shape)<jupyter_output>2494
2494
2494
(1, 128, 32)<jupyter_text>Now we'll start with speech recognition from the raw audio.  We'll concatenate the raw signals into a matrix X_sig.<jupyter_code>X_sig = np.vstack(all_wav_list)
X_sig = X_sig.reshape(X_sig.shape[0], X_sig.shape[1], 1)
print(X_sig.shape)<jupyter_output>(2494, 16000, 1)<jupyter_text>And we'll one hot encode our labels.  This will transform our labels from words to a vector that represents the label with a numerical value.  All of the encoded labels will have the same norm to prevent bias.  For instance, if there were five words then they would be encoded as:$y1 = [1, 0, 0, 0, 0]$$y2 = [0, 1, 0, 0, 0]$$y3 = [0, 0, 1, 0, 0]$$y4 = [0, 0, 0, 1, 0]$$y5 = [0, 0, 0, 0, 1]$This will be useful in comparing correct and incorrect labels later<jupyter_code>from sklearn import preprocessing
import scipy as sp

le = preprocessing.LabelEncoder()
le.fit(all_labels)
encoded_labels = le.transform(all_labels)

oh_enc = preprocessing.OneHotEncoder()
oh_enc.fit(encoded_labels.reshape(-1,1))

y = oh_enc.transform(encoded_labels.reshape(-1,1))
y = sp.sparse.csr_matrix.toarray(y)<jupyter_output><empty_output><jupyter_text>Then we'll devide the whole set into a testing and a training set.<jupyter_code>from sklearn.model_selection import train_test_split

X_sig_train, X_sig_test, y_train, y_test = train_test_split(X_sig, y, test_size=0.2, random_state=42)<jupyter_output><empty_output><jupyter_text>Create and run an end to end model for audio classification:  We create a sequential model using the keras module in tensorflow and then add the desired layer to it.The graph is created of a static size, so the size of every layer must be pre-defined.  Tensorflow will take care of calculated the sizes needed in each layer if we specify the size of the input layer, the size of the output layer, and the parameters of each desired layers.The input is a collection of 1D signals each one second long.  The static graph requires that all inputs be the same size which we make sure of in shaping the input matrix above.  This will have the shape (number_of_signals, len_of_signals, 1).  The number of signals we feed into the network does not affect its structure, so we just need to specify the shapes of these inputs in input layer as (fs, 1) for the one second long signals.This input is then passed into convolutional layers and then recurrent layers (GRU) before the fully connected network (ie multilayer perceptrion) makes the final output decision.  In betweek the layers are forms of regularization such as MaxPooling (downsampling the output of a layer), dropout(randomly setting some percentage of the weights of a layer to zero for given iteration), and batch normalization (normalizing the mean and standard deviation of the output of a layer so that the numbers don't become too large).  These methods try to ensure that the solution found by the neural network does not rely too strongly on any one trend that occurs in its weights or any trend that only occurs with a small numbe of its weights.  This helps ensure that the solution found by the network is not too specific to the training data and generalizes well to any other test set of a similar distribution.At the output layer, we set the size of the last fully connected layer to 10 to let the network know that we expect a classification decision between one of the ten classes.  The output of the last layer is of the form:$out=[x_0, x_1, x_2, x_3, x_4,x_5,x_6,x_7,x_8,x_9]$where each $x_i$ is the calculated probability that the given audio sample belongs to class i.  The classification decision is then $y_{pred} =argmax_i(out)$ which we compare to the actual labels stored in the actual $y_{test}$.<jupyter_code>import tensorflow as tf

model_e2e = tf.keras.models.Sequential()
model_e2e.add(tf.keras.Input(shape=(fs,1))) #Make sure that the input size is the size of the signal
model_e2e.add(tf.keras.layers.BatchNormalization(axis=-1, momentum=0.99, epsilon=1e-3, center=True, scale=True, )) #adjust input to unit variance and zero mean
#First Conv1D layer
model_e2e.add(tf.keras.layers.Conv1D(8,13, padding='valid', activation='relu', strides=1)) #Convolve with 8 1D kernels of length 13
model_e2e.add(tf.keras.layers.MaxPooling1D(3)) #Downsample - take the max out of every three elements
model_e2e.add(tf.keras.layers.Dropout(0.3)) #drop nodes with probability 0.3
#Second Conv1D layer
model_e2e.add(tf.keras.layers.Conv1D(16, 11, padding='valid', activation='relu', strides=1)) #Convolve with 16 1D kernels of length 11
model_e2e.add(tf.keras.layers.MaxPooling1D(3))
model_e2e.add(tf.keras.layers.Dropout(0.3))
#Third Conv1D layer
model_e2e.add(tf.keras.layers.Conv1D(32, 9, padding='valid', activation='relu', strides=1))
model_e2e.add(tf.keras.layers.MaxPooling1D(3))
model_e2e.add(tf.keras.layers.Dropout(0.3))
model_e2e.add(tf.keras.layers.BatchNormalization(axis=-1, momentum=0.99, epsilon=1e-3, center=True, scale=True))
model_e2e.add(tf.keras.layers.Bidirectional(tf.python.keras.layers.CuDNNGRU(128, return_sequences=True), merge_mode='sum')) #Recurrent layer, uses time series data
model_e2e.add(tf.keras.layers.Bidirectional(tf.python.keras.layers.CuDNNGRU(128, return_sequences=True), merge_mode='sum'))
model_e2e.add(tf.keras.layers.Bidirectional(tf.python.keras.layers.CuDNNGRU(128, return_sequences=False), merge_mode='sum')) #set return sequences to False for last recurrent layer
model_e2e.add(tf.keras.layers.BatchNormalization(axis=-1, momentum=0.99, epsilon=1e-3, center=True, scale=True))
#Flatten layer
model_e2e.add(tf.keras.layers.Flatten()) # Turn 2D result of convolution to a single vector
#Dense Layer 1
model_e2e.add(tf.keras.layers.Dense(256, activation='relu')) #Fully connected layer
model_e2e.add(tf.keras.layers. Dense(10, activation="softmax")) #output layer, need size = num_classes
model_e2e.summary() #show breakdown of parameters

model_e2e.compile(loss='categorical_crossentropy', optimizer='nadam', metrics=['accuracy']) #decide loss function and metrics

hist = model_e2e.fit(
    x=X_sig_train, 
    y=y_train,
    epochs=50, 
    batch_size=32)

model_e2e.evaluate(X_sig_test, y_test)<jupyter_output>16/16 [==============================] - 4s 275ms/step - loss: 0.4062 - accuracy: 0.8998<jupyter_text>Now we'll move on to performing speech recognition by identifying the image of the spectrogram.  We'll begon by concatenating the features extracted into a matrix<jupyter_code>X_im = np.vstack(all_feat_list)
X_im=np.array(X_im.reshape(X_im.shape[0],X_im.shape[1],X_im.shape[2], 1))
print(X_im.shape)<jupyter_output>(2494, 128, 32, 1)<jupyter_text>Let's view an image of our features<jupyter_code>import matplotlib.pyplot as plt
im = X_im[100,:,:].reshape(X_im.shape[1],X_im.shape[2])
im=im[::-1,:]
plt.figure(figsize=(20,10), dpi=100)
plt.imshow(im)
plt.ylabel('frequency (bin number)')
plt.xlabel('time (frame number)')
plt.title('A random spectrogram');<jupyter_output><empty_output><jupyter_text>We'll similarly split our whole set into testing and training sets<jupyter_code>from sklearn.model_selection import train_test_split

X_im_train, X_im_test, y_train, y_test = train_test_split(X_im, y, test_size=0.2, random_state=42)<jupyter_output><empty_output><jupyter_text>Here we similarly train a CNN to recognize the images generated from the speech signals.  We again create the sequential model, and specify the input size in the first layer.The imput matrix contains a collection of 2D signals.  Here the input matrix is of the shape (number_of_images, number_of_features (or height of each image), number_of_frames (or width of each image), 1).  If these were color RGB images, then we would need to replace the 1 with a 3 to let the network know that the signals have a depth of 3.  We will stick with grayscale images for simplicity.  We tell the network to expect some number of images with a shape of (num_features, num_timesteps, 1) as the input here.This input is then passed into multiple 2D convolutional layers with maxpooling as regularization between.  This 2D maxpooling downsamples accross both the x and y dimensions of the image.The output of the last conv layer is then flattened into a 1D vector and passed to the multilayer perceptiron for a final classification decision in the same way as in the end to end model.<jupyter_code>import tensorflow as tf

n_features = X_im.shape[1]
time_steps = X_im.shape[2]

model_cnn = tf.keras.models.Sequential()
model_cnn.add(tf.keras.layers.Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(n_features, time_steps, 1))) # convolve with 32 kernels of size 3 x 3
model_cnn.add(tf.keras.layers.Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same')) # convolve with 32 kernels of size 3 x 3
model_cnn.add(tf.keras.layers.MaxPooling2D((2, 2))) #Dowmsample by 2 in each direction- take max element of every 2
model_cnn.add(tf.keras.layers.Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same')) # convolve with 64 kernels of size 3 x 3
model_cnn.add(tf.keras.layers.Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))
model_cnn.add(tf.keras.layers.MaxPooling2D((2, 2)))
model_cnn.add(tf.keras.layers.Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same')) # convolve with 128 kernels of size 3 x 3
model_cnn.add(tf.keras.layers.Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))
model_cnn.add(tf.keras.layers.MaxPooling2D((2, 2)))
model_cnn.add(tf.keras.layers.Flatten()) # Flatten output into a vector
model_cnn.add(tf.keras.layers.Dense(128, activation='relu', kernel_initializer='he_uniform')) # Fully connected layer with 128 nodes
model_cnn.add(tf.keras.layers.Dense(10, activation='softmax')) #output layer, size must equal the number of classes

Adam=tf.keras.optimizers.Adam(lr=0.01, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.9, amsgrad=False)

model_cnn.compile(loss='categorical_crossentropy',
              optimizer=Adam,
              metrics=['accuracy'])
model_cnn.summary() #print out breakdown of model parameters
history=model_cnn.fit(X_im_train, y_train, validation_split = 0.2, batch_size=200, epochs=500)
model_cnn.evaluate(X_im_test, y_test)<jupyter_output>16/16 [==============================] - 0s 13ms/step - loss: 42245080.0000 - accuracy: 0.2585<jupyter_text>Now that we've run this experiment on clean, noise-free signals taken with high quality microphones, and recorded by speakers all with similar speaking styles and accents, let's explore a less ideal case.  In real life, you can rarely expect the input speech to be of such high quality and need to prepare to handle common degradations in the signals.  There could be additive, noise in the background, there could be multiple speakers talking over each other, a speaker could have an accent not present in the training data, etc..  We need to be creative in how we deal with these challenges.  Here we'll introduce another common problem in audio processing: Reverberation.  Reverberations, or echos, occur when not only the original signal reaches the recording device, but also reflections of the soundwave off of nearby surfaces are directed at the microphone and also appear in the signal.  This can be modeled as multiple delayed and attenuated versions of the original signal being added back into the original signal through convolution with a room impulse function.  For this reason, we often refer to reverberations as convolutional noise, as opposed to addative noise which is added on top of the signal Install the pyroomacoustics library.  It is a helpful python library that deals with calculating the effects of echos in a given<jupyter_code>!pip install pyroomacoustics<jupyter_output>Collecting pyroomacoustics
[?25l  Downloading https://files.pythonhosted.org/packages/4d/49/73e2d8fba52663bf8c1eaf70867cc0e70b8aef0cd3232b86117777af1ca8/pyroomacoustics-0.4.2.tar.gz (1.1MB)

[K     |▎                               | 10kB 22.8MB/s eta 0:00:01
[K     |▋                               | 20kB 5.8MB/s eta 0:00:01
[K     |█                               | 30kB 6.9MB/s eta 0:00:01
[K     |█▎                              | 40kB 7.6MB/s eta 0:00:01
[K     |█▋                              | 51kB 6.8MB/s eta 0:00:01
[K     |█▉                              | 61kB 7.7MB/s eta 0:00:01
[K     |██▏                             | 71kB 7.6MB/s eta 0:00:01
[K     |██▌                             | 81kB 8.6MB/s eta 0:00:01
[K     |██▉                             | 92kB 7.8MB/s eta 0:00:01
[K     |███▏                            | 102kB 7.9MB/s eta 0:00:01
[K     |███▍                            | 112kB 7.9MB/s eta 0:00:01
[K     |███▊                            | 122kB 7.9MB/s [...]<jupyter_text>Read in a new test set from another text file<jupyter_code>text_file = open("testing_digit_list.txt", "r")
testing_list = text_file.read().splitlines()<jupyter_output><empty_output><jupyter_text>Add reverberations to the test signals using the pyroom acoustic library<jupyter_code>from tqdm.notebook import tnrange
import pyroomacoustics as pra
import scipy.io.wavfile as wav
import numpy as np
import librosa.feature as lf

all_reverb_signals =[]
all_reverb_feat=[]
trial_labels = []

# The desired reverberation time and dimensions of the room
rt60 = 1.0  # seconds
room_dim = [20, 30, 10]  # meters

e_absorption, max_order = pra.inverse_sabine(rt60, room_dim)

# Create the room
for i in tnrange(len(testing_list)):

  fs, audio = wav.read(testing_list[i])
  room = pra.ShoeBox(
      room_dim, fs=fs, materials=pra.Material(e_absorption), max_order=max_order
  )

  # place the source in the room
  room.add_source([2.5, 3.73, 1.76], signal=audio, delay=1.3)

  mic_locs = np.c_[
      [10, 1, 1], 
  ]

  # finally place the array in the room
  room.add_microphone_array(mic_locs)

  # Run the simulation (this will also build the RIR automatically)
  room.simulate()

  mics_signals = room.mic_array.signals
  mics_signals = mics_signals.reshape(mics_signals.size,)
  z=mics_signals[int(1.5*fs):int(2.5*fs)]

  feat = lf.melspectrogram(z.astype('float'), sr =fs)
  all_reverb_signals.append(z.astype('float'))
  all_reverb_feat.append(feat.reshape(1, feat.shape[0], feat.shape[1]))

  # get labels from the file name (ie which word is in the audio file)
  which_word=testing_list[i].split('/')[0]
  trial_labels.append(which_word)<jupyter_output><empty_output><jupyter_text>Encode the labels from the new dataset using the same encoding scheme used previously.<jupyter_code>le.fit(trial_labels)

encoded_labels = le.transform(trial_labels)

oh_enc = preprocessing.OneHotEncoder()
oh_enc.fit(encoded_labels.reshape(-1,1))

y_trial = oh_enc.transform(encoded_labels.reshape(-1,1))
y_trial = sp.sparse.csr_matrix.toarray(y_trial)
print(len(all_reverb_signals),len(all_reverb_feat))<jupyter_output>2552 2552<jupyter_text>Listen to an audio signal with reverb<jupyter_code>IPython.display.Audio(all_reverb_signals[0], rate=fs)<jupyter_output><empty_output><jupyter_text>Confirm the size of the test signals<jupyter_code>X_sig_rev=np.vstack(all_reverb_signals)
X_sig_rev=X_sig_rev.reshape(X_sig_rev.shape[0],X_sig_rev.shape[1],1)
print(X_sig_rev.shape)
X_im_rev = np.vstack(all_reverb_feat)
X_im_rev=X_im_rev.reshape(X_im_rev.shape[0],X_im_rev.shape[1],X_im_rev.shape[2], 1)
print(all_reverb_feat[0].shape)
print(X_im_rev.shape)<jupyter_output>(1, 128, 32)
(2552, 128, 32, 1)<jupyter_text>View an image with reverb<jupyter_code>im = X_im_rev[1,:,:].reshape(X_im_rev.shape[1],X_im_rev.shape[2])
im=im[::-1,:]
plt.figure(figsize=(20,10), dpi=100)
plt.imshow(im)
plt.ylabel('frequency (bin number)')
plt.xlabel('time (frame number)')
plt.title('A random spectrogram with reverb');<jupyter_output><empty_output><jupyter_text>Now evaluate the performance of the models on the test sets with reverb.<jupyter_code>model_e2e.evaluate(X_sig_rev, y_trial)
model_cnn.evaluate(X_im_rev, y_trial)<jupyter_output>80/80 [==============================] - 1s 7ms/step - loss: 2942623.7500 - accuracy: 0.1983
{'fraction_non_alphanumeric': 0.09805495231800586, 'fraction_numerical': 0.03040317250495704, 'mean_word_length': 4.754686226568867, 'pattern_counts': {'":': 0, '<': 79, '<?xml version=': 0, '>': 82, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '658422', 'license': 'MIT', 'n_tokens_mistral': 7450, 'n_tokens_neox': 6879, 'n_words': 2314, 'path': 'ECE114/Project/ECE_114_project_F2020 (1).ipynb', 'repo_name': 'thomaskost17/fall2020', 'size': '21182'}