<jupyter_start><jupyter_text>Speech Emotion Detection Using Ravdess & Tess Dataset in CNN__DATASET:For this Project, we have Gathered a total no of 5252 Samples.From __RAVDESS__ we have included 1440 files containing Speech and 1012 files Containing Songs based on different human Emotions totalling to __2452__. The RAVDESS contains 24 professional actors (12 female, 12 male), vocalizing two lexically-matched statements in a neutral North American accent. Speech emotions includes calm, happy, sad, angry, fearful, surprise, and disgust expressions. and song contains calm, happy, sad, angry, and fearful emotions. Each file was rated 10 times on emotional validity, intensity, and genuineness. Ratings were provided by 247 individuals who were characteristic of untrained adult research participants from North America. A further set of 72 participants provided test-retest data. High levels of emotional validity, interrater reliability, and test-retest intrarater reliability were reported.From __TESS__ we have __2800__ files where a set of 200 target words were spoken in the carrier phrase "Say the word _' by two actresses (aged 26 and 64 years) and recordings were made of the set portraying each of seven emotions (anger, disgust, fear, happiness, pleasant surprise, sadness, and neutral). There are 2800 stimuli in total. Two actresses were recruited from the Toronto area. Both actresses speak English as their first language, are university educated, and have musical training. Audiometric testing indicated that both actresses have thresholds within the normal range.  Importing Librosa (A Python Package) for Audio and Music analysis so we can extract audio features such as pitch, tone, amplitude etc.<jupyter_code>import librosa
from librosa import display

#importing a Sample audio file from the DataSet
data, sampling_rate = librosa.load('dataset/03-01-03-01-01-01-17.wav')<jupyter_output><empty_output><jupyter_text>Plotting the Imported Audio file loaded by Librosa right now using Matplotlib and waveplot function of Librosa. The Function will plot the Amplitude envelope of the waveform.<jupyter_code>#Importing Matplotlib and setting matplotlib inline so the output will be visiv
import matplotlib.pyplot as plt
%matplotlib inline

plt.figure(figsize=(15, 4))
librosa.display.waveplot(data, sr=sampling_rate)<jupyter_output><empty_output><jupyter_text>In the above plot you can observe that there exist a Sudden spike in the amplitude at time around 1.4 second. Similarly from 1.8 to 2.3 second there is a shaking/Disorientation in the amplitude that can be obeserved.AS the above file is of a happy sound showing such behaviour that can be observed in any Laughing/happy sound, setting the basic criteria for distinguishing various emothions.  *Loading the DataSetWe will create our numpy array for extracting Mel-frequency cepstral coefficients (MFCCs), while the classes to predict will be extracted from the name of the file.The following list shows the notation for each emotion in our dataset 0 = neutral, 1 = calm, 2 = happy, 3 = sad, 4 = angry, 5 = fearful, 6 = disgust, 7 = surprised<jupyter_code>#Importing Numpy and Other Libraries

import numpy as np
import time
import os
path = 'dataset/'
lst = []

start_time = time.time()

for subdir, dirs, files in os.walk(path):
    for file in files:
        try:
            #Load librosa array, obtain mfcss, store the file and the mfcss information in a new array
            
            X, sample_rate = librosa.load(os.path.join(subdir,file), res_type='kaiser_fast')
            mfccs = np.mean(librosa.feature.mfcc(y=X, sr=sample_rate, n_mfcc=40).T,axis=0) 
            
            # The instruction below converts the labels (from 1 to 8) to a series from 0 to 7
            # This is because our predictor needs to start from 0 otherwise it will try to predict also 0.
            
            file = int(file[7:8]) - 1 
            arr = mfccs, file
            lst.append(arr)
            
            # If the file is not valid, skip it
            
        except ValueError:
            continue

print("Data loaded. Loading time: %s seconds" % (time.time() - start_time))<jupyter_output>Data loaded. Loading time: 303.0925397872925 seconds<jupyter_text>The Dataset is loaded. The timestamp above is to insure the dataset is loaded successfully & may vary everytime. Now lets Zip the X and y values so that it will make a list of first elements and 2nd Elements for future validation of our model<jupyter_code>X, y = zip(*lst)<jupyter_output><empty_output><jupyter_text>Now creating arrays of X and y for our impletention purpose<jupyter_code>import numpy as np
X = np.asarray(X)
y = np.asarray(y)

#Lets check the shape of X and y
X.shape, y.shape<jupyter_output><empty_output><jupyter_text>As expected 5252 files with their features are loaded from the dataset Now using Joblib to save files so that we won't have to perform the above Operations over and over<jupyter_code>import joblib

X_name = 'X.joblib'
y_name = 'y.joblib'
save_dir = 'C:\\Users\\Hrishikesh\\Desktop\\Project\\#Project\\Joblib'

savedX = joblib.dump(X, os.path.join(save_dir, X_name))
savedy = joblib.dump(y, os.path.join(save_dir, y_name))
# Loading saved files
import joblib
X = joblib.load('Joblib/X.joblib')
y = joblib.load('Joblib/y.joblib')<jupyter_output><empty_output><jupyter_text>*Decision Tree Classifier__To Accomplish Successfull classification we are using Decision tree classifier__<jupyter_code>from sklearn.model_selection import train_test_split
#creating train and test set from the dataset
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.25, random_state=50)
#Using Decision tree Classifier to train the model
from sklearn.tree import DecisionTreeClassifier
dtree = DecisionTreeClassifier()
dtree.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>__we have trained the model using Decision-tree-Classifier. Now let's try to predict the output using test set & observe the accuracy__<jupyter_code>predictions = dtree.predict(X_test)
from sklearn.metrics import classification_report,confusion_matrix
print(classification_report(y_test,predictions))<jupyter_output>precision    recall  f1-score   support

           0       0.83      0.80      0.82       157
           1       0.60      0.62      0.61        85
           2       0.69      0.67      0.68       206
           3       0.74      0.76      0.75       190
           4       0.72      0.69      0.70       201
           5       0.69      0.66      0.67       185
           6       0.70      0.74      0.72       134
           7       0.67      0.71      0.69       155

    accuracy                           0.71      1313
   macro avg       0.70      0.71      0.70      1313
weighted avg       0.71      0.71      0.71      1313<jupyter_text>The Classification Report shows that the Precision/Accuracy is around 71%. The precision is high enough to classify as a successful model but we require more precision than this to implement it in any future Projects.  *Neural Network__Creating a Convolutional Neural Network Using Keras & Tensorflow to create a more efficient model__<jupyter_code>#Now creating train and test set for our Neural Network
import numpy as np
x_traincnn = np.expand_dims(X_train, axis=2)
x_testcnn = np.expand_dims(X_test, axis=2)

#Lets check the Dimensions of the train and test set
x_traincnn.shape, x_testcnn.shape<jupyter_output><empty_output><jupyter_text>As we know the Neural Network consist of Layers of Neurons (Input layers) interlinked to each other and produce final output/prediction in last layer.  The Above operation is implemented in order to create a layered Neural network as the input will be in a 3 dimension.<jupyter_code>#Installing Keras and Tensorflow
pip install tensorflow
pip install Keras
#Importing Keras and Tensorflow
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
from tensorflow.keras.preprocessing import sequence
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Embedding
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.layers import Input, Flatten, Dropout, Activation
from tensorflow.keras.layers import Conv1D, MaxPooling1D
from tensorflow.keras.models import Model
from tensorflow.keras.callbacks import ModelCheckpoint

#creating a linear model using Sequential() method from Keras
model = Sequential()

#creating hidden layers for the model
model.add(Conv1D(64, 5,padding='same',input_shape=(40,1)))
model.add(Activation('relu'))
model.add(Dropout(0.1))
model.add(MaxPooling1D(pool_size=(4)))
model.add(Conv1D(128, 5,padding='same',))
model.add(Activation('relu'))
model.add(Dropout(0.1))
model.add(MaxPooling1D(pool_size=(4)))
model.add(Conv1D(256, 5,padding='same',))
model.add(Activation('relu'))
model.add(Dropout(0.1))
model.add(Flatten())
model.add(Dense(8))
model.add(Activation('softmax'))

#creating a Compliler using rmsprop() to Maintain a moving average of the square of gradients
opt = tf.keras.optimizers.RMSprop(lr=0.00005, rho=0.9, epsilon=1e-07, decay=0.0)<jupyter_output><empty_output><jupyter_text>__Now lets check our CNN model__<jupyter_code>model.summary()
#Lets make our model ready for training
model.compile(loss='sparse_categorical_crossentropy',
              optimizer=opt,
              metrics=['accuracy'])
cnnmodel = model.fit(x_traincnn, y_train, batch_size=16, epochs=400, validation_data=(x_testcnn, y_test))
# Plotting model Loss 
plt.plot(cnnmodel.history['loss'])
plt.plot(cnnmodel.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
# Plotting model accuracy
plt.plot(cnnmodel.history['acc'])
plt.plot(cnnmodel.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()<jupyter_output><empty_output><jupyter_text>Now lets check the predictions vs the actual values(y_test) to check accuracy of our CNN<jupyter_code>predictions = model.predict_classes(x_testcnn)
predictions
# checking against y_test so casting it to new array
cnn_Ytest = y_test.astype(int)
cnn_Ytest
#Its time for the report of our model
from sklearn.metrics import classification_report
report = classification_report(cnn_Ytest, predictions)
print(report)
#Now lets plot the Confusion Matrix of our plot to display the performance of our model
from sklearn.metrics import confusion_matrix
matrix = confusion_matrix(cnn_Ytest, predictions)
print (matrix)<jupyter_output>[[143   4   2   4   2   1   0   1]
 [  4  72   0   6   1   0   2   0]
 [  0  11 174   3   5   6   2   5]
 [  3   4   0 164   3  10   4   2]
 [  0   0   1   0 186   4   9   1]
 [  0   2   5  10   1 166   1   0]
 [  0   0   2   1   8   2 117   4]
 [  1   3   3   1   1   3   1 142]]<jupyter_text>Now its time to save our model for future use<jupyter_code>model.save('predict_Emotion_final_model.tf')
print('Model Saved')<jupyter_output>Model Saved<jupyter_text>Loading Our Model using different variable<jupyter_code>new_model_1 = tf.keras.models.load_model('predict_Emotion_final_model.tf')
new_model_1.summary()<jupyter_output>Model: "sequential_3"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv1d_9 (Conv1D)            (None, 40, 64)            384       
_________________________________________________________________
activation_12 (Activation)   (None, 40, 64)            0         
_________________________________________________________________
dropout_9 (Dropout)          (None, 40, 64)            0         
_________________________________________________________________
max_pooling1d_6 (MaxPooling1 (None, 10, 64)            0         
_________________________________________________________________
conv1d_10 (Conv1D)           (None, 10, 128)           41088     
_________________________________________________________________
activation_13 (Activation)   (None, 10, 128)           0         
______________________________________________________[...]<jupyter_text>Lets check The Accuracy of Our CNN model using a test run<jupyter_code>loss, acc = new_model_1.evaluate(x_testcnn, y_test)
print("Saved CNN model, accuracy: {:5.2f}%".format(100*acc))<jupyter_output>1313/1313 [==============================] - 0s 106us/sample - loss: 0.4114 - acc: 0.8865
Saved CNN model, accuracy: 88.65%
{'fraction_non_alphanumeric': 0.08251192368839427, 'fraction_numerical': 0.037599364069952304, 'mean_word_length': 3.998410806515693, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 55, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '396013', 'license': 'Apache-2.0', 'n_tokens_mistral': 4014, 'n_tokens_neox': 3674, 'n_words': 1317, 'path': 'Convolutional Neural Network.ipynb', 'repo_name': 'hrishi09/SED_MODEL', 'size': '12580'}