<jupyter_start><jupyter_text>Building an Audio Data Model using MFCC Features  <NAME> (@ryanbales)<EMAIL> ***  Import Packages<jupyter_code>import librosa
import librosa.display
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import os
import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import BatchNormalization
from keras.layers import LSTM<jupyter_output><empty_output><jupyter_text>Define Constants<jupyter_code>num_mfccs = 20
max_feature_length = 128

train_data_path = "data/audio_train/train/"
val_data_path = "data/audio_train/validation/"

model_output_path = "models/mfcc/"<jupyter_output><empty_output><jupyter_text>Feature Extraction  Let's take a look at the Librosa Python Library<jupyter_code>raw_audio, sample_rate = librosa.load("{}Applause/170eeda2.wav".format(train_data_path), sr=None)
raw_audio
raw_audio.shape
sample_rate<jupyter_output><empty_output><jupyter_text>Visualize the RAW Audio<jupyter_code>plt.figure(figsize=(10, 4))
librosa.display.waveplot(raw_audio, sr=sample_rate)
plt.title('RAW Audio')<jupyter_output><empty_output><jupyter_text>Extract MFCCs using Librosa<jupyter_code>mfccs = librosa.feature.mfcc(y=raw_audio, sr=sample_rate, n_mfcc=num_mfccs)
mfccs
mfccs.shape<jupyter_output><empty_output><jupyter_text>Visualize the MFCCs<jupyter_code>plt.figure(figsize=(10, 4))
librosa.display.specshow(mfccs, x_axis='time')
plt.colorbar()
plt.title('MFCC')
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>Define a Method to Extact MFCCs from an Audio File<jupyter_code>def get_mfccs(wav_file, num_mfcc=20, max_length=128):
    raw_audio, sample_rate = librosa.load(wav_file, sr=None, duration=1)
    mfccs = librosa.feature.mfcc(y=raw_audio, sr=sample_rate, n_mfcc=num_mfcc)
    
    # Truncate Each MFCC to max_length
    if (max_length > mfccs.shape[1]):
        pad_width = max_length - mfccs.shape[1]
        mfccs = np.pad(mfccs, pad_width=((0, 0), (0, pad_width)), mode='constant')
    else:
        mfccs = mfccs[:, :max_length]
        
    return mfccs<jupyter_output><empty_output><jupyter_text>Process all Files and Extract Features<jupyter_code>mfcc_list = []
category_list = []

for root, subdirs, files in os.walk(train_data_path):
    subdirs.sort()
    for subdir in subdirs:
        for file_name in os.listdir(os.path.join(root, subdir)):
            if file_name.endswith(".wav"):
                mfcc = get_mfccs(os.path.join(root, subdir, file_name), num_mfccs, max_feature_length)
                mfcc_list.append(mfcc)
                category_list.append(subdir)
X_features = np.array(mfcc_list)
from sklearn import preprocessing
le = preprocessing.LabelEncoder()
y_features = keras.utils.np_utils.to_categorical(le.fit_transform(category_list))<jupyter_output><empty_output><jupyter_text>Save Feature Extraction Results to Disk Note:  We're saving the extracted features to a file in case we need them later on.  This helps us not have to re-run the Feature Extraction Steps.<jupyter_code>np.save("{}features.npy".format(model_output_path), X_features)
np.save("{}labels.npy".format(model_output_path), y_features)
np.save("{}label_encoder.npy".format(model_output_path), le)<jupyter_output><empty_output><jupyter_text>Model Training<jupyter_code>model = Sequential()
model.add(BatchNormalization(input_shape=[X_features.shape[1], X_features.shape[2]]))
model.add(LSTM(units=128, dropout=0.05, recurrent_dropout=0.35, return_sequences=True))
model.add(LSTM(units=32, dropout=0.05, recurrent_dropout=0.35, return_sequences=False))
model.add(Dense(len(le.classes_), activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='Adam', metrics=['categorical_accuracy'])

model.fit(X_features, y_features, batch_size=16, epochs=50)<jupyter_output>Epoch 1/50
2968/2968 [==============================] - 10s 3ms/step - loss: 3.6581 - categorical_accuracy: 0.0677
Epoch 2/50
2968/2968 [==============================] - 8s 3ms/step - loss: 3.4578 - categorical_accuracy: 0.1088
Epoch 3/50
2968/2968 [==============================] - 8s 3ms/step - loss: 3.1795 - categorical_accuracy: 0.1577
Epoch 4/50
2968/2968 [==============================] - 8s 3ms/step - loss: 2.9635 - categorical_accuracy: 0.2038
Epoch 5/50
2968/2968 [==============================] - 8s 3ms/step - loss: 2.7682 - categorical_accuracy: 0.2554
Epoch 6/50
2968/2968 [==============================] - 8s 3ms/step - loss: 2.6138 - categorical_accuracy: 0.2921
Epoch 7/50
2968/2968 [==============================] - 8s 3ms/step - loss: 2.5132 - categorical_accuracy: 0.3123
Epoch 8/50
2968/2968 [==============================] - 8s 3ms/step - loss: 2.3757 - categorical_accuracy: 0.3467
Epoch 9/50
2968/2968 [==============================] - 8s 3ms/step - loss: 2.2850 - ca[...]<jupyter_text>Now that our Model is trained, let's take a look at the Model Layers<jupyter_code>model.summary()<jupyter_output>_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
batch_normalization_2 (Batch (None, 20, 128)           512       
_________________________________________________________________
lstm_11 (LSTM)               (None, 20, 128)           131584    
_________________________________________________________________
lstm_12 (LSTM)               (None, 32)                20608     
_________________________________________________________________
dense_6 (Dense)              (None, 41)                1353      
=================================================================
Total params: 154,057
Trainable params: 153,801
Non-trainable params: 256
_________________________________________________________________<jupyter_text>Save the Trained Model to Disk<jupyter_code>from keras.models import load_model
model.save("{}model.h5".format(model_output_path))<jupyter_output><empty_output><jupyter_text>Let's Take a Look at How the Model is Performing<jupyter_code>mfcc = np.array([get_mfccs("{}Bass_drum/f78fbf28.wav".format(val_data_path), num_mfccs, max_feature_length)])
prediction = model.predict(mfcc)
prediction
le.classes_[np.argmax(prediction)]<jupyter_output><empty_output><jupyter_text>Let's run Prediction on the files in our Validation DataSet  Load the Validation Dataset<jupyter_code>df_validation = pd.read_csv("data/validation_split.csv")
df_validation["predicted_label"] = ""
print(df_validation.shape)
df_validation.head()<jupyter_output>(742, 4)<jupyter_text>Iterate over the Validation Dataset and Run the Model Prediction Step<jupyter_code>for index, row in df_validation.iterrows():
    val_file_path = "{}{}/{}".format(val_data_path, row['label'], row['fname'])
  
    mfcc = np.array([get_mfccs(val_file_path, num_mfccs, max_feature_length)])
    prediction = model.predict(mfcc)
    predicted_label = le.classes_[np.argmax(prediction)]

    df_validation.at[index,'predicted_label'] = predicted_label
df_validation.head()<jupyter_output><empty_output><jupyter_text>Let's Review our F1 Score per Category<jupyter_code>y_val = df_validation.label
y_val_predicted = df_validation.predicted_label

# Classification Report
from sklearn.metrics import classification_report
print(classification_report(y_val, y_val_predicted))<jupyter_output>precision    recall  f1-score   support

      Acoustic_guitar       0.75      0.86      0.80        21
             Applause       0.48      0.83      0.61        12
                 Bark       0.60      0.69      0.64        13
            Bass_drum       0.77      0.77      0.77        13
Burping_or_eructation       0.86      0.80      0.83        15
                  Bus       0.50      0.50      0.50        12
                Cello       0.73      0.76      0.75        25
                Chime       0.62      0.38      0.48        13
             Clarinet       0.62      0.81      0.70        26
    Computer_keyboard       0.25      0.25      0.25        12
                Cough       0.43      0.21      0.29        14
              Cowbell       0.95      0.95      0.95        19
          Double_bass       0.53      0.53      0.53        19
 Drawer_open_or_close       0.31      0.31      0.31        13
       Electric_piano       0.53      0.53      0.53  [...]
{'fraction_non_alphanumeric': 0.14626257786296118, 'fraction_numerical': 0.057379012937230474, 'mean_word_length': 3.963733650416171, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 64, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '422550', 'license': 'Apache-2.0', 'n_tokens_mistral': 3102, 'n_tokens_neox': 2758, 'n_words': 539, 'path': '03_Audio_Data_Model_MFCC.ipynb', 'repo_name': 'ryanbales/Audio-Data-Insights', 'size': '8365'}