<jupyter_start><jupyter_text>TUTORIAL: Sound classification IntroductionThe aim of this Notebook is to use **AI NOTEBOOKS** product to train a model to classify sounds in *.wav* format. USE CASE:  [Best of Watkins Marine Mammal Sound Database](https://www.kaggle.com/shreyj1729/best-of-watkins-marine-mammal-sound-database/version/3)This dataset is composed of 55 different folders corresponding to the marine mammals. In each folder are stored several sound files of each animal in *.wav* format.You can get more information on this dataset on this [website](https://cis.whoi.edu/science/B/whalesounds/index.cfm).The data is then as follows:⚠️ For this example, we choose only the first 45 classes (or folders)To classify sounds, we will go through different steps.  Code- Install and import dependencies- Observe data- Data preprocess- Separate data for training- Define the neural network model- Train model- Evaluate results Step 1 - Import dependenciesBefore starting, import following dependencies.<jupyter_code>import librosa
import librosa.display as lplt
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import os
import csv
import IPython

# preprocessing
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder

# Keras / TensorFlow
import keras
import tensorflow as tf
from tensorflow.keras.models import Sequential<jupyter_output><empty_output><jupyter_text>Step 2 - Observe dataYou can start by looking at your data by displaying different parameters using the librosa library.First, you can do a test on a file.<jupyter_code># select a random sound
test_sound = "data/AtlanticSpottedDolphin/61025001.wav"<jupyter_output><empty_output><jupyter_text>- Loads and decodes the audio<jupyter_code>data, sr = librosa.load(test_sound)
print(type(data), type(sr))
librosa.load(test_sound ,sr = 45600)<jupyter_output><empty_output><jupyter_text>- Play audio in the notebook<jupyter_code>IPython.display.Audio(data, rate = sr)<jupyter_output><empty_output><jupyter_text>- **Waveforms** (visual representations of sound as time on the x-axis and amplitude on the y-axis)<jupyter_code>plt.show(librosa.display.waveplot(data))<jupyter_output><empty_output><jupyter_text>- **Spectrograms**<jupyter_code>stft = librosa.stft(data)
plt.colorbar(librosa.display.specshow(stft, sr = sr, x_axis = 'time', y_axis = 'hz'))
stft_db = librosa.amplitude_to_db(abs(stft))
plt.colorbar(librosa.display.specshow(stft_db, sr = sr, x_axis = 'time', y_axis = 'hz'))<jupyter_output><empty_output><jupyter_text>- **Spectral Rolloff**<jupyter_code>spectral_rolloff = librosa.feature.spectral_rolloff(data + 0.01, sr = sr)[0]
plt.show(librosa.display.waveplot(data, sr = sr, alpha = 0.4))<jupyter_output><empty_output><jupyter_text>- **Chroma Feature**<jupyter_code>chroma = librosa.feature.chroma_stft(data, sr = sr)
lplt.specshow(chroma, sr = sr, x_axis = "time" ,y_axis = "chroma", cmap = "coolwarm")
plt.colorbar()
plt.title("Chroma Features")
plt.show()<jupyter_output><empty_output><jupyter_text>- **Zero Crossing Rate**<jupyter_code>start = 1000
end = 1200
plt.plot(data[start:end])
plt.grid()<jupyter_output><empty_output><jupyter_text>Step 3 - Data preprocessTo train your model, you have to convert the *.wav* into a *.csv* file.<jupyter_code># define columns name
header = "filename length chroma_stft_mean chroma_stft_var rms_mean rms_var spectral_centroid_mean spectral_centroid_var \
    spectral_bandwidth_mean spectral_bandwidth_var rolloff_mean rolloff_var zero_crossing_rate_mean zero_crossing_rate_var \
    harmony_mean harmony_var perceptr_mean perceptr_var tempo mfcc1_mean mfcc1_var mfcc2_mean mfcc2_var mfcc3_mean mfcc3_var \
    mfcc4_mean mfcc4_var label".split()
file = open('data.csv', 'w', newline = '')
with file:
    writer = csv.writer(file)
    writer.writerow(header)

# character string of marine mammals (45)
marine_mammals = "AtlanticSpottedDolphin BeardedSeal Beluga_WhiteWhale BlueWhale BottlenoseDolphin Boutu_AmazonRiverDolphin BowheadWhale \
    ClymeneDolphin Commerson'sDolphin CommonDolphin Dall'sPorpoise DuskyDolphin FalseKillerWhale Fin_FinbackWhale FinlessPorpoise \
    Fraser'sDolphin Grampus_Risso'sDolphin GraySeal GrayWhale HarborPorpoise HarbourSeal HarpSeal Heaviside'sDolphin HoodedSeal \
    HumpbackWhale IrawaddyDolphin JuanFernandezFurSeal KillerWhale LeopardSeal Long_FinnedPilotWhale LongBeaked(Pacific)CommonDolphin \
    MelonHeadedWhale MinkeWhale Narwhal NewZealandFurSeal NorthernRightWhale PantropicalSpottedDolphin RibbonSeal RingedSeal RossSeal \
    Rough_ToothedDolphin SeaOtter Short_Finned(Pacific)PilotWhale SouthernRightWhale SpermWhale".split()

# transform each .wav file into a .csv row
for animal in marine_mammals:
    for filename in os.listdir(f"/workspace/data/{animal}/"):
        sound_name = f"/workspace/data/{animal}/{filename}"
        y, sr = librosa.load(sound_name, mono = True, duration = 30)
        chroma_stft = librosa.feature.chroma_stft(y = y, sr = sr)
        rmse = librosa.feature.rms(y = y)
        spec_cent = librosa.feature.spectral_centroid(y = y, sr = sr)
        spec_bw = librosa.feature.spectral_bandwidth(y = y, sr = sr)
        rolloff = librosa.feature.spectral_rolloff(y = y, sr = sr)
        zcr = librosa.feature.zero_crossing_rate(y)
        mfcc = librosa.feature.mfcc(y = y, sr = sr)
        to_append = f'{filename} {np.mean(chroma_stft)} {np.mean(rmse)} {np.mean(spec_cent)} {np.mean(spec_bw)} {np.mean(rolloff)} {np.mean(zcr)}'    
     
        for e in mfcc:
            to_append += f' {np.mean(e)}'
            
        to_append += f' {animal}'
        file = open('data.csv', 'a', newline = '')
        
        with file:
            writer = csv.writer(file)
            writer.writerow(to_append.split())
# display data.csv 
%cd /workspace/
df = pd.read_csv('data.csv')
df.head()
# dataframe shape
df.shape
# dataframe types
df.dtypes<jupyter_output><empty_output><jupyter_text>Step 4 - Separate data for training- Output:<jupyter_code># y = labels (the last column)
class_list = df.iloc[:,-1]
converter = LabelEncoder()
y = converter.fit_transform(class_list)
# labels -> class 0 to 44 
print("y :", y)<jupyter_output>y : [ 0  0  0 ... 44 44 44]<jupyter_text>- Inputs:<jupyter_code># inputs of the neural network
print("x :", df.iloc[:,:-1])
fit = StandardScaler()

# input: from column 1 to 26
X = fit.fit_transform(np.array(df.iloc[:, 1:26], dtype=float))
X<jupyter_output><empty_output><jupyter_text>- Split dataset for training:<jupyter_code># split dataset for training
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)<jupyter_output><empty_output><jupyter_text>Step 5 - Define the neural network model<jupyter_code># build the model
model = tf.keras.models.Sequential([
    tf.keras.layers.Dense(512, activation = 'relu', input_shape = (X_train.shape[1],)),
    tf.keras.layers.Dropout(0.2),
    
    tf.keras.layers.Dense(256, activation = 'relu'),
    keras.layers.Dropout(0.2),
    
    tf.keras.layers.Dense(128, activation = 'relu'),
    tf.keras.layers.Dropout(0.2),
    
    tf.keras.layers.Dense(64, activation = 'relu'),
    tf.keras.layers.Dropout(0.2),
    
    tf.keras.layers.Dense(45, activation = 'softmax'),
])

# display the model summary
print(model.summary())<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 512)               13312     
_________________________________________________________________
dropout (Dropout)            (None, 512)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 256)               131328    
_________________________________________________________________
dropout_1 (Dropout)          (None, 256)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 128)               32896     
_________________________________________________________________
dropout_2 (Dropout)          (None, 128)               0         
________________________________________________________[...]<jupyter_text>Step 6 - Train model<jupyter_code>def trainModel(model,epochs, optimizer):
    batch_size = 128
    model.compile(optimizer = optimizer, loss = 'sparse_categorical_crossentropy', metrics = 'accuracy')
    return model.fit(X_train, y_train, validation_data = (X_test, y_test), epochs = epochs, batch_size = batch_size)
# launch training
model_history = trainModel(model = model, epochs = 300, optimizer = 'adam')
# loss curves
loss_train_curve = model_history.history["loss"]
loss_val_curve = model_history.history["val_loss"]
plt.plot(loss_train_curve, label = "Train")
plt.plot(loss_val_curve, label = "Validation")
plt.legend(loc = 'upper right')
plt.title("Loss")
plt.show()
# accuracy curves
acc_train_curve = model_history.history["accuracy"]
acc_val_curve = model_history.history["val_accuracy"]
plt.plot(acc_train_curve, label = "Train")
plt.plot(acc_val_curve, label = "Validation")
plt.legend(loc = 'lower right')
plt.title("Accuracy")
plt.show()<jupyter_output><empty_output><jupyter_text>Step 7 -  Evaluate results<jupyter_code>test_loss, test_acc = model.evaluate(X_test, y_test, batch_size = 128)
print("The test loss is: ", test_loss)
print("The best accuracy is: ", test_acc*100)<jupyter_output>20/20 [==============================] - 0s 3ms/step - loss: 0.2854 - accuracy: 0.9371
The test loss is:  0.2853870093822479
The best accuracy is:  93.71269345283508
{'fraction_non_alphanumeric': 0.11426813006462201, 'fraction_numerical': 0.02061749923069033, 'mean_word_length': 4.877034358047016, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 63, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '370935', 'license': 'Apache-2.0', 'n_tokens_mistral': 3484, 'n_tokens_neox': 3176, 'n_words': 738, 'path': 'notebooks/tensorflow/tuto/notebook-marine-sound-classification.ipynb', 'repo_name': 'eleapttn/ai-training-examples', 'size': '9749'}