title english speaker accent recognition using transfer learning fadi badinehttps twitter comfadibadine converted to keras 3 by fadi badinehttps twitter comfadibadine date created 20220416 last modified 20230719 description training a model to classify uk ireland accents using feature extraction from yamnet accelerator gpu introduction the following example shows how to use feature extraction in order to train a model to classify the english accent spoken in an audio wave instead of training a model from scratch transfer learning enables us to take advantage of existing stateoftheart deep learning models and use them as feature extractors our process use a tf hub pretrained model yamnet and apply it as part of the tf data pipeline which transforms the audio files into feature vectors train a dense model on the feature vectors use the trained model for inference on a new audio file note we need to install tensorflow io in order to resample audio files to 16 khz as required by yamnet model in the test section ffmpeg is used to convert the mp3 file to wav you can install tensorflow io with the following command shell pip install u q tensorflowio configuration location where the dataset will be downloaded by default none keras utils getfile will use keras as the cachedir the location of the dataset list of datasets compressed files that contain the audio files we see that there are 2 compressed files for each accent except irish one for male speakers one for female speakers however we will be using a gender agnostic dataset list of gender agnostic categories imports set all random seeds in order to get reproducible results where to download the dataset yamnet model yamnet is an audio event classifier trained on the audioset dataset to predict audio events from the audioset ontology it is available on tensorflow hub yamnet accepts a 1d tensor of audio samples with a sample rate of 16 khz as output the model returns a 3tuple scores of shape n 521 representing the scores of the 521 classes embeddings of shape n 1024 the logmel spectrogram of the entire audio frame we will use the embeddings which are the features extracted from the audio samples as the input to our dense model for more detailed information about yamnet please refer to its tensorflow hubhttps tfhub devgoogleyamnet1 page dataset the dataset used is the crowdsourced highquality uk and ireland english dialect speech data sethttps openslr org83 which consists of a total of 17 877 highquality audio wav files this dataset includes over 31 hours of recording from 120 volunteers who selfidentify as native speakers of southern england midlands northern england wales scotland and ireland for more info please refer to the above link or to the following paper opensource multispeaker corpora of the english accents in the british isleshttps aclanthology org2020 lrec1 804 pdf download the data csv file that contains information about the dataset for each entry we have id wav file name transcript download the list of compressed files that contain the audio wav files load the data in a dataframe of the 3 columns id filename and transcript we are only interested in the filename column in order to read the audio file we will ignore the other two let s now preprocess the dataset by adjusting the filename removing a leading space adding wav extension to the filename creating a label using the first 2 characters of the filename which indicate the accent shuffling the samples the purpose of this function is to preprocess the dataframe by applying the following cleaning the filename from a leading space generating a label column that is gender agnostic i e welsh english male and welsh english female for example are both labeled as welsh english add extension wav to the filename shuffle samples remove leading space in filename column create gender agnostic labels based on the filename first 2 letters add the file path to the name shuffle the samples prepare training validation sets let s split the samples creating training and validation sets prepare a tensorflow dataset next we need to create a tf data dataset this is done by creating a dataframetodataset function that does the following create a dataset using filenames and labels get the yamnet embeddings by calling another function filepathtoembeddings apply caching reshuffling and setting batch size the filepathtoembeddings does the following load audio file resample audio to 16 khz generate scores and embeddings from yamnet model since yamnet generates multiple samples for each audio file this function also duplicates the label for all the generated samples that have score0 speech whereas sets the label for the others as other indicating that this audio segment is not a speech and we won t label it as one of the accents the below load16kaudiofile is copied from the following tutorial transfer learning with yamnet for environmental sound classificationhttps www tensorflow orgtutorialsaudiotransferlearningaudio read file content decode audio wave resample to 16k load 16k audio wave get audio embeddings scores the embeddings are the audio features extracted using transfer learning while scores will be used to identify time slots that are not speech which will then be gathered into a specific new category other number of embeddings in order to know how many times to repeat the label change labels for timeslots that are not speech into a new category other using onehot in order to use auc build the model the model that we use consists of an input layer which is the embedding output of the yamnet classifier 4 dense hidden layers and 4 dropout layers an output dense layer the model s hyperparameters were selected using kerastunerhttps keras iokerastuner class weights calculation since the dataset is quite unbalanced we wil use classweight argument during training getting the class weights is a little tricky because even though we know the number of audio files for each class it does not represent the number of samples for that class since yamnet transforms each audio file into multiple audio samples of 0 96 seconds each so every audio file will be split into a number of samples that is proportional to its length therefore to get those weights we have to calculate the number of samples for each class after preprocessing through yamnet callbacks we use keras callbacks in order to stop whenever the validation auc stops improving save the best model call tensorboard in order to later view the training and validation logs training results let s plot the training and validation auc and accuracy evaluation let s try to compare our model s performance to yamnet s using one of yamnet metrics dprime yamnet achieved a dprime value of 2 318 let s check our model s performance the following function calculates the dprime score from the auc we can see that the model achieves the following results results training validation accuracy 54 51 auc 0 91 0 89 dprime 1 882 1 740 confusion matrix let s now plot the confusion matrix for the validation dataset the confusion matrix lets us see for every class not only how many samples were correctly classified but also which other classes were the samples confused with it allows us to calculate the precision and recall for every class create x and y tensors generate predictions calculate confusion matrix plot the confusion matrix precision recall for every class recall is the ratio of correctly classified samples i e it shows how many samples of this specific class the model is able to detect it is the ratio of diagonal elements to the sum of all elements in the row precision shows the accuracy of the classifier it is the ratio of correctly predicted samples among the ones classified as belonging to this class it is the ratio of diagonal elements to the sum of all elements in the column run inference on test data let s now run a test on a single audio file let s check this example from the scottish voicehttps www thescottishvoice org ukhome we will download the mp3 file convert it to a 16k wav file run the model on the wav file plot the results the below function yamnetclassnamesfromcsv was copied and very slightly changed from this yamnet notebookhttps colab research google comgithubtensorflowhubblobmasterexamplescolabyamnet ipynb returns list of class names corresponding to score vector yamnetclassmapcsv io stringioyamnetclassmapcsvtext yamnetclassnames name for classindex mid name in csv readeryamnetclassmapcsv yamnetclassnames yamnetclassnames1 skip csv header return yamnetclassnames yamnetclassmappath yamnetmodel classmappath numpy yamnetclassnames yamnetclassnamesfromcsv tf io readfileyamnetclassmappath numpy decodeutf8 def calculatenumberofnonspeechscores numberofnonspeech tf math reducesum tf where tf math argmaxscores axis1 outputtypetf int32 0 1 0 return numberofnonspeech def filenametopredictionsfilename load 16k audio wave audiowav load16kaudiowavfilename get audio embeddings scores scores embeddings melspectrogram yamnetmodelaudiowav print out of samples are not speech format scores shape0 calculatenumberofnonspeechscores predict the output of the accent recognition model with embeddings as input predictions model predictembeddings return audiowav predictions melspectrogram audiowav predictions melspectrogram filenametopredictionsfilename inferedclass classnamespredictions meanaxis0 argmax printfthe main accent is inferedclass english audioaudiowav rate16000 plt figurefigsize10 6 plot the waveform plt subplot3 1 1 plt plotaudiowav plt xlim0 lenaudiowav plot the logmel spectrogram returned by the model plt subplot3 1 2 plt imshow melspectrogram numpy t aspectauto interpolationnearest originlower plot and label the model output scores for the topscoring classes meanpredictions np meanpredictions axis0 topclassindices np argsortmeanpredictions 1 plt subplot3 1 3 plt imshow predictions topclassindices t aspectauto interpolationnearest cmapgrayr patchpadding patchwindowseconds 2 patchhopseconds values from the model documentation patchpadding 0 025 2 0 01 plt xlimpatchpadding 0 5 predictions shape0 patchpadding 0 5 label the topn classes yticks range0 lenclassnames 1 plt yticksyticks classnamestopclassindicesx for x in yticks plt ylim0 5 np arraylenclassnames 0 introduction the following example shows how to use feature extraction in order to train a model to classify the english accent spoken in an audio wave instead of training a model from scratch transfer learning enables us to take advantage of existing state of the art deep learning models and use them as feature extractors our process use a tf hub pre trained model yamnet and apply it as part of the tf data pipeline which transforms the audio files into feature vectors train a dense model on the feature vectors use the trained model for inference on a new audio file note we need to install tensorflow io in order to resample audio files to 16 khz as required by yamnet model in the test section ffmpeg is used to convert the mp3 file to wav you can install tensorflow io with the following command shell pip install u q tensorflow_io configuration location where the dataset will be downloaded by default none keras utils get_file will use keras as the cache_dir the location of the dataset list of datasets compressed files that contain the audio files we see that there are 2 compressed files for each accent except irish one for male speakers one for female speakers however we will be using a gender agnostic dataset list of gender agnostic categories irish midlands northern scottish southern welsh imports set all random seeds in order to get reproducible results where to download the dataset yamnet model yamnet is an audio event classifier trained on the audioset dataset to predict audio events from the audioset ontology it is available on tensorflow hub yamnet accepts a 1 d tensor of audio samples with a sample rate of 16 khz as output the model returns a 3 tuple scores of shape n 521 representing the scores of the 521 classes embeddings of shape n 1024 the log mel spectrogram of the entire audio frame we will use the embeddings which are the features extracted from the audio samples as the input to our dense model for more detailed information about yamnet please refer to its tensorflow hub https tfhub dev google yamnet 1 page dataset the dataset used is the crowdsourced high quality uk and ireland english dialect speech data set https openslr org 83 which consists of a total of 17 877 high quality audio wav files this dataset includes over 31 hours of recording from 120 volunteers who self identify as native speakers of southern england midlands northern england wales scotland and ireland for more info please refer to the above link or to the following paper open source multi speaker corpora of the english accents in the british isles https aclanthology org 2020 lrec 1 804 pdf download the data csv file that contains information about the dataset for each entry we have id wav file name transcript download the list of compressed files that contain the audio wav files load the data in a dataframe of the 3 columns id filename and transcript we are only interested in the filename column in order to read the audio file we will ignore the other two let s now preprocess the dataset by adjusting the filename removing a leading space adding wav extension to the filename creating a label using the first 2 characters of the filename which indicate the accent shuffling the samples the purpose of this function is to preprocess the dataframe by applying the following cleaning the filename from a leading space generating a label column that is gender agnostic i e welsh english male and welsh english female for example are both labeled as welsh english add extension wav to the filename shuffle samples remove leading space in filename column create gender agnostic labels based on the filename first 2 letters add the file path to the name shuffle the samples prepare training validation sets let s split the samples creating training and validation sets prepare a tensorflow dataset next we need to create a tf data dataset this is done by creating a dataframe_to_dataset function that does the following create a dataset using filenames and labels get the yamnet embeddings by calling another function filepath_to_embeddings apply caching reshuffling and setting batch size the filepath_to_embeddings does the following load audio file resample audio to 16 khz generate scores and embeddings from yamnet model since yamnet generates multiple samples for each audio file this function also duplicates the label for all the generated samples that have score 0 speech whereas sets the label for the others as other indicating that this audio segment is not a speech and we won t label it as one of the accents the below load_16k_audio_file is copied from the following tutorial transfer learning with yamnet for environmental sound classification https www tensorflow org tutorials audio transfer_learning_audio read file content decode audio wave resample to 16k load 16k audio wave get audio embeddings scores the embeddings are the audio features extracted using transfer learning while scores will be used to identify time slots that are not speech which will then be gathered into a specific new category other number of embeddings in order to know how many times to repeat the label change labels for time slots that are not speech into a new category other using one hot in order to use auc build the model the model that we use consists of an input layer which is the embedding output of the yamnet classifier 4 dense hidden layers and 4 dropout layers an output dense layer the model s hyperparameters were selected using kerastuner https keras io keras_tuner class weights calculation since the dataset is quite unbalanced we wil use class_weight argument during training getting the class weights is a little tricky because even though we know the number of audio files for each class it does not represent the number of samples for that class since yamnet transforms each audio file into multiple audio samples of 0 96 seconds each so every audio file will be split into a number of samples that is proportional to its length therefore to get those weights we have to calculate the number of samples for each class after preprocessing through yamnet callbacks we use keras callbacks in order to stop whenever the validation auc stops improving save the best model call tensorboard in order to later view the training and validation logs training results let s plot the training and validation auc and accuracy evaluation let s try to compare our model s performance to yamnet s using one of yamnet metrics d prime yamnet achieved a d prime value of 2 318 let s check our model s performance the following function calculates the d prime score from the auc we can see that the model achieves the following results results training validation accuracy 54 51 auc 0 91 0 89 d prime 1 882 1 740 confusion matrix let s now plot the confusion matrix for the validation dataset the confusion matrix lets us see for every class not only how many samples were correctly classified but also which other classes were the samples confused with it allows us to calculate the precision and recall for every class create x and y tensors generate predictions calculate confusion matrix plot the confusion matrix precision recall for every class recall is the ratio of correctly classified samples i e it shows how many samples of this specific class the model is able to detect it is the ratio of diagonal elements to the sum of all elements in the row precision shows the accuracy of the classifier it is the ratio of correctly predicted samples among the ones classified as belonging to this class it is the ratio of diagonal elements to the sum of all elements in the column run inference on test data let s now run a test on a single audio file let s check this example from the scottish voice https www thescottishvoice org uk home we will download the mp3 file convert it to a 16k wav file run the model on the wav file plot the results the below function yamnet_class_names_from_csv was copied and very slightly changed from this yamnet notebook https colab research google com github tensorflow hub blob master examples colab yamnet ipynb returns list of class names corresponding to score vector skip csv header load 16k audio wave get audio embeddings scores predict the output of the accent recognition model with embeddings as input let s run the model on the audio file listen to the audio the below function was copied from this yamnet notebook tinyurl com 4a8xn7at and adjusted to our need this function plots the following audio waveform mel spectrogram predictions for every time step plot the waveform plot the log mel spectrogram returned by the model plot and label the model output scores for the top scoring classes patch_padding patch_window_seconds 2 patch_hop_seconds values from the model documentation label the top_n classes
SEED = 1337
EPOCHS = 1
BATCH_SIZE = 64
VALIDATION_RATIO = 0.1
MODEL_NAME = "uk_irish_accent_recognition"



CACHE_DIR = None


URL_PATH = "https://www.openslr.org/resources/83/"


zip_files = {
    0: "irish_english_male.zip",
    1: "midlands_english_female.zip",
    2: "midlands_english_male.zip",
    3: "northern_english_female.zip",
    4: "northern_english_male.zip",
    5: "scottish_english_female.zip",
    6: "scottish_english_male.zip",
    7: "southern_english_female.zip",
    8: "southern_english_male.zip",
    9: "welsh_english_female.zip",
    10: "welsh_english_male.zip",
}







gender_agnostic_categories = [
    "ir",  
    "mi",  
    "no",  
    "sc",  
    "so",  
    "we",  
]

class_names = [
    "Irish",
    "Midlands",
    "Northern",
    "Scottish",
    "Southern",
    "Welsh",
    "Not a speech",
]



import os

os.environ["KERAS_BACKEND"] = "tensorflow"

import io
import csv
import numpy as np
import pandas as pd
import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_io as tfio
import keras
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
from IPython.display import Audio



keras.utils.set_random_seed(SEED)


DATASET_DESTINATION = os.path.join(
    CACHE_DIR if CACHE_DIR else "~/.keras/", "datasets"
)



yamnet_model = hub.load("https://tfhub.dev/google/yamnet/1")









line_index_file = keras.utils.get_file(
    fname="line_index_file", origin=URL_PATH + "line_index_all.csv"
)


for i in zip_files:
    fname = zip_files[i].split(".")[0]
    url = URL_PATH + zip_files[i]

    zip_file = keras.utils.get_file(fname=fname, origin=url, extract=True)
    os.remove(zip_file)



dataframe = pd.read_csv(
    line_index_file,
    names=["id", "filename", "transcript"],
    usecols=["filename"],
)
dataframe.head()











def preprocess_dataframe(dataframe):
    
    dataframe["filename"] = dataframe.apply(
        lambda row: row["filename"].strip(), axis=1
    )

    
    dataframe["label"] = dataframe.apply(
        lambda row: gender_agnostic_categories.index(row["filename"][:2]),
        axis=1,
    )

    
    dataframe["filename"] = dataframe.apply(
        lambda row: os.path.join(DATASET_DESTINATION, row["filename"] + ".wav"),
        axis=1,
    )

    
    dataframe = dataframe.sample(frac=1, random_state=SEED).reset_index(
        drop=True
    )

    return dataframe


dataframe = preprocess_dataframe(dataframe)
dataframe.head()



split = int(len(dataframe) * (1 - VALIDATION_RATIO))
train_df = dataframe[:split]
valid_df = dataframe[split:]

print(
    f"We have {train_df.shape[0]} training samples & {valid_df.shape[0]} validation ones"
)




@tf.function
def load_16k_audio_wav(filename):
    
    file_content = tf.io.read_file(filename)

    
    audio_wav, sample_rate = tf.audio.decode_wav(
        file_content, desired_channels=1
    )
    audio_wav = tf.squeeze(audio_wav, axis=-1)
    sample_rate = tf.cast(sample_rate, dtype=tf.int64)

    
    audio_wav = tfio.audio.resample(
        audio_wav, rate_in=sample_rate, rate_out=16000
    )

    return audio_wav


def filepath_to_embeddings(filename, label):
    
    audio_wav = load_16k_audio_wav(filename)

    
    
    
    
    scores, embeddings, _ = yamnet_model(audio_wav)

    
    embeddings_num = tf.shape(embeddings)[0]
    labels = tf.repeat(label, embeddings_num)

    
    labels = tf.where(
        tf.argmax(scores, axis=1) == 0, label, len(class_names) - 1
    )

    
    return (embeddings, tf.one_hot(labels, len(class_names)))


def dataframe_to_dataset(dataframe, batch_size=64):
    dataset = tf.data.Dataset.from_tensor_slices(
        (dataframe["filename"], dataframe["label"])
    )

    dataset = dataset.map(
        lambda x, y: filepath_to_embeddings(x, y),
        num_parallel_calls=tf.data.experimental.AUTOTUNE,
    ).unbatch()

    return dataset.cache().batch(batch_size).prefetch(tf.data.AUTOTUNE)


train_ds = dataframe_to_dataset(train_df)
valid_ds = dataframe_to_dataset(valid_df)



keras.backend.clear_session()


def build_and_compile_model():
    inputs = keras.layers.Input(shape=(1024,), name="embedding")

    x = keras.layers.Dense(256, activation="relu", name="dense_1")(inputs)
    x = keras.layers.Dropout(0.15, name="dropout_1")(x)

    x = keras.layers.Dense(384, activation="relu", name="dense_2")(x)
    x = keras.layers.Dropout(0.2, name="dropout_2")(x)

    x = keras.layers.Dense(192, activation="relu", name="dense_3")(x)
    x = keras.layers.Dropout(0.25, name="dropout_3")(x)

    x = keras.layers.Dense(384, activation="relu", name="dense_4")(x)
    x = keras.layers.Dropout(0.2, name="dropout_4")(x)

    outputs = keras.layers.Dense(
        len(class_names), activation="softmax", name="ouput"
    )(x)

    model = keras.Model(
        inputs=inputs, outputs=outputs, name="accent_recognition"
    )

    model.compile(
        optimizer=keras.optimizers.Adam(learning_rate=1.9644e-5),
        loss=keras.losses.CategoricalCrossentropy(),
        metrics=["accuracy", keras.metrics.AUC(name="auc")],
    )

    return model


model = build_and_compile_model()
model.summary()



class_counts = tf.zeros(shape=(len(class_names),), dtype=tf.int32)

for x, y in iter(train_ds):
    class_counts = class_counts + tf.math.bincount(
        tf.cast(tf.math.argmax(y, axis=1), tf.int32), minlength=len(class_names)
    )

class_weight = {
    i: tf.math.reduce_sum(class_counts).numpy() / class_counts[i].numpy()
    for i in range(len(class_counts))
}

print(class_weight)



early_stopping_cb = keras.callbacks.EarlyStopping(
    monitor="val_auc", patience=10, restore_best_weights=True
)

model_checkpoint_cb = keras.callbacks.ModelCheckpoint(
    MODEL_NAME + ".keras", monitor="val_auc", save_best_only=True
)

tensorboard_cb = keras.callbacks.TensorBoard(
    os.path.join(os.curdir, "logs", model.name)
)

callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]



history = model.fit(
    train_ds,
    epochs=EPOCHS,
    validation_data=valid_ds,
    class_weight=class_weight,
    callbacks=callbacks,
    verbose=2,
)



fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(14, 5))

axs[0].plot(range(EPOCHS), history.history["accuracy"], label="Training")
axs[0].plot(range(EPOCHS), history.history["val_accuracy"], label="Validation")
axs[0].set_xlabel("Epochs")
axs[0].set_title("Training & Validation Accuracy")
axs[0].legend()
axs[0].grid(True)

axs[1].plot(range(EPOCHS), history.history["auc"], label="Training")
axs[1].plot(range(EPOCHS), history.history["val_auc"], label="Validation")
axs[1].set_xlabel("Epochs")
axs[1].set_title("Training & Validation AUC")
axs[1].legend()
axs[1].grid(True)

plt.show()



train_loss, train_acc, train_auc = model.evaluate(train_ds)
valid_loss, valid_acc, valid_auc = model.evaluate(valid_ds)





def d_prime(auc):
    standard_normal = stats.norm()
    d_prime = standard_normal.ppf(auc) * np.sqrt(2.0)
    return d_prime


print(
    "train d-prime: {0:.3f}, validation d-prime: {1:.3f}".format(
        d_prime(train_auc), d_prime(valid_auc)
    )
)






x_valid = None
y_valid = None

for x, y in iter(valid_ds):
    if x_valid is None:
        x_valid = x.numpy()
        y_valid = y.numpy()
    else:
        x_valid = np.concatenate((x_valid, x.numpy()), axis=0)
        y_valid = np.concatenate((y_valid, y.numpy()), axis=0)


y_pred = model.predict(x_valid)


confusion_mtx = tf.math.confusion_matrix(
    np.argmax(y_valid, axis=1), np.argmax(y_pred, axis=1)
)


plt.figure(figsize=(10, 8))
sns.heatmap(
    confusion_mtx,
    xticklabels=class_names,
    yticklabels=class_names,
    annot=True,
    fmt="g",
)
plt.xlabel("Prediction")
plt.ylabel("Label")
plt.title("Validation Confusion Matrix")
plt.show()



for i, label in enumerate(class_names):
    precision = confusion_mtx[i, i] / np.sum(confusion_mtx[:, i])
    recall = confusion_mtx[i, i] / np.sum(confusion_mtx[i, :])
    print(
        "{0:15} Precision:{1:.2f}%; Recall:{2:.2f}%".format(
            label, precision * 100, recall * 100
        )
    )



filename = "audio-sample-Stuart"
url = "https://www.thescottishvoice.org.uk/files/cm/files/"

if os.path.exists(filename + ".wav") == False:
    print(f"Downloading {filename}.mp3 from {url}")
    command = f"wget {url}{filename}.mp3"
    os.system(command)

    print(f"Converting mp3 to wav and resampling to 16 kHZ")
    command = (
        f"ffmpeg -hide_banner -loglevel panic -y -i {filename}.mp3 -acodec "
        f"pcm_s16le -ac 1 -ar 16000 {filename}.wav"
    )
    os.system(command)

filename = filename + ".wav"





def yamnet_class_names_from_csv(yamnet_class_map_csv_text):
    
    yamnet_class_map_csv = io.StringIO(yamnet_class_map_csv_text)
    yamnet_class_names = [
        name for (class_index, mid, name) in csv.reader(yamnet_class_map_csv)
    ]
    yamnet_class_names = yamnet_class_names[1:]  
    return yamnet_class_names


yamnet_class_map_path = yamnet_model.class_map_path().numpy()
yamnet_class_names = yamnet_class_names_from_csv(
    tf.io.read_file(yamnet_class_map_path).numpy().decode("utf-8")
)


def calculate_number_of_non_speech(scores):
    number_of_non_speech = tf.math.reduce_sum(
        tf.where(
            tf.math.argmax(scores, axis=1, output_type=tf.int32) != 0, 1, 0
        )
    )

    return number_of_non_speech


def filename_to_predictions(filename):
    
    audio_wav = load_16k_audio_wav(filename)

    
    scores, embeddings, mel_spectrogram = yamnet_model(audio_wav)

    print(
        "Out of {} samples, {} are not speech".format(
            scores.shape[0], calculate_number_of_non_speech(scores)
        )
    )

    
    predictions = model.predict(embeddings)

    return audio_wav, predictions, mel_spectrogram




audio_wav, predictions, mel_spectrogram = filename_to_predictions(filename)

infered_class = class_names[predictions.mean(axis=0).argmax()]
print(f"The main accent is: {infered_class} English")



Audio(audio_wav, rate=16000)



plt.figure(figsize=(10, 6))


plt.subplot(3, 1, 1)
plt.plot(audio_wav)
plt.xlim([0, len(audio_wav)])


plt.subplot(3, 1, 2)
plt.imshow(
    mel_spectrogram.numpy().T,
    aspect="auto",
    interpolation="nearest",
    origin="lower",
)


mean_predictions = np.mean(predictions, axis=0)

top_class_indices = np.argsort(mean_predictions)[::-1]
plt.subplot(3, 1, 3)
plt.imshow(
    predictions[:, top_class_indices].T,
    aspect="auto",
    interpolation="nearest",
    cmap="gray_r",
)



patch_padding = (0.025 / 2) / 0.01
plt.xlim([-patch_padding - 0.5, predictions.shape[0] + patch_padding - 0.5])

yticks = range(0, len(class_names), 1)
plt.yticks(yticks, [class_names[top_class_indices[x]] for x in yticks])
_ = plt.ylim(-0.5 + np.array([len(class_names), 0]))