<jupyter_start><jupyter_text>Multi-Domain: Audio ModelThe notebook shows the models and training process for the audio-side of the multi-domain model.<jupyter_code>!apt-get install -y xxd
import random

import tensorflow as tf
import numpy as np
from google.colab import drive 
drive.mount('/content/gdrive', force_remount=True)<jupyter_output>Mounted at /content/gdrive<jupyter_text>DataThe audio data is taken from the same video as the image data. The following preprocessing steps were taken to ensure consistency between all data samples.- Setting the same sample rate- Setting the same audio length- Conversion into mfccs with `python-speech-features` packageAfter the preprocessing the data was saved into numpy arrays ready for training.<jupyter_code>!cp gdrive/MyDrive/multi_domain/audio_dataset.npz .
dataset = np.load("audio_dataset.npz")

x_train = dataset['x_train']
y_train = dataset['y_train']
x_val = dataset['x_val']
y_val = dataset['y_val']<jupyter_output><empty_output><jupyter_text>Set the output values to be 1 if "happy" and 0 if "angry".<jupyter_code>y_train = np.array([1 if y == 'happy' else 0 for y in y_train])
y_val = np.array([1 if y == 'happy' else 0 for y in y_val])<jupyter_output><empty_output><jupyter_text>To be used by a CNN, the input arrays need to be reshaped to be similar to an image. One way to look at this is to imaging the mfccs as a greyscale image.<jupyter_code>x_train.shape
x_train = x_train.reshape(x_train.shape[0], 
                          x_train.shape[1], 
                          x_train.shape[2], 
                          1)
x_val = x_val.reshape(x_val.shape[0], 
                      x_val.shape[1], 
                      x_val.shape[2], 
                      1)
x_train.shape
#create a list of tuples
c = list(zip(x_train, y_train))
#shuffle the tuples
random.shuffle(c)
#return back to x_train and y_train
x_train, y_train = zip(*c)
x_train = np.array(x_train)
y_train = np.array(y_train)<jupyter_output><empty_output><jupyter_text>ModellingThe next part is to create the neural network to fit to the images. Because this is a simple problem (binary classification) a model is built from scratch. However Tensorfow and Keras have many pretrained models which can be adapted to your problem. Prequantized models can also be found on the [TensorFlow Hub](https://tfhub.dev/s?q=quantized). When creating a model for a microcontroller you need to think more carefully about your model selection. A few important points:- Are the layers supported by TensorFlow Lite for Microcontrollers?- Is the model too big?- Is there a more efficient architectureFor example when running on a laptop you may create a really simple dense network by flattening the image. This will result in too many weights for a microcontroller and waste precious memory. The model in this example is a simple feed-forward convolutional network which uses a sigmoid classifier to shift the output between 0 and 1 (like the y values we have).<jupyter_code>model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(32, kernel_size = (2,2), activation="relu", input_shape = (16,16,1)),
    tf.keras.layers.MaxPool2D(pool_size=(2,2)),
    tf.keras.layers.Conv2D(32, kernel_size = (2,2), activation="relu"),
    tf.keras.layers.MaxPool2D(pool_size=(2,2)),
    tf.keras.layers.Conv2D(64, kernel_size = (2,2), activation="relu"),
    tf.keras.layers.MaxPool2D(pool_size=(2,2)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(64, activation="relu"),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.Dense(1, activation = "sigmoid")
    ]
)
model.compile(loss='binary_crossentropy', 
              optimizer='rmsprop', 
              metrics=['acc'])
model.build((1,16,16,1))
model.summary()<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 15, 15, 32)        160       
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 7, 7, 32)          0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 6, 6, 32)          4128      
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 3, 3, 32)          0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 2, 2, 64)          8256      
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 1, 1, 64)          0         
________________________________________________________[...]<jupyter_text>TrainingBecause the model is so simple, the model is simple trainined for 30 epochs at the default learning rate for Keras's `'rmsprop'` optimization function.<jupyter_code>history = model.fit(x_train, 
                    y_train, 
                    epochs=30, 
                    batch_size=8, 
                    validation_data=(x_val, y_val))<jupyter_output>Epoch 1/30
6/6 [==============================] - 1s 92ms/step - loss: 1.0502 - acc: 0.4432 - val_loss: 0.6779 - val_acc: 0.5455
Epoch 2/30
6/6 [==============================] - 0s 10ms/step - loss: 0.7625 - acc: 0.3949 - val_loss: 0.6875 - val_acc: 0.4545
Epoch 3/30
6/6 [==============================] - 0s 12ms/step - loss: 0.7001 - acc: 0.4461 - val_loss: 0.6699 - val_acc: 0.5455
Epoch 4/30
6/6 [==============================] - 0s 11ms/step - loss: 0.6763 - acc: 0.5452 - val_loss: 0.6714 - val_acc: 0.5455
Epoch 5/30
6/6 [==============================] - 0s 11ms/step - loss: 0.6986 - acc: 0.5256 - val_loss: 0.7308 - val_acc: 0.4545
Epoch 6/30
6/6 [==============================] - 0s 10ms/step - loss: 0.7516 - acc: 0.4324 - val_loss: 0.6534 - val_acc: 0.5455
Epoch 7/30
6/6 [==============================] - 0s 10ms/step - loss: 0.5472 - acc: 0.7798 - val_loss: 0.6427 - val_acc: 1.0000
Epoch 8/30
6/6 [==============================] - 0s 10ms/step - loss: 0.5694 - acc: 0.6857 - val[...]<jupyter_text>Post-Training QuantizationThe model is quantized ready for use. This helps keep the model as small as possible. Here we use int8 quantization to keep it as small as possible.<jupyter_code>def representative_dataset():
  for data in x_train:
    yield [x_train.astype(np.float32)]
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8  # or tf.uint8
converter.inference_output_type = tf.int8  # or tf.uint8
quant_model = converter.convert()
with open("audio_model.tflite", "wb") as f:
  f.write(quant_model)
!xxd -i "audio_model.tflite" > "audio_model.cc"<jupyter_output><empty_output><jupyter_text>Once converted, the model can be exported as a .tflite. For tensorflow lite for microcontrollers. One final step is required and that is to convert the model into a .cc file using `xxd`.  The C source file contains the TensorFlow Lite model as a char array.  Test DataTo test the model on the microcontroller, we can create some test data which can be used to simulate the model.To do this we need to convert the float32 input data into int8 data using the conversion provided with the Tensorflow Lite for Microcontrollers<jupyter_code>interpreter = tf.lite.Interpreter("audio_model.tflite")
input_details = interpreter.get_input_details()
scale, zero = input_details[0]["quantization"]
test_x_data = (x_val[0] / scale + zero).astype(input_details[0]['dtype'])
test_x_data.tofile("x_audio_test.txt")
!xxd -i x_audio_test.txt > x_audio_test.cc<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13214513049013368, 'fraction_numerical': 0.04392106938255888, 'mean_word_length': 4.013401403956605, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 32, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '165636', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 2654, 'n_tokens_neox': 2410, 'n_words': 735, 'path': 'projects/multi_domain_pico/training/audio_train.ipynb', 'repo_name': 'henriwoodcock/developer', 'size': '7855'}