<jupyter_start><jupyter_text>"Engine knock detection AI part 4/5"
> "Testing the trained model on a previously unheard engine"
- image: images/knock_ai_4_5_tohatsu.png > Important: This notebook is not guaranteed to work without modification. It's a while since I wrote this and haven't tested it since. For a summary of this project see [Engine knock detection AI part 5/5](http://niklasekman.com/2021/02/10/Train-model-updated.html)  Method  Setup<jupyter_code>!curl -s https://course.fast.ai/setup/colab | bash
#collapse-output
from google.colab import drive
drive.mount('/content/drive')
from fastai.vision import *
path = Path('/content/drive/My Drive/Colab Notebooks/fast.ai/KnockKnock/data')
defaults.device = torch.device('cpu')<jupyter_output><empty_output><jupyter_text>Loading the previously trained model<jupyter_code>classes = ['knocking','normal']
#collapse-output
data = ImageDataBunch.single_from_classes(path, classes, size=224).normalize(imagenet_stats)
learn = create_cnn(data,models.resnet34)
learn.load('stage-2')<jupyter_output><empty_output><jupyter_text>Using the model for prediciton  Loading a video file<jupyter_code>#collapse-output
!pip install moviepy<jupyter_output>Requirement already satisfied: moviepy in /usr/local/lib/python3.6/dist-packages (0.2.3.5)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from moviepy) (1.16.5)
Requirement already satisfied: decorator<5.0,>=4.0.2 in /usr/local/lib/python3.6/dist-packages (from moviepy) (4.4.0)
Requirement already satisfied: tqdm<5.0,>=4.11.2 in /usr/local/lib/python3.6/dist-packages (from moviepy) (4.28.1)
Requirement already satisfied: imageio<3.0,>=2.1.2 in /usr/local/lib/python3.6/dist-packages (from moviepy) (2.4.1)
Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from imageio<3.0,>=2.1.2->moviepy) (4.3.0)
Requirement already satisfied: olefile in /usr/local/lib/python3.6/dist-packages (from pillow->imageio<3.0,>=2.1.2->moviepy) (0.46)<jupyter_text>Extracting 2 seconds from the video file for analysis.<jupyter_code>project_path = '/content/drive/My Drive/Colab Notebooks/fast.ai/KnockKnock'
mov_filename = 'IMG_3414.MOV'

import moviepy.editor as mp
clip = mp.VideoFileClip(project_path + '/' + mov_filename).subclip(4,6); clip

clip.ipython_display(width=360)<jupyter_output><empty_output><jupyter_text>> youtube: https://youtu.be/Er0m4sy7ZyM  Generating a spectrogram Preparing the audio track from the video for librosa.<jupyter_code>sr = clip.audio.fps
y = clip.audio.to_soundarray()
y = y[...,0:1:1].flatten() # Take one channel, transform rows to columns, 1D<jupyter_output><empty_output><jupyter_text>The following is the same process which was done when creating the training dataset.<jupyter_code>import librosa
D = librosa.stft(y)
D_harmonic, D_percussive = librosa.decompose.hpss(D)
# Pre-compute a global reference power from the input spectrum
rp = np.max(np.abs(D))
import matplotlib.pyplot as plt
import librosa.display

side_px=256
dpi=150
plot = plt.figure(figsize=(side_px/dpi, side_px/dpi))

CQT = librosa.amplitude_to_db(np.abs(librosa.cqt(librosa.istft(D_percussive), sr=sr)), ref=np.max)
p=librosa.display.specshow(CQT,x_axis=None,y_axis=None)
plt.axis('off')<jupyter_output><empty_output><jupyter_text>The image is actually a matplotlib figure. The predict function of the fastai learner expects to be given a tensor to work with. Could be that the conversion had been easier if the spectrogram was saved out to a file first and then loaded from disk. The idea is to deploy the model later on a web server and it's best to avoid unnecessary disk access for temporary files.<jupyter_code>type(plot)<jupyter_output><empty_output><jupyter_text>A little googling turned up a short tutorial on turning a figure into a PIL RGBA image. The idea being that the PIL image could then be converted to a tensor. The original URL in the comment below is now broken but as luck would have it the wayback machine has a snapshot of it [here](https://web.archive.org/web/20200712151644/http://www.icare.univ-lille1.fr/tutorials/convert_a_matplotlib_figure).<jupyter_code>#collapse-output
!pip install Pillow
# http://www.icare.univ-lille1.fr/tutorials/convert_a_matplotlib_figure
def fig2data ( fig ):
    """
    @brief Convert a Matplotlib figure to a 4D numpy array with RGBA channels and return it
    @param fig a matplotlib figure
    @return a numpy 3D array of RGBA values
    """
    # draw the renderer
    fig.canvas.draw ( )
 
    # Get the RGBA buffer from the figure
    w,h = fig.canvas.get_width_height()
    buf = np.fromstring ( fig.canvas.tostring_argb(), dtype=np.uint8 )
    buf.shape = ( w, h,4 )
 
    # canvas.tostring_argb give pixmap in ARGB mode. Roll the ALPHA channel to have it in RGBA mode
    buf = np.roll ( buf, 3, axis = 2 )
    return buf
from PIL import Image as pil_Image

# http://www.icare.univ-lille1.fr/tutorials/convert_a_matplotlib_figure
def fig2img ( fig ):
    """
    @brief Convert a Matplotlib figure to a PIL Image in RGBA format and return it
    @param fig a matplotlib figure
    @return a Python Imaging Library ( PIL ) image
    """
    # put the figure pixmap into a numpy array
    buf = fig2data ( fig )
    w, h, d = buf.shape
    return pil_Image.frombytes( "RGBA", ( w ,h ), buf.tostring( ) )<jupyter_output><empty_output><jupyter_text>Converting the figure to an image<jupyter_code>im = fig2img(plot)
im<jupyter_output><empty_output><jupyter_text>and turning the image into a tensor.<jupyter_code>from PIL import ImageMath

im_float = ImageMath.eval("convert(a,'F')", a=im)
im_tensor = pil2tensor(im_float, np.float32)

im_fastai = Image(px=im_tensor)<jupyter_output><empty_output><jupyter_text>Predicting the state of the engine Finally the tensor can be given to fastai. The model seems to think that the engine runs normally.<jupyter_code>pred_class,pred_idx,outputs = learn.predict(im_fastai)
pred_class<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1068676716917923, 'fraction_numerical': 0.022948073701842545, 'mean_word_length': 5.130390143737166, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 55, 'https://': 3, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '180535', 'license': 'Apache-2.0', 'n_tokens_mistral': 2128, 'n_tokens_neox': 2041, 'n_words': 563, 'path': '_notebooks/2019-09-16-Test_model_commented.ipynb', 'repo_name': 'niklasekman/my_fastpages', 'size': '5970'}