<jupyter_start><jupyter_text>DeepPoseKit Step 4b - Predict on new dataThis is step 4b of the example notebooks for using DeepPoseKit. This notebook shows you how to use your trained model to make predictions on a novel video, detect outliers, merge the outliers with the existing annotation set, and visualize the data output.**NOTE**: If you run into problems, you can help us improve DeepPoseKit by [opening an issue](https://github.com/jgraving/deepposekit/issues/new) or [submitting a pull request](https://help.github.com/en/articles/creating-a-pull-request-from-a-fork)**If you're using Colab**: make sure to go to the “Runtime” dropdown menu, select “Change runtime type” and select `GPU` in the "Hardware accelerator" drop-down menu If you haven't already installed DeepPoseKit you can run the next cell<jupyter_code>import sys
!{sys.executable} -m pip install -U deepposekit
import numpy as np
import matplotlib.pyplot as plt
import cv2
import glob

from deepposekit.models import load_model
from deepposekit.io import DataGenerator, VideoReader, VideoWriter
from deepposekit.io.utils import merge_new_images

import tqdm
import time

from scipy.signal import find_peaks

from os.path import expanduser
try:
    import google.colab
    IN_COLAB = True
except:
    IN_COLAB = False

HOME = expanduser("~") if not IN_COLAB else '.'<jupyter_output>/home/jake/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/jake/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/jake/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/jake/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWa[...]<jupyter_text>Use the next cell to download the example data into your home directory<jupyter_code>!git clone https://github.com/jgraving/deepposekit-data {HOME + '/deepposekit-data'}<jupyter_output>fatal: destination path '/home/jake/deepposekit-data' already exists and is not an empty directory.<jupyter_text>Find the saved models<jupyter_code>models = sorted(glob.glob(HOME + '/deepposekit-data/datasets/fly/*model*.h5'))
models<jupyter_output><empty_output><jupyter_text>Load the trained modelThis loads the trained model into memory for making predictions<jupyter_code>model = load_model(HOME + '/deepposekit-data/datasets/fly/best_model_densenet.h5')<jupyter_output>WARNING: Logging before flag parsing goes to stderr.
W0930 12:51:06.157776 140198169065280 deprecation.py:506] From /home/jake/.local/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0930 12:51:06.159418 140198169065280 deprecation.py:506] From /home/jake/.local/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0930 12:51:06.160889 140198169065280 deprecation.py:506] From /home/jake/.local/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:[...]<jupyter_text>Find the available videos<jupyter_code>videos = sorted(glob.glob(HOME + '/deepposekit-data/datasets/fly/*video.avi'))
videos<jupyter_output><empty_output><jupyter_text>Test the videoCheck to make sure `VideoReader` is correctly loading the video frames<jupyter_code>reader = VideoReader(HOME + '/deepposekit-data/datasets/fly/video.avi', batch_size=10, gray=True)
frames = reader[0]
reader.close()
plt.imshow(frames[0,...,0], cmap='gray')
plt.show()<jupyter_output><empty_output><jupyter_text>Makes predictions for the full videoThis loads batches of frames and makes predictions.<jupyter_code>reader = VideoReader(HOME + '/deepposekit-data/datasets/fly/video.avi', batch_size=50, gray=True)
predictions = model.predict(reader, verbose=1)
reader.close()<jupyter_output>675/675 [==============================] - 59s 87ms/step<jupyter_text>Save the dataOnce you have your predictions you can save it however you wish. Here is an example using numpy.save<jupyter_code>np.save(HOME + '/deepposekit-data/datasets/fly/predictions.npy', predictions)<jupyter_output><empty_output><jupyter_text>This splits the predictions into their x-y coordinates, and confidence scores from each confidence map.<jupyter_code>x, y, confidence = np.split(predictions, 3, -1)<jupyter_output><empty_output><jupyter_text>Visualize the data output<jupyter_code>data_generator = DataGenerator(HOME + '/deepposekit-data/datasets/fly/annotation_data_release.h5')

image = frames[0]
keypoints = predictions[0]

plt.figure(figsize=(5,5))
image = image if image.shape[-1] is 3 else image[..., 0]
cmap = None if image.shape[-1] is 3 else 'gray'
plt.imshow(image, cmap=cmap, interpolation='none')
for idx, jdx in enumerate(data_generator.graph):
    if jdx > -1:
        plt.plot(
            [keypoints[idx, 0], keypoints[jdx, 0]],
            [keypoints[idx, 1], keypoints[jdx, 1]],
            'r-'
        )
plt.scatter(keypoints[:, 0], keypoints[:, 1],
            c=np.arange(data_generator.keypoints_shape[0]),
            s=50, cmap=plt.cm.hsv, zorder=3)

plt.show()<jupyter_output>/home/jake/.local/lib/python3.6/site-packages/h5py/_hl/dataset.py:313: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.
  "Use dataset[()] instead.", H5pyDeprecationWarning)<jupyter_text>Detect outlier framesThis is a basic example of how to use confidence scores and temporal derivatives to detect potential outliers and add them to the annotation set. Plot the confidence scores<jupyter_code>confidence_diff = np.abs(np.diff(confidence.mean(-1).mean(-1)))

plt.figure(figsize=(15, 3))
plt.plot(confidence_diff)
plt.show()<jupyter_output><empty_output><jupyter_text>Use `scipy.signal.find_peaks` to detect outliers<jupyter_code>confidence_outlier_peaks = find_peaks(confidence_diff, height=0.1)[0]

plt.figure(figsize=(15, 3))
plt.plot(confidence_diff)
plt.plot(confidence_outlier_peaks, confidence_diff[confidence_outlier_peaks], 'ro')
plt.show()<jupyter_output><empty_output><jupyter_text>Calculate the keypoint derivatives and plot them<jupyter_code>time_diff = np.diff(predictions[..., :2], axis=0)
time_diff = np.abs(time_diff.reshape(time_diff.shape[0], -1))
time_diff = time_diff.mean(-1)
time_diff.shape
plt.figure(figsize=(15, 3))
plt.plot(time_diff)
plt.show()<jupyter_output><empty_output><jupyter_text>Use `scipy.signal.find_peaks` to detect outliers<jupyter_code>time_diff_outlier_peaks = find_peaks(time_diff, height=10)[0]

plt.figure(figsize=(15, 3))
plt.plot(time_diff)
plt.plot(time_diff_outlier_peaks, time_diff[time_diff_outlier_peaks], 'ro')
plt.show()<jupyter_output><empty_output><jupyter_text>Combine the detected outliers into a single index<jupyter_code>outlier_index = np.concatenate((confidence_outlier_peaks, time_diff_outlier_peaks))
outlier_index = np.unique(outlier_index) # make sure there are no repeats<jupyter_output><empty_output><jupyter_text>Grab the frames and corresponding keypoints for the selected outliers<jupyter_code>reader = VideoReader(HOME + '/deepposekit-data/datasets/fly/video.avi', batch_size=1, gray=True)

outlier_images = []
outlier_keypoints = []
for idx in outlier_index:
    outlier_images.append(reader[idx])
    outlier_keypoints.append(predictions[idx])

outlier_images = np.concatenate(outlier_images)
outlier_keypoints = np.stack(outlier_keypoints)

reader.close()

outlier_images.shape, outlier_keypoints.shape<jupyter_output><empty_output><jupyter_text>Visualize the outlier frames and keypoint predictions<jupyter_code>data_generator = DataGenerator(HOME + '/deepposekit-data/datasets/fly/annotation_data_release.h5')

for idx in range(5):
    image = outlier_images[idx]
    keypoints = outlier_keypoints[idx]

    plt.figure(figsize=(5,5))
    image = image if image.shape[-1] is 3 else image[..., 0]
    cmap = None if image.shape[-1] is 3 else 'gray'
    plt.imshow(image, cmap=cmap, interpolation='none')
    for idx, jdx in enumerate(data_generator.graph):
        if jdx > -1:
            plt.plot(
                [keypoints[idx, 0], keypoints[jdx, 0]],
                [keypoints[idx, 1], keypoints[jdx, 1]],
                'r-'
            )
    plt.scatter(keypoints[:, 0], keypoints[:, 1],
                c=np.arange(data_generator.keypoints_shape[0]),
                s=50, cmap=plt.cm.hsv, zorder=3)

    plt.show()<jupyter_output><empty_output><jupyter_text>Merge outliers with the annotation setHere we'll use a utility function `merge_new_images` to merge the outliers with our existing annotation set. You can then go annotate them with `deepposekit.annotate.Annotator`. Make sure to use the merged output file!<jupyter_code>merge_new_images(
    datapath=HOME + '/deepposekit-data/datasets/fly/annotation_data_release.h5',
    merged_datapath=HOME + '/deepposekit-data/datasets/fly/annotation_data_release_merged.h5',
    images=outlier_images,
    keypoints=outlier_keypoints,
    # overwrite=True # This overwrites the merged dataset if it already exists
)<jupyter_output><empty_output><jupyter_text>Load the data with `DataGenerator` and check that the merged data are there<jupyter_code>merged_generator = DataGenerator(HOME + '/deepposekit-data/datasets/fly/annotation_data_release_merged.h5', mode="unannotated")

image, keypoints = merged_generator[0]

plt.figure(figsize=(5,5))
image = image[0] if image.shape[-1] is 3 else image[0, ..., 0]
cmap = None if image.shape[-1] is 3 else 'gray'
plt.imshow(image, cmap=cmap, interpolation='none')
for idx, jdx in enumerate(data_generator.graph):
    if jdx > -1:
        plt.plot(
            [keypoints[0, idx, 0], keypoints[0, jdx, 0]],
            [keypoints[0, idx, 1], keypoints[0, jdx, 1]],
            'r-'
        )
plt.scatter(keypoints[0, :, 0], keypoints[0, :, 1], c=np.arange(data_generator.keypoints_shape[0]), s=50, cmap=plt.cm.hsv, zorder=3)

plt.show()<jupyter_output><empty_output><jupyter_text>Visualize the data as a videoThis is an example of how to visualize the predicted posture data on the original video.<jupyter_code># TODO: Turn this into a function

data_generator = DataGenerator(HOME + '/deepposekit-data/datasets/fly/annotation_data_release.h5')
predictions = predictions[...,:2]
predictions *= 2

resized_shape = (data_generator.image_shape[0]*2, data_generator.image_shape[1]*2)
cmap = plt.cm.hsv(np.linspace(0, 1, data_generator.keypoints_shape[0]))[:, :3][:, ::-1] * 255

writer = VideoWriter(HOME + '/fly_posture.mp4', (192*2,192*2), 'MP4V', 30.0)
reader = VideoReader(HOME + '/deepposekit-data/datasets/fly/video.avi', batch_size=1)

for frame, keypoints in tqdm.tqdm(zip(reader, predictions)):
    frame = frame[0]
    frame = frame.copy()
    frame = cv2.resize(frame, resized_shape)
    for idx, node in enumerate(data_generator.graph):
        if node >= 0:
            pt1 = keypoints[idx]
            pt2 = keypoints[node]
            cv2.line(frame, (pt1[0], pt1[1]), (pt2[0], pt2[1]), (0,0,255), 2, cv2.LINE_AA)
    for idx, keypoint in enumerate(keypoints):
        keypoint = keypoint.astype(int)
        cv2.circle(frame, (keypoint[0], keypoint[1]), 5, tuple(cmap[idx]), -1, lineType=cv2.LINE_AA)
        
    writer.write(frame)

writer.close()
reader.close()
plt.imshow(frame[...,::-1])
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12611142833836364, 'fraction_numerical': 0.028387307284444083, 'mean_word_length': 5.242362525458248, 'pattern_counts': {'":': 0, '<': 76, '<?xml version=': 0, '>': 80, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '368293', 'license': 'Apache-2.0', 'n_tokens_mistral': 4556, 'n_tokens_neox': 4154, 'n_words': 977, 'path': 'examples/step4b_predict_new_data.ipynb', 'repo_name': 'sugyan/DeepPoseKit', 'size': '12259'}