<jupyter_start><jupyter_text>Collision Avoidance - Live DemoIn this notebook we'll use the model we trained to detect whether the robot is ``free`` or ``blocked`` to enable a collision avoidance behavior on the robot.   Load the trained modelWe'll assumed that you've already downloaded the ``collision_avoidance.9999999999.h5`` to your Jetbot.  > Please make sure the file has uploaded fully before calling the next cellExecute the code below to initialize the TensorFlow model.<jupyter_code>try:
  # Use the %tensorflow_version magic if in colab.
  %tensorflow_version 2.x
except Exception:
    print('passing')
    pass<jupyter_output>passing<jupyter_text>Import the necessary packages<jupyter_code>from __future__ import absolute_import, division, print_function, unicode_literals
import time
import numpy as np
import matplotlib.pylab as plt

import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_datasets as tfds
from tensorflow.keras import layers<jupyter_output><empty_output><jupyter_text>Next, load the trained model from the ``collision_avoidance.9999999999.h5`` file that you uploaded<jupyter_code>model_path = '/home/jetbot/Notebooks/my_collision_avoidance/collision_avoidance.1577936737.h5'
reloaded = tf.keras.models.load_model(
  model_path, 
  # `custom_objects` tells keras how to load a `hub.KerasLayer`
  custom_objects={'KerasLayer': hub.KerasLayer})

reloaded.summary()
IMAGE_RES = 224
import numpy as np
import PIL.Image as Image

test_image = tf.keras.utils.get_file('image.jpg','file:///home/jetbot/Notebooks/my_collision_avoidance/content/free_blocked/validation/blocked/00ad54cc-2ccd-11ea-ae3b-72b5f773b75d.jpg')
test_image = Image.open(test_image).resize((IMAGE_RES, IMAGE_RES))
test_image 
test_image = np.array(test_image)/255.0
test_image.shape
result = reloaded.predict(test_image[np.newaxis, ...])
result.shape
from tensorflow.keras.preprocessing import image
test_image = image.load_img('/home/jetbot/Notebooks/my_collision_avoidance/content/free_blocked/validation/blocked/00ad54cc-2ccd-11ea-ae3b-72b5f773b75d.jpg', 
                            target_size=(IMAGE_RES,IMAGE_RES))

test_image = image.img_to_array(test_image)
#test_image = np.expand_dims(test_image, axis=0)
test_image = np.array(test_image)/255.0
#test_image = test_image.reshape(IMAGE_RES, IMAGE_RES,3)
result = reloaded.predict(test_image[np.newaxis, ...])
print result
model_dir = '/home/jetbot/Notebooks/my_collision_avoidance/collision_avoidance.1577941915'
reload_sm_keras = tf.keras.models.load_model(
  model_dir,
  custom_objects={'KerasLayer': hub.KerasLayer})

reload_sm_keras.summary()
result = reload_sm_keras.predict(test_image[np.newaxis, ...])
result.shape
reloaded_sm = tf.saved_model.load(model_dir)
reloaded_sm.summary()<jupyter_output><empty_output><jupyter_text>Currently, the model weights are located on the CPU memory execute the code below to transfer to the GPU device. Great! We've now defined our pre-processing function which can convert images from the camera format to the neural network input format.Now, let's start and display our camera.  We'll also create a slider that will display the probability that the robot is blocked.<jupyter_code>import traitlets
from IPython.display import display
import ipywidgets.widgets as widgets
from jetbot import Camera, bgr8_to_jpeg

camera = Camera.instance(width=224, height=224)
image = widgets.Image(format='jpeg', width=224, height=224)
blocked_slider = widgets.FloatSlider(description='blocked', min=0.0, max=1.0, orientation='vertical')

camera_link = traitlets.dlink((camera, 'value'), (image, 'value'), transform=bgr8_to_jpeg)

display(widgets.HBox([image, blocked_slider]))<jupyter_output><empty_output><jupyter_text>We'll also create our robot instance which we'll need to drive the motors.<jupyter_code>from jetbot import Robot

robot = Robot()<jupyter_output><empty_output><jupyter_text>Next, we'll create a function that will get called whenever the camera's value changes.  This function will do the following steps1. Pre-process the camera image2. Execute the neural network3. While the neural network output indicates we're blocked, we'll turn left, otherwise we go forward.<jupyter_code>import torch.nn.functional as F
import time

def update(change):
    global blocked_slider, robot
    x = change['new'] 
    x = preprocess(x)
    y = model(x)
    
    # we apply the `softmax` function to normalize the output vector so it sums to 1 (which makes it a probability distribution)
    y = F.softmax(y, dim=1)
    
    prob_blocked = float(y.flatten()[0])
    
    blocked_slider.value = prob_blocked
    
    if prob_blocked < 0.5:
        robot.forward(0.4)
    else:
        robot.left(0.4)
    
    time.sleep(0.001)
        
update({'new': camera.value})  # we call the function once to intialize<jupyter_output><empty_output><jupyter_text>Cool! We've created our neural network execution function, but now we need to attach it to the camera for processing. We accomplish that with the ``observe`` function.> WARNING: This code will move the robot!! Please make sure your robot has clearance.  The collision avoidance should work, but the neural> network is only as good as the data it's trained on!<jupyter_code>camera.observe(update, names='value')  # this attaches the 'update' function to the 'value' traitlet of our camera<jupyter_output><empty_output><jupyter_text>Awesome! If your robot is plugged in it should now be generating new commands with each new camera frame.  Perhaps start by placing your robot on the ground and seeing what it does when it reaches an obstacle.If you want to stop this behavior, you can unattach this callback by executing the code below.<jupyter_code>camera.unobserve(update, names='value')
robot.stop()<jupyter_output><empty_output><jupyter_text>Perhaps you want the robot to run without streaming video to the browser.  You can unlink the camera as below.<jupyter_code>camera_link.unlink()  # don't stream to browser (will still run camera)<jupyter_output><empty_output><jupyter_text>To continue streaming call the following.<jupyter_code>camera_link.link()  # stream to browser (wont run camera)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08789252185173195, 'fraction_numerical': 0.019909355778569116, 'mean_word_length': 5.5664187035069075, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '355988', 'license': 'MIT', 'n_tokens_mistral': 2016, 'n_tokens_neox': 1905, 'n_words': 576, 'path': 'tf_live_demo.ipynb', 'repo_name': 'skarlekar/ml-exercises', 'size': '6178'}