<jupyter_start><jupyter_text>Autopilot - Data collectionSteps.1.Data collection2. Training3. DeploymentIn this case, we will do the same thing. However, in addition to classification, we will add, 'regression'-regression, which we will use to enable Jetbot to follow a path (actually any path or target point).1. Place the Jetbot at different locations on the path (offset from the center, different angles, etc.).2. Display the live camera input from Robot 3.3. Using the gamepad controller, place a “green dot” on the image that corresponds to the target direction we want the robot to move.4. Store the X, Y values of this green dot along with the image of the robot camera.Then, in the training notebook, we will train a neural network to predict the X, Y values of our labels. In the live demo, we will use the predicted X, Y values to calculate an approximate steering value (it is not an "exact" angle, because this requires image calibration, but it is roughly proportional to the angle, so our controller Will work normally).So, how to determine the target position of this example?Here are some guidelines that we think might be helpful:1. Look at the live video of the camera.2. Imagine the path that the robot should follow (try to get close to the distance it needs to avoid running off the road, etc.).3. Place the target as far as possible so that the robot can rush directly to the target without “running away” from the road.> For example, if we are on a very straight road, we can put it on the horizon. If we are making a sharp turn, it may need to be placed closer to the robot so that it does not run out of the border.Assuming our deep learning model works as expected, these markup guidelines should ensure the following:1. The robot can move directly to the target safely (not out of bounds, etc.)2. The goal will continue to move along the path we imagined  Data acquisition example video<jupyter_code>from IPython.display import HTML
HTML('<iframe width="560" height="315" src="https://www.youtube.com/embed/FW4En6LejhI" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>')<jupyter_output><empty_output><jupyter_text>Import library moduleImport all the libraries needed for Data Collection.We will primarily use OpenCV to visualize and save images with labels. Uuid, datetime and other libraries are used for image naming.<jupyter_code># IPython libraries be used to diplay and parts
import traitlets
import ipywidgets.widgets as widgets
from IPython.display import display
# Camera and servo interface foe Jetbot robot car
from jetbot import Robot, Camera, bgr8_to_jpeg
# Python base package for image annotations
from uuid import uuid1
import os
import json
import glob
import datetime
import numpy as np
import cv2
import time
from servoserial import ServoSerial
import threading
# Kill pthread
import inspect
import ctypes<jupyter_output><empty_output><jupyter_text>Display camera data in real timeOur neural network takes an image of 224x224 pixels as input. We set the camera to this parameter to minimize the file size of the dataset (we have tested it for this task).In some scenarios, it's best to collect the data at a larger image size and then reduce it to the desired size.<jupyter_code>camera = Camera()

image_widget = widgets.Image(format='jpeg', width=224, height=224)
target_widget = widgets.Image(format='jpeg', width=224, height=224)

x_slider = widgets.FloatSlider(min=-1.0, max=1.0, step=0.001, description='x')
y_slider = widgets.FloatSlider(min=-1.0, max=1.0, step=0.001, description='y')

def display_xy(camera_image):
    image = np.copy(camera_image)
    x = x_slider.value
    y = y_slider.value
    x = int(x * 224 / 2 + 112)
    y = int(y * 224 / 2 + 112)
    image = cv2.circle(image, (x, y), 8, (0, 255, 0), 3)
    image = cv2.circle(image, (112, 224), 8, (0, 0,255), 3)
    image = cv2.line(image, (x,y), (112,224), (255,0,0), 3)
    jpeg_image = bgr8_to_jpeg(image)
    return jpeg_image

time.sleep(1)
traitlets.dlink((camera, 'value'), (image_widget, 'value'), transform=bgr8_to_jpeg)
traitlets.dlink((camera, 'value'), (target_widget, 'value'), transform=display_xy)

display(widgets.HBox([image_widget, target_widget]), x_slider, y_slider)<jupyter_output><empty_output><jupyter_text>Create a handle controller instanceCreate an instance of the Controller widget, which we will use to mark the image with "x" and "y" values, as described in the introduction. The Controller widget accepts an index parameter that specifies the number of controllers. This is useful if you have multiple controllers, or if some gamepads appear as multiple controllers. To determine the index of the controller we are using, then before we create the handle instance we will follow the steps we have just learned to use the remote control handle:1. Visit http://html5gamepad.com2. Press the button on the gamepad you are using3. Remember the index of the gamepad that responds to the buttonThen, we will use this index to create and display the controller.<jupyter_code>controller = widgets.Controller(index=0)
robot = Robot()
display(controller)<jupyter_output><empty_output><jupyter_text>Connect the Gamepad controller to the label image.<jupyter_code>widgets.jsdlink((controller.axes[2], 'value'), (x_slider, 'value'))
widgets.jsdlink((controller.axes[3], 'value'), (y_slider, 'value'))<jupyter_output><empty_output><jupyter_text>Data collectionThe code below will display the live image feed and the number of images we saved.We store the value of the target X, Y:1. Put the green dot on the target2. Press the 13th button to saveThen the data we want will be saved to the ``dataset_xy`` folder. The saved file naming format is:``xy___.jpg``When we train, we load the image and parse the x and y values in the file name.<jupyter_code>DATASET_DIR = 'dataset_xy'

# we have this "try/except" statement because these next functions can throw an error if the directories exist already
try:
    os.makedirs(DATASET_DIR)
except FileExistsError:
    print('Directories not created becasue they already exist')

for b in controller.buttons:
    b.unobserve_all()

count_widget = widgets.IntText(description='count', value=len(glob.glob(os.path.join(DATASET_DIR, '*.jpg'))))

def xy_uuid(x, y):
    return 'xy_%03d_%03d_%s' % (x * 50 + 50, y * 50 + 50, uuid1())

def save_snapshot(change):
    if change['new']:
        uuid = xy_uuid(x_slider.value, y_slider.value)
        image_path = os.path.join(DATASET_DIR, uuid + '.jpg')
        with open(image_path, 'wb') as f:
            f.write(image_widget.value)
        count_widget.value = len(glob.glob(os.path.join(DATASET_DIR, '*.jpg')))

# Jetbot Yahboom Handle L side No.1 key save data
controller.buttons[4].observe(save_snapshot, names='value')

display(widgets.VBox([
    target_widget,
    count_widget
]))<jupyter_output><empty_output><jupyter_text>For ease to using, we can open a new thread to control the Jetbot robot car by handled to collect data through the handle.<jupyter_code>def jetbot_motion():
    while 1:
        #Robot car left and right DC motor
        if controller.axes[1].value <= 0.1:
            if (controller.axes[0].value <= 0.1 and controller.axes[0].value >= -0.1 
                and controller.axes[1].value <= 0.1 and controller.axes[1].value >= -0.1):
                robot.stop()
            else:
                robot.set_motors(-controller.axes[1].value + controller.axes[0].value, -controller.axes[1].value - controller.axes[0].value)
            
            time.sleep(0.01)
        else:
            robot.set_motors(-controller.axes[1].value - controller.axes[0].value, -controller.axes[1].value + controller.axes[0].value)
            time.sleep(0.01)
          #Handle control code---2(Xbox360 Handle)
#         if controller.axes[1].value <= 0:
#             robot.set_motors(-controller.axes[1].value + controller.axes[0].value, -controller.axes[1].value - controller.axes[0].value)
#             time.sleep(0.01)
#         else:
#             robot.set_motors(-controller.axes[1].value - controller.axes[0].value, -controller.axes[1].value + controller.axes[0].value)
#             time.sleep(0.01)<jupyter_output><empty_output><jupyter_text>Start pthread<jupyter_code>thread1 = threading.Thread(target=jetbot_motion)
thread1.setDaemon(True)
thread1.start()<jupyter_output><empty_output><jupyter_text>Add method of close pthread<jupyter_code>def _async_raise(tid, exctype):
    """raises the exception, performs cleanup if needed"""
    tid = ctypes.c_long(tid)
    if not inspect.isclass(exctype):
        exctype = type(exctype)
    res = ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, ctypes.py_object(exctype))
    if res == 0:
        raise ValueError("invalid thread id") 
    elif res != 1:
        # """if it returns a number greater than one, you're in trouble,
        # and you should call it again with exc=NULL to revert the effect"""
        ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, None)
        

def stop_thread(thread):
    _async_raise(thread.ident, SystemExit)<jupyter_output><empty_output><jupyter_text>Create a method to adjust the position of the Jetbot to the autopilot angle and call this method to adjust.<jupyter_code>servo_device = ServoSerial() 
def camservoInitFunction():
    global leftrightpulse, updownpulse
    leftrightpulse = 2048
    updownpulse = 2048
    servo_device.Servo_serial_control(1, 2048)
    time.sleep(0.1)
    servo_device.Servo_serial_control(2, 1300)<jupyter_output><empty_output><jupyter_text>Call the above method to adjust the pan/tilt to the autopilot angle<jupyter_code>camservoInitFunction()
stop_thread(thread1)<jupyter_output><empty_output><jupyter_text>Once you have collected enough data, we need to copy it to our GPU desktop or cloud machine for training. We can call the following terminal command to compress the dataset folder backup into a zip file.>  If you are training on JetBot, you can skip this step! The -r flag in the zip command indicates recursion, so we include all nested files, and the -q flag indicates silence, so the zip command will not print any output.<jupyter_code>def timestr():
    return str(datetime.datetime.now().strftime('%Y-%m-%d_%H-%M-%S'))

!zip -r -q road_following_{DATASET_DIR}_{timestr()}.zip {DATASET_DIR}<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09311975296728746, 'fraction_numerical': 0.020553893660137025, 'mean_word_length': 4.420502092050209, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 55, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '231659', 'license': 'MIT', 'n_tokens_mistral': 3229, 'n_tokens_neox': 3062, 'n_words': 1150, 'path': 'Autopilot-Basic/Data collection.ipynb', 'repo_name': 'WyckliffeAluga/vulcan', 'size': '10363'}