<jupyter_start><jupyter_text>Setup<jupyter_code>from __future__ import absolute_import, division, print_function, unicode_literals
import IPython.display as display
from PIL import Image
import numpy as np
import matplotlib.pyplot as plt
import time
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2' 
# 0 = all messages are logged (default behavior)
# 1 = INFO messages are not printed
# 2 = INFO and WARNING messages are not printed
# 3 = INFO, WARNING, and ERROR messages are not printed

#On Mac you may encounter an error related to OMP, this is a workaround, but slows down the code
os.environ['KMP_DUPLICATE_LIB_OK']='True' #https://github.com/dmlc/xgboost/issues/1715
import tensorflow as tf
from openbot import dataloader, utils
AUTOTUNE = tf.data.experimental.AUTOTUNE
tf.__version__<jupyter_output><empty_output><jupyter_text>Data processing Define the dataset directory and give it a name.<jupyter_code>base_dir = "dataset"
dataset_name = "my_openbot"<jupyter_output><empty_output><jupyter_text>By default, all datasets in the training data directory will be used. You can also specify only specific datasets, for example:```train_datasets = ["my_openbot_1", "my_openbot_2"]test_datasets = ["my_openbot_3"]```<jupyter_code>train_data_dir = os.path.join(base_dir,"train_data")
test_data_dir = os.path.join(base_dir, "test_data")

train_datasets = [d for d in os.listdir(train_data_dir) if os.path.isdir(os.path.join(train_data_dir, d))] 
test_datasets = [d for d in os.listdir(test_data_dir) if os.path.isdir(os.path.join(test_data_dir, d))] 
print('Train Datasets: ',len(train_datasets))
print('Test Datasets: ',len(test_datasets))<jupyter_output><empty_output><jupyter_text>Running this for the first time will take some time. This code will match image frames to the controls (labels) and indicator signals (commands).  By default, data samples where the vehicle was stationary will be removed. If this is not desired, you need to pass `remove_zeros=False`. If you have made any changes to the sensor files, changed `remove_zeros` or moved your dataset to a new directory, you need to pass `redo_matching=True`.<jupyter_code>from openbot import associate_frames
max_offset = 1e3 #1ms
train_frames = associate_frames.match_frame_ctrl_cmd(train_data_dir, 
                                                     train_datasets, 
                                                     max_offset, 
                                                     redo_matching=False, 
                                                     remove_zeros=True)
test_frames = associate_frames.match_frame_ctrl_cmd(test_data_dir, 
                                                    test_datasets, 
                                                    max_offset, 
                                                    redo_matching=False, 
                                                    remove_zeros=True)
image_count_train = len(train_frames) 
image_count_test = len(test_frames) 
print("There are %d train images and %d test images" %(image_count_train, image_count_test))<jupyter_output><empty_output><jupyter_text>Hyperparameters You may have to tune the learning rate and batch size depending on your available compute resources and dataset. As a general rule of thumb, if you increase the batch size by a factor of n, you can increase the learning rate by a factor of sqrt(n). For debugging and hyperparamter tuning, you can set the number of epochs to a small value like 10. If you want to train a model which will achieve good performance, you should set it to 50 or more. In our paper we used 100.<jupyter_code>TRAIN_BATCH_SIZE = 16 #128
TEST_BATCH_SIZE = 16 #128
LR = 0.0001 #0.0003
NUM_EPOCHS = 10 #100<jupyter_output><empty_output><jupyter_text>Don't change these unless you know what you are doing<jupyter_code>#Input dimensions
IMG_HEIGHT = 720
IMG_WIDTH = 1280
#Offset dimensions (crop)
OFFSET_IMG_HEIGHT = 240 
OFFSET_IMG_WIDTH = 0
#Target dimensions
TARGET_IMG_HEIGHT = IMG_HEIGHT-OFFSET_IMG_HEIGHT
TARGET_IMG_WIDTH = IMG_WIDTH-OFFSET_IMG_WIDTH
#Network dimensions
NETWORK_IMG_HEIGHT = TARGET_IMG_HEIGHT//5 
NETWORK_IMG_WIDTH = TARGET_IMG_WIDTH//5 

STEPS_PER_EPOCH = np.ceil(image_count_train/TRAIN_BATCH_SIZE)

BN = True
FLIP_AUG = False
CMD_AUG = False<jupyter_output><empty_output><jupyter_text>Load using `tf.data` Inspired by:https://github.com/tensorflow/docs/blob/master/site/en/tutorials/load_data/images.ipynb To load the files as a `tf.data.Dataset` first create a dataset of the file paths. Depending on dataset size, this may take some time. If you encounter issues, you can use the commented lines instead. However, this will take **much** longer.<jupyter_code># list_train_ds = tf.data.Dataset.list_files(train_frames)
# list_test_ds = tf.data.Dataset.list_files(test_frames)

list_train_ds = tf.data.Dataset.list_files([str(train_data_dir+'/'+dataset+'/*/images/*') 
                                            for dataset in train_datasets])
list_test_ds = tf.data.Dataset.list_files([str(test_data_dir+'/'+dataset+'/*/images/*') 
                                           for dataset in test_datasets])<jupyter_output><empty_output><jupyter_text>Random dataset samples for verification<jupyter_code>for f in list_train_ds.take(5):
  print(f.numpy())
print()
for f in list_test_ds.take(5):
  print(f.numpy())<jupyter_output><empty_output><jupyter_text>Build dataset<jupyter_code>train_data = dataloader.dataloader(train_data_dir, train_datasets)
print ("Number of train samples: %d" %len(train_data.labels))
test_data = dataloader.dataloader(test_data_dir, test_datasets)
print ("Number of test samples: %d" %len(test_data.labels))<jupyter_output><empty_output><jupyter_text>Some functions for augmentation<jupyter_code>def augment_img(img):
  """Color augmentation

  Args:
    img: input image

  Returns:
    img: augmented image
  """
  img = tf.image.random_hue(img, 0.08)
  img = tf.image.random_saturation(img, 0.6, 1.6)
  img = tf.image.random_brightness(img, 0.05)
  img = tf.image.random_contrast(img, 0.7, 1.3)
  return img
def augment_cmd(cmd):
  """
  Command augmentation

  Args:
    cmd: input command

  Returns:
    cmd: augmented command
  """
  if not (cmd > 0 or cmd < 0):
    coin = tf.random.uniform(shape=[1],minval=0, maxval=1, dtype=tf.dtypes.float32)
    if (coin < 0.25):
      cmd = -1.0
    elif coin < 0.5:
      cmd = 1.0
  return cmd
def flip_sample(img,cmd,label):
  coin = tf.random.uniform(shape=[1],minval=0, maxval=1, dtype=tf.dtypes.float32)
  if coin < 0.5:
    img = tf.image.flip_left_right(img)
    cmd = -cmd
    label = tf.reverse(label, axis=[0])
  return img,cmd,label
def decode_resize_img(img):
  img = decode_img (img)
  img = tf.image.crop_to_bounding_box(img,OFFSET_IMG_HEIGHT,OFFSET_IMG_WIDTH,
    TARGET_IMG_HEIGHT,
    TARGET_IMG_WIDTH)
  # resize the image to the desired size.
  img = tf.image.resize(img, [NETWORK_IMG_HEIGHT, NETWORK_IMG_WIDTH])
  return img
def decode_img(img):
  # convert the compressed string to a 3D uint8 tensor
  img = tf.image.decode_jpeg(img, channels=3)
  # Use `convert_image_dtype` to convert to floats in the [0,1] range.
  img = tf.image.convert_image_dtype(img, tf.float32)
  return img<jupyter_output><empty_output><jupyter_text>Short pure-tensorflow function that converts a file path to an (image_data, label) pair:<jupyter_code>def process_train_path(file_path):
  cmd, label = train_data.get_label(tf.strings.regex_replace(file_path,"[/\\\\]","/"))
  # load the raw data from the file as a string
  img = tf.io.read_file(file_path)
  img = decode_img(img)
  img = augment_img(img)
  if FLIP_AUG:
    img, cmd, label = flip_sample(img, cmd, label)
  if CMD_AUG:
    cmd = augment_cmd(cmd)
  return (img, cmd), label
def process_test_path(file_path):
  cmd, label = test_data.get_label(tf.strings.regex_replace(file_path,"[/\\\\]","/"))
  # load the raw data from the file as a string
  img = tf.io.read_file(file_path)
  img = decode_img(img)
  return (img, cmd), label<jupyter_output><empty_output><jupyter_text>Use `Dataset.map` to create a dataset of `image, label` pairs:<jupyter_code># Set `num_parallel_calls` so multiple images are loaded/processed in parallel.
labeled_ds = list_train_ds.map(process_train_path, num_parallel_calls=4)
for (image, cmd), label in labeled_ds.take(1):
  print("Image shape: ", image.numpy().shape)
  print("Command: ", cmd.numpy())
  print("Label: ", label.numpy())
train_ds = utils.prepare_for_training(ds=labeled_ds, 
                                      batch_sz=TRAIN_BATCH_SIZE, 
                                      shuffle_buffer_sz=100*TRAIN_BATCH_SIZE, 
                                      prefetch_buffer_sz=10*TRAIN_BATCH_SIZE)
(image_batch, cmd_batch), label_batch = next(iter(train_ds))
utils.show_train_batch(image_batch.numpy(), cmd_batch.numpy(), label_batch.numpy())
test_ds = list_test_ds.map(process_test_path, num_parallel_calls=4)
test_ds = test_ds.batch(TEST_BATCH_SIZE)
test_ds = test_ds.prefetch(buffer_size=10*TRAIN_BATCH_SIZE)<jupyter_output><empty_output><jupyter_text>Training<jupyter_code>from openbot import callbacks, losses, metrics, models

model = models.cil_mobile(NETWORK_IMG_WIDTH,NETWORK_IMG_HEIGHT,BN)
loss_fn = losses.sq_weighted_mse_angle 
metric_list = ['MeanAbsoluteError', metrics.direction_metric, metrics.angle_metric]
optimizer = tf.keras.optimizers.Adam(lr=LR)

model.compile(optimizer=optimizer,
          loss=loss_fn, 
          metrics=metric_list)
print(model.summary())
MODEL_NAME = dataset_name + "_" + model.name + "_lr" + str(LR) + "_bz" + str(TRAIN_BATCH_SIZE)
if BN:
    MODEL_NAME += "_bn"
if FLIP_AUG:
    MODEL_NAME += "_flip"
if CMD_AUG:
    MODEL_NAME += "_cmd"    
    
checkpoint_path = os.path.join('models', MODEL_NAME, 'checkpoints')
log_path = os.path.join('models',MODEL_NAME,'logs')
print(MODEL_NAME)
history = model.fit(train_ds, 
                    epochs=NUM_EPOCHS, 
                    steps_per_epoch=STEPS_PER_EPOCH, 
                    validation_data=test_ds, 
                    callbacks=[callbacks.checkpoint_cb(checkpoint_path),
                               callbacks.tensorboard_cb(log_path),
                               callbacks.logger_cb(log_path)])<jupyter_output><empty_output><jupyter_text>Evaluation Plot metrics and loss<jupyter_code>plt.plot(history.history['MeanAbsoluteError'], label='mean_absolute_error')
plt.plot(history.history['val_MeanAbsoluteError'], label = 'val_mean_absolute_error')
plt.xlabel('Epoch')
plt.ylabel('Mean Absolute Error')
plt.legend(loc='lower right')
plt.savefig(os.path.join(log_path,'error.png'))
plt.plot(history.history['direction_metric'], label='direction_metric')
plt.plot(history.history['val_direction_metric'], label = 'val_direction_metric')
plt.xlabel('Epoch')
plt.ylabel('Direction Metric')
plt.legend(loc='lower right')
plt.savefig(os.path.join(log_path,'direction.png'))
plt.plot(history.history['angle_metric'], label='angle_metric')
plt.plot(history.history['val_angle_metric'], label = 'val_angle_metric')
plt.xlabel('Epoch')
plt.ylabel('Angle Metric')
plt.legend(loc='lower right')
plt.savefig(os.path.join(log_path,'angle.png'))
plt.plot(history.history['loss'], label='loss')
plt.plot(history.history['val_loss'], label = 'val_loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend(loc='lower right')
plt.savefig(os.path.join(log_path,'loss.png'))<jupyter_output><empty_output><jupyter_text>Save tf lite models for best and last checkpoint<jupyter_code>best_index = np.argmax(np.array(history.history['val_angle_metric']) \
                     + np.array(history.history['val_direction_metric']))
best_checkpoint = str("cp-%04d.ckpt" % (best_index+1))
best_tflite = utils.generate_tflite(checkpoint_path, best_checkpoint)
utils.save_tflite (best_tflite, checkpoint_path, "best")
print("Best Checkpoint (val_angle: %s, val_direction: %s): %s" \
      %(history.history['val_angle_metric'][best_index],\
        history.history['val_direction_metric'][best_index],\
        best_checkpoint))
last_checkpoint = sorted([d for d in os.listdir(checkpoint_path) if os.path.isdir(os.path.join(checkpoint_path, d))])[-1]
last_tflite = utils.generate_tflite(checkpoint_path, last_checkpoint)
utils.save_tflite (last_tflite, checkpoint_path, "last")
print("Last Checkpoint (val_angle: %s, val_direction: %s): %s" \
      %(history.history['val_angle_metric'][-1], \
        history.history['val_direction_metric'][-1], \
        last_checkpoint))<jupyter_output><empty_output><jupyter_text>Evaluate the best model<jupyter_code>best_model = utils.load_model(os.path.join(checkpoint_path,best_checkpoint),loss_fn,metric_list)
test_loss, test_acc, test_dir, test_ang = best_model.evaluate(test_ds, steps=image_count_test/TEST_BATCH_SIZE, verbose=2)
NUM_SAMPLES = 15
(image_batch, cmd_batch), label_batch = next(iter(test_ds))
pred_batch = best_model.predict( (tf.slice(image_batch, [0, 0, 0, 0], [NUM_SAMPLES, -1, -1, -1]), tf.slice(cmd_batch, [0], [NUM_SAMPLES])) )
utils.show_test_batch(image_batch.numpy(), cmd_batch.numpy(), label_batch.numpy(), pred_batch)   
utils.compare_tf_tflite(best_model,best_tflite)<jupyter_output><empty_output><jupyter_text>Save the notebook as HTML<jupyter_code>#time.sleep(30)
utils.save_notebook()
current_file = 'policy_learning.ipynb'
output_file = os.path.join(log_path,'notebook.html')
utils.output_HTML(current_file, output_file)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11245429445563764, 'fraction_numerical': 0.0101484963808671, 'mean_word_length': 4.434712084347121, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 70, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 23, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '207298', 'license': 'MIT', 'n_tokens_mistral': 4669, 'n_tokens_neox': 4487, 'n_words': 959, 'path': 'policy/policy_learning.ipynb', 'repo_name': 'igcxl/OpenBot', 'size': '13401'}