<jupyter_start><jupyter_text>Capstone Project  Steering Angle Predictionby: <NAME> DSI-15  Notebook 01: EDA  Executive Summary The push for self-driving vehicles has increased dramatically over the last few years after the success of the DARPAGrand Challenge 2. An autonomous vehicle works seamlessly when multiple components come together and work in synergy. The most important parts are the various sensors and the AI software powering the vehicle. One of the most critical functions of the AI is to predict the steering angle of the vehicle for the stretch of road lying immediately in front of it and accordingly even steer the car. Increase in computational capabilities over the years allows us to train deep neural networks to become the ”brain” of these cars which then understands the surroundings of the car and makes navigation decisions in real time.   Comma.ai released 7 hours of driving footage which is available for open source usage. The data set of images was taken while a car was being manually driven, annotated with the corresponding steering angle applied by the human driver. The goal of one of the challenges was to find a model that, given an image taken while driving, will minimize for RMSE (root mean square error) between the model predictions and the actual steering angle.   In this project, we explore deep convolutional neural networks to predict steering angle values. Predicting the steering angle is one of the most important parts of creating a self-driving car. This task would allow us to explore the full power of neural networks, using only steering angle as the training signal, deep neural networks can automatically extract features from the images to help understand the position of the car with respect to the road to make the prediction.  Based on the dataset from comma.ai, my model was able to attain an overall RSME score of 0.805.    Problem StatementTo develop a Convolutional Neural Network-based regression model, based on Highway Driving Footage to determine if the model is capable of predicting the steering wheel angle. The evaluation metrics RMSE (root mean square error) where we try to minimise the value as much as possible.  Content - [Import libraries](Import-libraries)- [Dataset source information](Dataset-source-information)- [Dataset summary](Dataset-summary)- [Exploratory Data Analysis](Exploratory-Data-Analysis)    - [Log1 Data](Log1-Data)    - [Camera1 Data](Camera1-Data)- [Pre-processing](Pre-processing)       - [Camera1 and log1](Camera1-and-log1)       - [Camera2 and log2](Camera2-and-log2)       - [Camera3 and log3](Camera3-and-log3)       - [Camera4 and log4](Camera4-and-log4)       - [Camera5 and log5](Camera5-and-log5)       - [Camera6 and log6](Camera6-and-log6)       - [Camera7 and log7](Camera7-and-log7)       - [Camera8 and log8](Camera8-and-log8)       - [Camera9 and log9](Camera9-and-log9)       - [Camera10 and log10](Camera10-and-log10)       - [Camera11 and log11](Camera11-and-log11)  Import libraries<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import h5py
import cv2
import imageio
from numpy import asarray
from numpy import savez_compressed<jupyter_output><empty_output><jupyter_text>Dataset source information Source 1: `Udacity Self Driving Car` [Link](http://academictorrents.com/details/9b0c6c1044633d076b0f73dc312aa34433a25c56)- Include 3 cameras data; left, center and right in `ROSBag` format - Steering angles in `CSV` format- Size `70.19GB`Was initially facing issue with unpacking the dataset that was found from Udacity as it is in ROSBag format.  As I'm not running on UNIX OS (Linux, macOS) I was unable to find a viable solution to unpack ROSBag format using Windows. It will require me to run a virtual machine within Windows for ubuntu and then access the necessary scripts via docker container to extract the relevant dataset. I then decide to look at the alternative dataset from comma.ai which is in h5 format, this was much easier to deal with.Source 2: `comma.ai driving dataset` [Link](http://academictorrents.com/details/58c41e8bcc8eb4e2204a3b263cdf728c0a7331eb)- Contains only center camera data and the car's measurement data in `h5` format- 2016-01-30--11-24-51 (7.7G)- 2016-01-30--13-46-00 (8.5G)- 2016-01-31--19-19-25 (3.0G)- 2016-02-02--10-16-58 (8.1G)- 2016-02-08--14-56-28 (3.9G)- 2016-02-11--21-32-47 (13G)- 2016-03-29--10-50-20 (12G)- 2016-04-21--14-48-08 (4.4G)- 2016-05-12--22-20-00 (7.5G)- 2016-06-02--21-39-29 (6.5G)- 2016-06-08--11-46-01 (2.7G)- Size `80GB uncompressed`  Dataset summary Each individual dataset is labelled as followed throughout my notebook.- `camera1`, `log1`:   2016-01-30--11-24-51 - `camera2`, `log2`:   2016-01-30--13-46-00 - `camera3`, `log3`:   2016-01-31--19-19-25 - `camera4`, `log4`:   2016-02-02--10-16-58 - `camera5`, `log5`:   2016-02-08--14-56-28 - `camera6`, `log6`:   2016-02-11--21-32-47- `camera7`, `log7`:   2016-03-29--10-50-20 - `camera8`, `log8`:   2016-04-21--14-48-08 - `camera9`, `log9`:   2016-05-12--22-20-00- `camera10`, `log10`: 2016-06-02--21-39-29- `camera11`, `log11`: 2016-06-08--11-46-01`camera1`, `camera2`, `camera4`, `camera5`, `camera7`, `camera8`, `camera11` are day time footages.`camera3`, `camera6`, `camera9`, `camera10` are night time footages.`camera10` and `camera11` data are left untouched to use for unseen prediction (day & night) on our trained model.  Exploratory Data Analysis  A thorough run through of `camera1` and `log1` dataset will be explored as the sample of the entire flow. For the subsequent datasets, they are run through function for ease of analysis.  Log1 Data With reference to https://github.com/commaai/research/blob/master/Logs.md for data dictionary, not much information is available apart from a few important ones that is of note- `cam1_ptr`: camera frame index at this time- `times`: seconds- `steering_torque`: deg/s, despite the name, this is the steering angle rate- `speed`: m/s, from encoder after transmission, negative when gear is Revese- `speed_abs`: m/s, from encoder after transmission- `gas`: 0-1- `brake`: brake_computer + brake_user<jupyter_code># loading the data
log1 = h5py.File('./datasets/log/2016-01-30--11-24-51.h5', 'r')
# dataset is kept in a dictionary
log1.keys()
# print out every dictionary key and it's relevant shape and type
for index in log1.keys():
    print(log1[index])
# convert the relevant keys and value pair that I require into a pandas dataframe
log1 = pd.DataFrame({'cam1_ptr': log1['cam1_ptr'],
                    'times': log1['times'],
                    'steering_rate': log1['steering_torque'],
                    'speed': log1['speed'],
                    'speed_abs': log1['speed_abs'],
                    'gas': log1['gas'],
                    'brake': log1['brake']},
                    columns = ['cam1_ptr', 'times', 'steering_rate', 'speed', 'speed_abs', 'gas', 'brake'])
log1.info()
log1.tail(10)
# since the input data is at 100hz and the image data is at 20hz I took the mean value as the actual input for the 
# corresponding individual frame id (cam1_ptr)
log1['steering_avg'] = log1.groupby('cam1_ptr')['steering_rate'].transform('mean')
log1['brake_avg'] = log1.groupby('cam1_ptr')['brake'].transform('mean')
log1['speed_avg'] = log1.groupby('cam1_ptr')['speed'].transform('mean')
log1['gas_avg'] = log1.groupby('cam1_ptr')['gas'].transform('mean')
log1.head()
# the original 100hz input data is dropped for times, steering_rate, speed, speed_bas, gas, brake
log1.drop(['steering_rate', 'speed', 'speed_abs', 'gas', 'brake'], axis=1, inplace=True)
# keeping only 1 input for the relevant cam1_ptr id 
log1 = log1.drop_duplicates('cam1_ptr', keep='first', ignore_index=True)
log1.info()
# ensure that the log1 file has the same rows as camera1 file
log1.shape
plt.figure(figsize = (16, 10))
plt.title('Steering Angle of vehicle for Camera1', fontsize=20)
plt.xlabel('Number of Image Frames', fontsize=15)
plt.ylabel('Steering Angle (degrees)', fontsize=15)
log1['steering_avg'].plot.line();
plt.figure(figsize = (16, 10))
plt.title('Speed of vehicle for Camera1', fontsize=20)
plt.xlabel('Number of Image Frames', fontsize=15)
plt.ylabel('Speed (m/s)', fontsize=15)
log1['speed_avg'].plot.line();
plt.figure(figsize = (16, 10))
plt.title('Gas throttle of vehicle for Camera1', fontsize=20)
plt.xlabel('Number of Image Frames', fontsize=15)
plt.ylabel('Gas throttle (0-1)', fontsize=15)
log1['gas_avg'].plot.line();
plt.figure(figsize = (16, 10))
plt.title('Brake of vehicle for Camera1', fontsize=20)
plt.xlabel('Number of Image Frames', fontsize=15)
plt.ylabel('Brake', fontsize=15)
log1['brake_avg'].plot.line();
plt.figure(figsize = (12, 8))
plt.title('Histogram of steering angle for Camera1', fontsize=20)
plt.xlabel('Steering Angle (degrees)', fontsize=15)
log1['steering_avg'].plot.hist();
plt.figure(figsize = (12, 8))
plt.title('Histogram of Speed for Camera1', fontsize=20)
plt.xlabel('Speed (m/s)', fontsize=15)
log1['speed_avg'].plot.hist();
plt.figure(figsize = (12, 8))
plt.title('Histogram of gas throttle for Camera1', fontsize=20)
plt.xlabel('Gas throttle (0-1)', fontsize=15)
log1['gas_avg'].plot.hist();
plt.figure(figsize = (16, 10))
plt.title('Histogram of brake for Camera1', fontsize=20)
plt.xlabel('Brake', fontsize=15)
log1['brake_avg'].plot.hist();<jupyter_output><empty_output><jupyter_text>The parameters that is of interest for our specific case will only be the steering angle, we can see from the line plot as well as the bar plot that majority of the data is specific to minimal steering (within a range of 50 degrees clockwise and counter-clockwise) this is because majority of the dataset is of highway driving as is typically cruise speed and generally do not have steep turning curvature on the roads. The plots of the speed, gas throttle and braking also further show that majority of the dataset is of highway driving.Due to the lack of sufficient data for steering angles greater than 50 degrees, I would have to augment the data so as to ensure that the model will be capable of predicting large steering angles. However, upon research I was unable to find a decent way to augment the existing data to oversample large steering angles. I then proceeded to select for data only with larger steering angle and train a seperate model for predicting larger steering angles but even with data specific to larger steering angle I was unable to a low root mean square error (RMSE). I suspect that this is likely due to insufficient larger steering angles data for the model to generalise, another factor that might have result in poor performance is also due to the fact that the images are not labelled to show that which areas are roads/pedestrian paths, throttle, brake and speed is also not factored in. All these are areas of improvement that further projects can look into incorporating to improve the model performance.  Camera1 Data<jupyter_code>camera1 = h5py.File('./datasets/camera/2016-01-30--11-24-51.h5', 'r')
# data is in dictionary
camera1.keys()
camera1 = camera1['X']
# this are images in a numpy array
camera1.shape
"""
opencv and matplotlib reads numpy arrays to images via rows, columns, channels so I'll have to manipulate the array to
the correct format. I first take a single image and see if I'm manipulating the array properly
"""
single_image = camera1[2000]
single_image = single_image.swapaxes(0, 2).swapaxes(0, 1)
single_image.shape
plt.imshow(single_image);
batch_image = camera1[2000:2050]
batch_image.shape
"""
I then try a loop through a few images to see if my code works through a batch of numpy arrays
"""
empty_array = []

for i in range(batch_image.shape[0]):
    single_image = batch_image[i].swapaxes(0, 2).swapaxes(0, 1)
    empty_array.append(single_image)
    
empty_array = np.array(empty_array)
empty_array.shape
plt.imshow(empty_array[0]);
"""
Now I create a function so that the entire Camera1 data can be iterate through
I also added in a line of code to convert the 3 color channel image to grayscale to reduce the dataset size as I deem color
to not be important in predicting the steering wheel angle
"""
def reshape_togray_function(variable, start, end):
    empty_array = []   
    
    for i in range(start, end):
        single_image = variable[i].swapaxes(0, 2).swapaxes(0, 1)
        # convert the corrected image shape to grayscale
        single_image = cv2.cvtColor(single_image, cv2.COLOR_RGB2GRAY)
        empty_array.append(single_image)
    
    empty_array = np.array(empty_array)
    return empty_array
camera1 = reshape_togray_function(camera1, 0, camera1.shape[0])
camera1.shape
"""
I also convert the camera1 numpy array into a video file to inspect the vehicle footage fps = 20 since data is 20 hz
"""
imageio.mimwrite('./video/camera1.mp4', camera1, fps= 20.0)
plt.imshow(camera1[24746], cmap='gray');
"""
to further reduce the dataset size I reduce the image size by half to 80 x 160 and after inspecting the video I've decided
brighten the images as there are segments in the video that are dark
"""
# credit https://stackoverflow.com/questions/32609098/how-to-fast-change-image-brightness-with-python-opencv
# after some testing decided to use a value of 50

def increase_brightness_resize(variable, value=50):
    empty_array = []
    
    for i in range(variable.shape[0]):
        # resizing
        img = cv2.resize(variable[i], dsize=(160, 80), interpolation=cv2.INTER_AREA)
        
        # brightening
        lim = 255 - value
        img[img > lim] = 255
        img[img <= lim] += value

        empty_array.append(img)
        
    empty_array = np.array(empty_array)    
    return empty_array
camera1 = increase_brightness_resize(camera1)
plt.imshow(camera1[24746], cmap='gray');
camera1.shape<jupyter_output><empty_output><jupyter_text>Pre-processing For `camera2` to `camera11` and `log2` to `log11` data they are run through a function and saved individual as `camera{number}_cleaned.npz` or `log{number}_cleaned.csv`. `camera{number}` and `log{number}` steps:- load the h5py file for both camera and log- run the log file with the `log_file` function and check shape- run the camera file with `camera_file` function and check shape- convert the camera file `numpy.array` to a `.mp4` and check front and back of the video- comparison operator to find the relevant frame id- slice both the `camera` and `log` data- save both as `.npz` and `.csv` file respectively  Camera1 and log1<jupyter_code>log1.shape
camera1.shape
# after inspecting the video remove the first 50 seconds as it is not driving footage
log1[log1.times > 50].head()
# after inspecting the video remove the last 23 seconds as it is not driving footage
log1.times.max()-23
log1[log1.times > 2612].head()
log1 = log1.iloc[993:52247].reset_index(drop=True)
log1.shape
camera1 = camera1[993:52247]
camera1.shape
log1.to_csv('./datasets/log1_cleaned.csv', index=False)
savez_compressed('./datasets/camera1_cleaned.npz', camera1)<jupyter_output><empty_output><jupyter_text>Camera2 and log2<jupyter_code># fucntion for log
def log_file(log):
    
    # convert the necessary columns of the logfile to dataframe
    df = pd.DataFrame({'cam1_ptr': log['cam1_ptr'],
                       'times': log['times'],
                       'steering_rate': log['steering_torque'],
                       'speed': log['speed'],
                       'speed_abs': log['speed_abs'],
                       'gas': log['gas'],
                       'brake': log['brake']},
                       columns = ['cam1_ptr', 'times', 'steering_rate', 'speed', 'speed_abs', 'gas', 'brake'])
    
    # mean transform them as from 100hz to 20hz    
    df['steering_avg'] = df.groupby('cam1_ptr')['steering_rate'].transform('mean')
    df['brake_avg'] = df.groupby('cam1_ptr')['brake'].transform('mean')
    df['speed_avg'] = df.groupby('cam1_ptr')['speed'].transform('mean')
    df['gas_avg'] = df.groupby('cam1_ptr')['gas'].transform('mean')
    
    # drop times, steering_rate, speed, speed_bas, gas, brake
    df.drop(['steering_rate', 'speed', 'speed_abs', 'gas', 'brake'], axis=1, inplace=True)
    
    # keeping only 1 input for the relevant cam1_ptr id
    df = df.drop_duplicates('cam1_ptr', keep='first', ignore_index=True)    
    
    return df
# function for camera
def camera_file(camera, start, end, value=50):    
    empty_array = []
    
    for i in range(start, end):
        # iterate through all the images and shift the image from RGB, height, width to height, width, RGB
        img = camera[i].swapaxes(0, 2).swapaxes(0, 1)
        
        # convert to grayscale
        img = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
        
        # resize the image
        img = cv2.resize(img, dsize=(160, 80), interpolation=cv2.INTER_AREA)
        
        # brighten the image
        lim = 255 - value
        img[img > lim] = 255
        img[img <= lim] += value
               
        empty_array.append(img)
        
    empty_array = np.array(empty_array)    
    return empty_array
camera2 = h5py.File('./datasets/camera/2016-01-30--13-46-00.h5', 'r')
log2 = h5py.File('./datasets/log/2016-01-30--13-46-00.h5', 'r')
log2 = log_file(log2)
log2.shape
camera2 = camera2['X']
camera2 = camera_file(camera2, 0, camera2.shape[0])
camera2.shape
# write the array into a video to inspect
imageio.mimwrite('./video/camera2.mp4', camera2, fps = 20.0)
log2.head()
# remove the first 70 secs and last 70 seconds of the video
log2[log2.times > 70].head()
log2[log2.times < log2.times.max()-70].tail()
log2 = log2.iloc[1393:57591].reset_index(drop=True)
log2.shape
camera2 = camera2[1393:57591]
camera2.shape
log2.to_csv('./datasets/log2_cleaned.csv', index=False)
savez_compressed('./datasets/camera2_cleaned.npz', camera2)<jupyter_output><empty_output><jupyter_text>Camera3 and log3<jupyter_code>camera3 = h5py.File('./datasets/camera/2016-01-31--19-19-25.h5', 'r')
log3 = h5py.File('./datasets/log/2016-01-31--19-19-25.h5', 'r')
log3 = log_file(log3)
camera3 = camera3['X']
camera3 = camera_file(camera3, 0, camera3.shape[0])
imageio.mimwrite('./video/camera3.mp4', camera3, fps = 20.0)
log3[log3.times > 70].head()
log3[log3.times < log3.times.max()-70].tail()
log3 = log3.iloc[1393:18329].reset_index(drop=True)
log3.to_csv('./datasets/log3_cleaned.csv', index=False)
log3.shape
camera3_full = camera3_full[1393:18329]
camera3_full.shape
savez_compressed('./datasets/camera3_cleaned.npz', camera3)<jupyter_output><empty_output><jupyter_text>Camera4 and log4<jupyter_code>camera4 = h5py.File('./datasets/camera/2016-02-02--10-16-58.h5', 'r')
log4 = h5py.File('./datasets/log/2016-02-02--10-16-58.h5', 'r')
log4 = log_file(log4)
camera4 = camera4['X']
camera4 = camera_file(camera4, 0, camera4.shape[0])
imageio.mimwrite('./video/camera4.mp4', camera4, fps = 20.0)
log4[log4.times > 70].head()
log4[log4.times < log4.times.max()-70].tail()
log4 = log4.iloc[1393:54764].reset_index(drop=True)
log4.to_csv('./datasets/log4_cleaned.csv', index=False)
log4.shape
camera4_full = camera4_full[1393:54764]
camera4_full.shape
savez_compressed('./datasets/camera4_cleaned.npz', camera4)<jupyter_output><empty_output><jupyter_text>Camera5 and log5<jupyter_code>camera5 = h5py.File('./datasets/camera/2016-02-08--14-56-28.h5', 'r')
log5 = h5py.File('./datasets/log/2016-02-08--14-56-28.h5', 'r')
log5 = log_file(log5)
camera5 = camera5['X']
camera5 = camera_file(camera5, 0, camera5.shape[0])
imageio.mimwrite('./video/camera5.mp4', camera5, fps = 20.0)
log5[log5.times > 70].head()
log5[log5.times < log5.times.max()-70].tail()
log5 = log5.iloc[1393:24463].reset_index(drop=True)
log5.to_csv('./datasets/log5_cleaned.csv', index=False)
log5.shape
camera5 = camera5[1393:24463]
camera5.shape
savez_compressed('./datasets/camera5_cleaned.npz', camera5)<jupyter_output><empty_output><jupyter_text>Camera6 and log6<jupyter_code>camera6 = h5py.File('./datasets/camera/2016-02-11--21-32-47.h5', 'r')
log6 = h5py.File('./datasets/log/2016-02-11--21-32-47.h5', 'r')
log6 = log_file(log6)
camera6 = camera6['X']
camera6 = camera_file(camera6, 0, camera6.shape[0])
imageio.mimwrite('./video/camera6.mp4', camera6, fps = 20.0)
log6[log6.times > 70].head()
log6[log6.times < log6.times.max()-70].tail()
log6 = log6.iloc[1392:83895].reset_index(drop=True)
log6.to_csv('./datasets/log6_cleaned.csv', index=False)
log6.shape
camera6 = camera6[1392:83895]
camera6.shape
savez_compressed('./datasets/camera6_cleaned.npz', camera6)<jupyter_output><empty_output><jupyter_text>Camera7 and log7<jupyter_code>camera7 = h5py.File('./datasets/camera/2016-03-29--10-50-20.h5', 'r')
log7 = h5py.File('./datasets/log/2016-03-29--10-50-20.h5', 'r')
log7 = log_file(log7)
camera7 = camera7['X']
camera7 = camera_file(camera7, 0, camera7.shape[0])
imageio.mimwrite('./video/camera7.mp4', camera7, fps = 20.0)
log7[log7.times > 150].head()
log7[log7.times < log7.times.max()-70].tail()
log7 = log7.iloc[3000:77060].reset_index(drop=True)
log7.to_csv('./datasets/log7_cleaned.csv', index=False)
log7.shape
camera7 = camera7[3000:77060]
camera7.shape
savez_compressed('./datasets/camera7_cleaned.npz', camera7)<jupyter_output><empty_output><jupyter_text>Camera8 and log8<jupyter_code>camera8 = h5py.File('./datasets/camera/2016-04-21--14-48-08.h5', 'r')
log8 = h5py.File('./datasets/log/2016-04-21--14-48-08.h5', 'r')
log8 = log_file(log8)
camera8 = camera8['X']
camera8 = camera_file(camera8, 0, camera8.shape[0])
imageio.mimwrite('./video/camera8.mp4', camera8, fps = 20.0)
log8[log8.times > 70].head()
log8[log8.times < log8.times.max()-70].tail()
log8 = log8.iloc[1400:29136].reset_index(drop=True)
log8.to_csv('./datasets/log8_cleaned.csv', index=False)
log8.shape
camera8 = camera8[1400:29136]
camera8.shape
savez_compressed('./datasets/camera8_cleaned.npz', camera8)<jupyter_output><empty_output><jupyter_text>Camera9 and log9<jupyter_code>camera9 = h5py.File('./datasets/camera/2016-05-12--22-20-00.h5', 'r')
log9 = h5py.File('./datasets/log/2016-05-12--22-20-00.h5', 'r')
log9 = log_file(log9)
camera9 = camera9['X']
camera9 = camera_file(camera9, 0, camera9.shape[0])
imageio.mimwrite('./video/camera9.mp4', camera9, fps = 20.0)
log9[log9.times > 120].head()
log9[log9.times < log9.times.max()-70].tail()
log9 = log9.iloc[2400:50290].reset_index(drop=True)
log9.to_csv('./datasets/log9_cleaned.csv', index=False)
log9.shape
camera9 = camera9[2400:50290]
camera9.shape
savez_compressed('./datasets/camera9_cleaned.npz', camera9)<jupyter_output><empty_output><jupyter_text>Camera10 and log10<jupyter_code>camera10 = h5py.File('./datasets/camera/2016-06-02--21-39-29.h5', 'r')
log10 = h5py.File('./datasets/log/2016-06-02--21-39-29.h5', 'r')
log10 = log_file(log10)
camera10 = camera10['X']
camera10 = camera_file(camera10, 0, camera10.shape[0])
imageio.mimwrite('./video/camera10.mp4', camera10, fps = 20.0)
log10[log10.times > 120].head()
log10[log10.times < log10.times.max()-70].tail()
log10 = log10.iloc[2399:43373].reset_index(drop=True)
log10.shape
camera10 = camera10[2399:43373]
camera10.shape
log10.to_csv('./datasets/log10_cleaned.csv', index=False)
savez_compressed('./datasets/camera10_cleaned.npz', camera10)<jupyter_output><empty_output><jupyter_text>Camera11 and log11<jupyter_code>camera11 = h5py.File('./datasets/camera/2016-06-08--11-46-01.h5', 'r')
log11 = h5py.File('./datasets/log/2016-06-08--11-46-01.h5', 'r')
log11 = log_file(log11)
camera11 = camera11['X']
camera11 = camera_file(camera11, 0, camera11.shape[0])
imageio.mimwrite('./video/camera11.mp4', camera11, fps = 20.0)
log11[log11.times > 60].head()
log11[log11.times < log11.times.max()-60].tail()
log11 = log11.iloc[1200:16975].reset_index(drop=True)
log11.shape
camera11 = camera11[1200:16975]
camera11.shape
log11.to_csv('./datasets/log11_cleaned.csv', index=False)
savez_compressed('./datasets/camera11_cleaned.npz', camera11)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13200184617966684, 'fraction_numerical': 0.07665841480300424, 'mean_word_length': 4.4929707305830835, 'pattern_counts': {'":': 0, '<': 70, '<?xml version=': 0, '>': 72, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '620146', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 9024, 'n_tokens_neox': 8025, 'n_words': 2264, 'path': '01_eda.ipynb', 'repo_name': 'leemjm92/dsi15_capstone_steering_wheel_prediction', 'size': '23837'}