<jupyter_start><jupyter_text>Automatic License Plate Recognition System for Parking Lot ML Project Group 35 <NAME>  (hhg210) <NAME>  (yl4786) For our project, we will focus on implementing a sign-in system in a parking lot. Such a system will check a vehicle in a parking lot noting the vehicles license plate and the time of the checkin and checkout. Other features such as number of passengers, colour of the vehicle, etc. can be derived from an image sequence or a video sequence of the vehicles at the checkin gate set at license plate level and pointing perpendicular to the plate to minimize skew.To implement this design, we first must build a dataset with the appropriate characters. For the English letters, we can use the Char74k dataset. This famous dataset contains an assortment of images of 0-9 and A-Z and can be conveniently found at the following link http://www.ee.surrey.ac.uk/CVSSP/demos/chars74k/.![Char74k](http://www.ee.surrey.ac.uk/CVSSP/demos/chars74k/Samples/english.png) First, we will read in a number of images of 0-9 and A-Z to train our letter recognition model. Because of limited images, we can use Kera's image generator to create more training data. We have selected to slightly rotate, stretch, shift, but not flip the images. Then, we will rename the images in a character_imagenumber fashion.Once the training images are prepared, we will read them into an array to hold the image (image_array) and another array to hold the ground truth of that image (target_data).   Train Model<jupyter_code>import os
import numpy as np
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score
from sklearn.externals import joblib
from skimage.io import imread
from skimage.filters import threshold_otsu
from skimage import transform,data<jupyter_output><empty_output><jupyter_text>Creating dataset and reading in data<jupyter_code>from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img
letters = [
            '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D',
            'E', 'F', 'G', 'H', 'J', 'K', 'L', 'M', 'N', 'P', 'Q', 'R', 'S', 'T',
            'U', 'V', 'W', 'X', 'Y', 'Z'
        ]
letters_num = [i for i in range(0,len(letters))]
# Generate more training pictures by slightly modifying the given data.
# If the training data is not much enough, we can create more training data to get a better acuracy.
# Now we just want 50 pictures for each character, so we don't create more pictures.
datagen = ImageDataGenerator(
        rotation_range=10,
        width_shift_range=0.1,
        height_shift_range=0.1,
        shear_range=0.1,
        horizontal_flip=False,
        fill_mode='nearest')
for ltr in letters:
    for idx in range(0,20): 
        img_path =  'train/'+ltr+'/'+ltr+'_'+str(idx)+'.jpg'
        if os.path.exists(img_path)==False:
                continue
        else:
            img = load_img(img_path)  
            
            x = img_to_array(img)  
            x = x.reshape((1,) + x.shape)  

            i = 0
            pic=20
            for batch in datagen.flow(x, batch_size=1,save_to_dir='train/'+ltr+'/', save_prefix=ltr, save_format='jpeg'):
                i += 1
                pic+=1
                if i > 50: 
                    break 
# Rename the image paths to a uniform fashion
paths = ['train/'+j+'/' for j in letters]
for path, ltr in zip(paths, letters):
    files = os.listdir(path)
    i = 0
    for file in files:
        os.rename(os.path.join('train/'+ltr, file), os.path.join(path, ltr+'_'+str(i)+'.jpg'))
        i = i+1<jupyter_output><empty_output><jupyter_text>We will use SVM with a linear kernel as the model with K=10 fold validation, then save the model.<jupyter_code>def read_training_data(training_directory):
    image_data = []
    target_data = []
    for each_letter in letters:
        # read in 50 images for each character
        for each in range(50):
            image_path = os.path.join(training_directory, each_letter, each_letter + '_' + str(each) + '.jpg')
            if os.path.exists(image_path)==True:
                try:
                    img_details = imread(image_path, as_grey=True)
                    img_details=transform.resize(img_details, (20, 20), mode='constant') # we transform the image to 20x20
                    # converts each character image to binary image
                    binary_image = img_details < threshold_otsu(img_details)
                    # flatten then add to the image_data
                    flat_bin_image = binary_image.reshape(-1)
                    image_data.append(flat_bin_image)
                    target_data.append(each_letter)
                except OSError as e:
                    os.remove(img_path)
            else:
                continue
    return (np.array(image_data), np.array(target_data))
def cross_validation(model,train_data,train_label,num_of_fold):
    accuracy_result = cross_val_score(model, train_data, train_label,cv=num_of_fold)
    print("Cross Validation Result for ", str(num_of_fold),"-fold")
    print(accuracy_result * 100)
current_dir = os.path.dirname(os.path.realpath("__file__"))
training_dataset_dir = os.path.join(current_dir, 'train')
image_data, target_data = read_training_data(training_dataset_dir)

svc_model = SVC(kernel='linear', probability=True)
cross_validation(svc_model, image_data, target_data,5)
# we save the model
svc_model.fit(image_data, target_data)

save_directory = os.path.join(current_dir, 'models/svc/')
if not os.path.exists(save_directory):
    os.makedirs(save_directory)
joblib.dump(svc_model, save_directory+'/svc.pkl')<jupyter_output><empty_output><jupyter_text>Load Image To demonstrate the functionality of our system, we will show example.<jupyter_code>from skimage.filters import threshold_otsu
import matplotlib.pyplot as plt
# We found some car images from google.
# You can change the file name to test: 56Y2,FELON,NOMURCI,FGJ3663,AHJ29M3,FBX5004,HAMMAS
car_image_path="./cars/FBX5004.jpg"
car_image = imread(car_image_path, as_grey=True)

gray_car_image = car_image * 255
fig, (ax1, ax2) = plt.subplots(1, 2)
ax1.imshow(gray_car_image, cmap="gray")
threshold_value = threshold_otsu(gray_car_image)
binary_car_image = gray_car_image > threshold_value
ax2.imshow(binary_car_image, cmap="gray")
plt.show()<jupyter_output><empty_output><jupyter_text>License Plate Detection Now we use localization technique to figure out where in the image is a license plate. We characterize this as a long rectangular shape for the dimensions. For this project, we first plot the image with the pixels, then come up with a general rule of the dimensions for the smallest/largest height/width a plate could be. This general rule would apply to almost all license plates. This is similiar to the technique used in Lab 6 for the MNIST data to draw a bounding box.<jupyter_code>from skimage import measure
from skimage.measure import regionprops
import matplotlib.patches as patches
# this gets the connected regions and allows use to iterate through them
label_image = measure.label(binary_car_image)


# From observation of the standard universal license plate, these are the dimensions we came up with
plate_dimensions = (0.1*label_image.shape[0], 0.6*label_image.shape[0], 0.25*label_image.shape[1], 0.3*label_image.shape[1])
# Why this work?
# Because in a parking lot, the picture are take in almost the same place. So the license plate should have similar size in the picture.

min_height, max_height, min_width, max_width = plate_dimensions

plate_objects_cordinates = []
plate_like_objects = []
fig, (ax1) = plt.subplots(1)
ax1.imshow(gray_car_image, cmap="gray");

for region in regionprops(label_image):
    if region.area < 40:
        #if the region is so small then it's likely not a license plate
        continue

    # the bounding box coordinates
    min_row, min_col, max_row, max_col = region.bbox
    region_height = max_row - min_row
    region_width = max_col - min_col
    
    if region_height >= min_height and region_height <= max_height and region_width >= min_width and region_width <= max_width and region_width > region_height:
        plate_like_objects.append(binary_car_image[min_row:max_row,
                                  min_col:max_col])
        plate_objects_cordinates.append((min_row, min_col,
                                              max_row, max_col))
        rectBorder = patches.Rectangle((min_col, min_row), max_col-min_col, max_row-min_row, edgecolor="red", linewidth=2, fill=False)
        ax1.add_patch(rectBorder)
    
plt.show()<jupyter_output><empty_output><jupyter_text>Character Segmentation<jupyter_code>from skimage.transform import resize
from skimage import measure
from skimage.measure import regionprops
import matplotlib.patches as patches
import sys
# Make sure there is a LP found
if np.shape(plate_like_objects)[0] == 0:
    print('No LP found, please back up and park again just in front of the line')
    sys.exit()
    
# Invert the colors b->w, w->b
license_plate = np.invert(plate_like_objects[0])
#license_plate = plate_like_objects[0]


labelled_plate = measure.label(license_plate)

fig, ax1 = plt.subplots(1)
ax1.imshow(license_plate, cmap="gray")

# Again from observation, we note the approximate proportions of the characters on the plate
character_dimensions = (0.35*license_plate.shape[0], 0.8*license_plate.shape[0], 0.05*license_plate.shape[1], 0.25*license_plate.shape[1])
min_height, max_height, min_width, max_width = character_dimensions

characters = []
counter=0
column_list = []

for regions in regionprops(labelled_plate):
    y0, x0, y1, x1 = regions.bbox
    region_height = y1 - y0
    region_width = x1 - x0

    if region_height > min_height and region_height < max_height and region_width > min_width and region_width < max_width:
        roi = license_plate[y0:y1, x0:x1]

        
        rect_border = patches.Rectangle((x0, y0), x1 - x0, y1 - y0, edgecolor="green",
                                       linewidth=2, fill=False)
        ax1.add_patch(rect_border)

        resized_char = resize(roi, (20, 20), mode='constant')
        characters.append(resized_char)

        # keep track of the arrangement of the characters
        column_list.append(x0)

plt.show()

column_list<jupyter_output><empty_output><jupyter_text>Character Prediction<jupyter_code># load the model
current_dir = os.path.dirname(os.path.realpath("__file__"))
model_dir = os.path.join(current_dir, 'models/svc/svc.pkl')
model = joblib.load(model_dir)

classification_result = []
for each_character in characters:
    # converts it to a 1D array
    each_character = each_character.reshape(1, -1);
    result = model.predict(each_character)
    classification_result.append(result)

plate_string = ''
for eachPredict in classification_result:
    plate_string += eachPredict[0]
    
# arrange in correct order
column_list_copy = column_list[:]
column_list.sort()


temp_string = ''
for each in column_list:
    temp_string += plate_string[column_list_copy.index(each)]
    
#predicted_plate = temp_string[len(temp_string)-1]+temp_string[0:len(temp_string)-1]
print(temp_string)<jupyter_output>FBX5004<jupyter_text>LP RegistrationCheck if the car's license plate has been on the list already * if yes, that means the car is going out, then we output the time it parked* if no, that means it is a newcoming car, we will register it onto the list<jupyter_code>import pandas as pd
from datetime import datetime
import csv

names =['LP','Time']
carlist = pd.read_csv('./list.csv',header=None,index_col=None,names=names,na_values='?')
LPs = (np.array(carlist.LP)).tolist()
Times = (np.array(carlist.Time)).tolist()
LP = temp_string
T = datetime.now().strftime("%Y-%m-%d %H:%M:%S")

if LP in LPs:
    ind = np.squeeze(np.where(LP in LPs))
    s1 = Times[ind] # getting time corrsponding to the license plate
    FMT = '%Y-%m-%d %H:%M:%S'
    tdelta = datetime.strptime(T, FMT) - datetime.strptime(s1, FMT)
    print('Thanks for parking with us, your parking time is',str(tdelta))
    
    LPs.remove(LP) # deleting the car going out
    Times.remove(s1)
    lines = [np.zeros(len(LP)),np.zeros(len(LP))]
    with open('list.csv', 'w',newline='') as csvfile:
        writer = csv.writer(csvfile)
        k = 0
        for k in range (len(LPs)):
            lines[k] = [LPs[k],Times[k]]
            writer.writerow(lines[k])
else:
    newitem = (LP,T) # adding the new car
    with open('list.csv', 'a',newline='') as csvfile:
        writer = csv.writer(csvfile)
        writer.writerow([])
        writer.writerow(newitem)
        print('Welcome to Parking Lot')<jupyter_output>Welcome to Parking Lot<jupyter_text>Show the already parked cars<jupyter_code>carlist = pd.read_csv('./list.csv',header=None,index_col=None,names=names,na_values='?')        
print('existing cars:')
print(carlist)<jupyter_output>existing cars:
        LP                 Time
0     56Y2  2017-12-11 12:32:21
1    FEL0N  2017-12-12 20:01:58
2  N0MURC1  2017-12-12 22:11:15
3  FBX5004  2017-12-20 05:25:50
{'fraction_non_alphanumeric': 0.10039023643737088, 'fraction_numerical': 0.019588338817047975, 'mean_word_length': 3.6695248302965346, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '354897', 'license': 'FTL', 'n_tokens_mistral': 4337, 'n_tokens_neox': 4116, 'n_words': 1237, 'path': 'project.ipynb', 'repo_name': 'kohuala/ML-group-35', 'size': '13074'}