<jupyter_start><jupyter_text>Automatic License Plate Recognition model that runs on the Raspberry Pi Zero with an EdgeTPU (Coral USB)Original models from: https://github.com/GreenWaves-Technologies/licence_plate_recognition The EdgeTPU compiler had a problem with the LPRNet:```!wget https://github.com/ricardodeazambuja/licence_plate_recognition/raw/master/model/lprnet.tflite```By recreating this model using Keras, I noticed the problem occurred when a CONV2D was connected to a dense layer. After splitting the model into two, the compiler didn't have the memory swelling problem.Splitting the model into two worked and also using the ```-a``` option together with ```-i```:```!edgetpu_compiler -i "Conv_14/Relu,Squeeze":"TFLite_Detection_PostProcess" -s -a lprnet.tflite ssdlite_ocr.tflite```However, the best solution so far was to convert the first dense layer into a conv layer (below).  Download the TFLite models<jupyter_code>!wget https://github.com/ricardodeazambuja/licence_plate_recognition/raw/master/model/ssdlite_ocr.tflite -O ssdlite_ocr.tflite
!wget https://github.com/ricardodeazambuja/licence_plate_recognition/raw/master/model/lprnet.tflite -O lprnet.tflite<jupyter_output>--2021-05-31 18:18:44--  https://github.com/ricardodeazambuja/licence_plate_recognition/raw/master/model/ssdlite_ocr.tflite
Resolving github.com (github.com)... 192.168.127.12
Connecting to github.com (github.com)|192.168.127.12|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/ricardodeazambuja/licence_plate_recognition/master/model/ssdlite_ocr.tflite [following]
--2021-05-31 18:18:44--  https://raw.githubusercontent.com/ricardodeazambuja/licence_plate_recognition/master/model/ssdlite_ocr.tflite
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 172.16.58.3, 192.168.127.12, 172.16.17.32, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|172.16.58.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3150480 (3.0M) [application/octet-stream]
Saving to: ‘ssdlite_ocr.tflite’

ssdlite_ocr.tflite  100%[===================>]   3.00M  --.-KB/s    in 0.1s    

202[...]<jupyter_text>Install EdgeTPU Compiler<jupyter_code># Install the edgetpu compiler.
!curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
!echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list
!sudo apt-get update
!sudo apt-get install edgetpu-compiler<jupyter_output>% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  2537  100  2537    0     0  97576      0 --:--:-- --:--:-- --:--:-- 97576
OK
deb https://packages.cloud.google.com/apt coral-edgetpu-stable main
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:2 https://packages.cloud.google.com/apt coral-edgetpu-stable InRelease [6,722 B]
Get:3 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ InRelease [3,626 B]
Ign:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease
Get:5 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic InRelease [15.9 kB]
Hit:6 http://archive.ubuntu.com/ubuntu bionic InRelease
Ign:7 https://packages.cloud.google.com/apt coral-edgetpu-stable/main amd64 Packages
Ign:8 https://developer.download.nvidia.com/co[...]<jupyter_text>Flatbuffer Install / compile flatc<jupyter_code># %%bash
# rm -rf /content/flatbuffers
# git clone https://github.com/ricardodeazambuja/flatbuffers.git
# cd flatbuffers
# mkdir build 
# cd build 
# cmake -DFLATBUFFERS_STATIC_FLATC=ON -DFLATBUFFERS_BUILD_FLATLIB=OFF .. 
# make -j4 
# sudo make install 
# flatc --version

# !sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test 
# !sudo apt-get update
# !sudo apt-get install --only-upgrade libstdc++6
# !sudo add-apt-repository -y --remove ppa:ubuntu-toolchain-r/test 
# !sudo apt-get update
# !strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX

# I compiled it as static file (no dependencies!)
!wget https://github.com/ricardodeazambuja/flatbuffers/releases/download/v2.0.1a/flatc.zip
!unzip flatc.zip
!rm flatc.zip
!chmod +x flatc
!sudo mv flatc /usr/local/bin/
!flatc --version
# Download the schema for tflite
!wget https://raw.githubusercontent.com/tensorflow/tensorflow/master/tensorflow/lite/schema/schema.fbs -O schema.fbs
# Convert tflite to json
%env schema=schema.fbs
# If schema version is smaller than v3, it needs to have --raw-binary
%env raw=
!flatc -t $raw --strict-json --defaults-json $schema -- lprnet.tflite<jupyter_output>env: schema=schema.fbs
env: raw=<jupyter_text>Convert first dense layer of LPRNet.tflite into convolutional Replace the dense1 with conv2d. The same weights can be used!<jupyter_code>from copy import deepcopy
import json
import numpy as np

with open("lprnet.json") as f:
  model_json2 = json.load(f)

with open("lprnet.json") as f:
  model_json = json.load(f)
subgraphs = model_json2['subgraphs'][0]
subgraphs['operators'][19]
model_json['subgraphs'][0]['operators'][17]
model_json['subgraphs'][0]['operators'][18]
model_json['subgraphs'][0]['operators'][18] = deepcopy(subgraphs['operators'][17]) # reuse the other conv layer
model_json['subgraphs'][0]['operators'][18]['builtin_options']['stride_w'] = 88
model_json['subgraphs'][0]['operators'][18]['inputs'] = [19,58,56]
model_json['subgraphs'][0]['operators'][18]['outputs'] = [57]
model_json['subgraphs'][0]['operators'][18]
model_json['subgraphs'][0]['tensors'][58]
model_json['subgraphs'][0]['tensors'][58]['name'] = 'Conv_15/Conv2D/ReadVariableOp'
model_json['subgraphs'][0]['tensors'][58]['shape'] = [64,1,88,71]
model_json['subgraphs'][0]['tensors'][58] # only the weights need to change shape
                                          # while the biases don't change
# Save the modified version to json
with open('model_dense2conv_mod.json', 'w') as f:
    json.dump(model_json, f)

# Convert from json to tflite (flatbuffer compiler)
!flatc -b --strict-json --defaults-json schema.fbs model_dense2conv_mod.json

# Compile for the edgetpu both models together to share the cache
!edgetpu_compiler -s model_dense2conv_mod.tflite ssdlite_ocr.tflite<jupyter_output>Edge TPU Compiler version 15.0.340273435

Models compiled successfully in 1565 ms.

Input model: model_dense2conv_mod.tflite
Input size: 1.40MiB
Output model: model_dense2conv_mod_edgetpu.tflite
Output size: 1.79MiB
On-chip memory used for caching model parameters: 1.69MiB
On-chip memory remaining for caching model parameters: 246.00KiB
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 1
Total number of operations: 25
Operation log: model_dense2conv_mod_edgetpu.log

Operator                       Count      Status

RESHAPE                        2          Mapped to Edge TPU
CONCATENATION                  1          Mapped to Edge TPU
FULLY_CONNECTED                1          Mapped to Edge TPU
MAX_POOL_2D                    3          Mapped to Edge TPU
CONV_2D                        17         Mapped to Edge TPU
LOGISTIC                       1          Mapped to Edge TPU

Input model: ssdlite_ocr.tflite
Input size: 3.00MiB
Output model[...]<jupyter_text>Testing the convertion<jupyter_code># Now we install tflite_runtime package to evaluate the model (Python 3.7 => cp37).
!pip3 install -q https://github.com/google-coral/pycoral/releases/download/release-frogfish/tflite_runtime-2.5.0-cp37-cp37m-linux_x86_64.whl  
#
# Uncomment one of the lines bellow to download a test image:
#

!wget https://stage-drupal.car.co.uk/s3fs-public/styles/original_size/public/2019-09/why-are-number-plates-yellow-and-white.jpg -O car_with_plate.jpg
# !wget https://i.pinimg.com/originals/b3/c9/5e/b3c95e698e61c2d7ac9b076f61194fcc.jpg -O car_with_plate.jpg
# !wget https://www.menswearstyle.co.uk/content/blogs/2c23c162-3858-4d36-a38f-632b6a09dc40_blog_ln_.jpg -O car_with_plate.jpg
from PIL import Image
from PIL import ImageDraw
from IPython.display import display


def draw_bboxes(image, boxes, classes, scores, threshold=0.5, labels={1:'mask', 2: 'no_mask', 3:'poor_mask'}):
  colors = ((128, 255, 102), (102, 255, 255), (232, 123, 212))
  colors = dict(zip(labels.keys(),colors))
  image_width, image_height = image.size
  draw = ImageDraw.Draw(image)
  # Visualization of the results of a detection.
  for i in range(len(boxes)):
    if scores[i] > threshold:
      ymin = int(max(1, (boxes[i][0] * image_height)))
      xmin = int(max(1, (boxes[i][1] * image_width)))
      ymax = int(min(image_height, (boxes[i][2] * image_height)))
      xmax = int(min(image_width, (boxes[i][3] * image_width)))
      draw.rectangle((xmin, ymin, xmax, ymax), width=3, outline=colors[int(classes[i])])
      text = f'{labels[int(classes[i])]} {scores[i]*100:1.2f}%'
      draw.text((xmin+3, ymin-10), text, fill=colors[int(classes[i])], width=2)
  return image, (xmin, ymin, xmax, ymax)
from tflite_runtime.interpreter import Interpreter
# import tensorflow as tf
# Interpreter = tf.lite.Interpreter

class inference_for_single_image_tflite:
  def __init__(self, path_to_model):
    self.interpreter = Interpreter(path_to_model)
    self.interpreter.allocate_tensors()
    self.interpreter.invoke() # warmup

    self.input_details = self.interpreter.get_input_details()
    self.output_details = self.interpreter.get_output_details()
    self.width = self.input_details[0]['shape'][2]
    self.height = self.input_details[0]['shape'][1]

  def __call__(self, input_tensor):
    self.interpreter.set_tensor(self.input_details[0]['index'], input_tensor)
    self.interpreter.invoke()

    return [self.interpreter.get_tensor(self.output_details[out_i]['index'])[0] for out_i in range(len(self.output_details))]
PATH_TO_TFLITE = 'ssdlite_ocr.tflite'
run_inference_for_single_image_tflite_ssd = inference_for_single_image_tflite(PATH_TO_TFLITE)
import numpy as np

image = Image.open("car_with_plate.jpg")
image = image.convert('RGB')
image_width, image_height = image.size

# tflite will not have the image resizing in the model
input_width = run_inference_for_single_image_tflite_ssd.width
input_height = run_inference_for_single_image_tflite_ssd.height

if image_width!=input_width or image_height!=input_height:
  resized_image = image.resize((input_width, input_height))
  np_image = np.asarray(resized_image)
else:
  np_image = np.asarray(image)

image_np_expanded = np.expand_dims(np_image, axis=0)

# Actual detection.
boxes, classes, scores, _ = run_inference_for_single_image_tflite_ssd(image_np_expanded)

classes = [i+1 for i in classes] # class indices here start at 0

img, box = draw_bboxes(image.copy(), boxes, classes, scores, threshold=0.5, labels={1:'plate'})
plate = image.crop(box)
plate
# This LPRNet was trained on Chinese license plates:
char2value = {'0': 0, '1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8, '9': 9, '<Anhui>': 10, '<Beijing>': 11, '<Chongqing>': 12, '<Fujian>': 13, '<Gansu>': 14, '<Guangdong>': 15, '<Guangxi>': 16, '<Guizhou>': 17, '<Hainan>': 18, '<Hebei>': 19, '<Heilongjiang>': 20, '<Henan>': 21, '<HongKong>': 22, '<Hubei>': 23, '<Hunan>': 24, '<InnerMongolia>': 25, '<Jiangsu>': 26, '<Jiangxi>': 27, '<Jilin>': 28, '<Liaoning>': 29, '<Macau>': 30, '<Ningxia>': 31, '<Qinghai>': 32, '<Shaanxi>': 33, '<Shandong>': 34, '<Shanghai>': 35, '<Shanxi>': 36, '<Sichuan>': 37, '<Tianjin>': 38, '<Tibet>': 39, '<Xinjiang>': 40, '<Yunnan>': 41, '<Zhejiang>': 42, '<police>': 43, 'A': 44, 'B': 45, 'C': 46, 'D': 47, 'E': 48, 'F': 49, 'G': 50, 'H': 51, 'I': 52, 'J': 53, 'K': 54, 'L': 55, 'M': 56, 'N': 57, 'O': 58, 'P': 59, 'Q': 60, 'R': 61, 'S': 62, 'T': 63, 'U': 64, 'V': 65, 'W': 66, 'X': 67, 'Y': 68, 'Z': 69, '_': 70}

# Generates a dictionary to revert:
value2char = {v:k for k,v in char2value.items()}
PATH_TO_TFLITE = 'model_dense2conv_mod.tflite'
run_inference_lprnet_conv = inference_for_single_image_tflite(PATH_TO_TFLITE)

PATH_TO_TFLITE = 'lprnet.tflite'
run_inference_lprnet_dense = inference_for_single_image_tflite(PATH_TO_TFLITE)
image_width, image_height = plate.size

# tflite will not have the image resizing in the model
input_width = run_inference_lprnet_dense.width
input_height = run_inference_lprnet_dense.height

if image_width!=input_width or image_height!=input_height:
  resized_image = plate.resize((input_width, input_height))
  np_image = np.asarray(resized_image)
else:
  np_image = np.asarray(plate)

image_np_expanded = np.expand_dims(np_image, axis=0)

lprnet_dense_output = run_inference_lprnet_dense(image_np_expanded)[0]
lprnet_conv_output = run_inference_lprnet_conv(image_np_expanded)[0]
lprnet_conv_output.shape
lprnet_conv_output
abs(lprnet_dense_output-lprnet_conv_output).sum()
plate_characters = " "
output_characters = ["_"]
for c in lprnet_conv_output.argmax(axis=1)[7:]:
  output_characters.append(str(value2char[c]))
  # Ignores if repeated or "_"
  if (output_characters[-1] == output_characters[-2]) or (output_characters[-1] == "_"):
    continue
  plate_characters += str(output_characters[-1])

plate_characters<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14296457847634805, 'fraction_numerical': 0.04572459953372941, 'mean_word_length': 4.709746672391584, 'pattern_counts': {'":': 1, '<': 51, '<?xml version=': 0, '>': 54, 'https://': 22, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 6, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '290495', 'license': '0BSD', 'n_tokens_mistral': 5355, 'n_tokens_neox': 4857, 'n_words': 1076, 'path': 'SSDLite_(plate)_and_LPRNet_(ocr)_2_EdgeTPU_ANPR_ALPR.ipynb', 'repo_name': 'ricardodeazambuja/ALPR-EdgeTPU-RPI0', 'size': '13305'}