<jupyter_start><jupyter_text>Optical Character RecognitionThis notebook explores and compares different methods of optical character recognition Tesseract OCR and Google Vision API. OCR is an important tool, if we want to process large quantities of printed or hand-written textual data. The accompanying blog post can be found [here](https://fuzzylabs.ai/blog/the-battle-of-the-ocr-engines/)The examples provided:* Sample prescription (amoxicillin, a type of antibiotic): `gs://fuzzylabs-jupyter-delicacies/amoxicillin_prescription.png`* Receipt from a hotel (with a hand-written note): `gs://fuzzylabs-jupyter-delicacies/receipt.jpg`You are encouraged to try it on different images (local files and Google Cloud Storage are supported) to experiment and identify which methods and models best suit your needs. Run this cell to install the necessary packages, if they are not already installed.<jupyter_code>!pip install tesserocr matplotlib opencv-python google-cloud-vision gcsfs
import io
import matplotlib.pyplot as plt
from cv2 import cv2
import locale
locale.setlocale(locale.LC_ALL, 'C')
from tesserocr import *
from google.cloud import vision
from PIL import Image
import numpy as np
import gcsfs
def show_image(img, title=""):
    plt.figure(figsize=(5,88))
    plt.imshow(img)
    plt.title(title)
    plt.show()
    
# Accepts either a local file 
def imread_wrapper(uri):
    if uri.startswith("gs://"):
        with gcsfs.GCSFileSystem().open(uri, "rb") as f:
            arr = np.asarray(bytearray(f.read()), dtype="uint8")
            img = cv2.imdecode(arr, cv2.IMREAD_COLOR)
            return img
    else: # Assume local file
        return cv2.imread(uri, cv2.IMREAD_COLOR)<jupyter_output><empty_output><jupyter_text>Load the image (filename specified in `img`) to run the recognition on.You can optionally set `PERFORM_THRESHOLDING` to `True`, to perform thresholding. It is advised to preprocess an image if there are unwanted objects in the background (e.g. img/amoxicillin_prescription.png)<jupyter_code>PERFORM_THRESHOLDING = False
img = imread_wrapper("gs://fuzzylabs-jupyter-delicacies/receipt.jpg")
if PERFORM_THRESHOLDING:
    _, img = cv2.threshold(cv2.cvtColor(img, cv2.COLOR_BGR2GRAY), 0, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)
    img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
show_image(img)<jupyter_output><empty_output><jupyter_text>Tesseract OCRTesseract can run in different page segmentation modes. In many cases it is enough to run it on auto (PSM.AUTO), but it is often useful to provide it with some hint on what the layout of the input image is.We consider three modes here:* `PSM.SINGLE_BLOCK` -- Assume a single uniform block of text.* `PSM.SPARSE_TEXT` -- Sparse text. Find as much text as possible in no particular order.* `PSM.SINGLE_COLUMN` -- Assume a single column of text of variable sizes.The illustration of how they behave is provided below.One can read more on different PSMs in Tesseract OCR documentation.<jupyter_code>tesseract_modes = [
    ("Tesseract SINGLE BLOCK", PyTessBaseAPI(psm=PSM.SINGLE_BLOCK)),
    ("Tesseract SPARSE TEXT", PyTessBaseAPI(psm=PSM.SPARSE_TEXT)),
    ("Tesseract SINGLE COLUMN", PyTessBaseAPI(psm=PSM.SINGLE_COLUMN))
]
def tesseract_ocr(tesseract, img):
    pil_image = Image.fromarray(img)
    tesseract.SetImage(pil_image)
    tesseract.Recognize()
    return tesseract.GetIterator()

def display_tesseract_results(img, level):
    n = len(tesseract_modes)
    fig, axs = plt.subplots(1,n, figsize=(5*n, 88))
    for (name, tesseract), ax in zip(tesseract_modes, axs):
        results = tesseract_ocr(tesseract, img)
        _img = img.copy()
        tokens = []
        for r in iterate_level(results, level):
            text = r.GetUTF8Text(level)
            tokens += [text]
            bb = r.BoundingBox(level)
            cv2.rectangle(_img, (bb[0], bb[1]), (bb[2], bb[3]), (255, 0, 0), 2)
        print(name)
        print(tokens)
        print()
        ax.set_title(name)
        ax.imshow(_img)<jupyter_output><empty_output><jupyter_text>The results (i.e. recognised texts and their locations) that are returned by Tesseract are arranged in a hierarchy, and therefore can be accessed at different levels. In this notebook, we show the results by paragraph, text line and word. One may find other levels (namely blocks of text and characters) useful for their application.  By paragraph<jupyter_code>display_tesseract_results(img, RIL.PARA)<jupyter_output><empty_output><jupyter_text>By text line<jupyter_code>display_tesseract_results(img, RIL.TEXTLINE)<jupyter_output><empty_output><jupyter_text>By word<jupyter_code>display_tesseract_results(img, RIL.WORD)<jupyter_output><empty_output><jupyter_text>Google Vision APIAs an alternative to Tesseract OCR, we also explore Google Vision. As opposed to Tesseract, it does not run locally, but on the Google's servers. In order to use it, valid credentials need to be provided in `GOOGLE_APPLICATION_CREDENTIALS` environment variable (more details in the Google's documentation)<jupyter_code>google_image_annotator = vision.ImageAnnotatorClient()
def google_document_text_detection(img):
    buf = io.BytesIO()
    img_pil = Image.fromarray(img)
    img_pil.save(buf, 'PNG')
    byte_image = buf.getvalue()

    document_image = vision.types.Image(content=byte_image)
    image_response = google_image_annotator.document_text_detection(image=document_image, image_context={"language_hints": ["en"]})
    page = image_response.full_text_annotation.pages[0]
    return page

def google_text_detection(img):
    buf = io.BytesIO()
    img_pil = Image.fromarray(img)
    img_pil.save(buf, 'PNG')
    byte_image = buf.getvalue()

    document_image = vision.types.Image(content=byte_image)
    image_response = google_image_annotator.text_detection(image=document_image, image_context={"language_hints": ["en"]})
    page = image_response.full_text_annotation.pages[0]
    return page<jupyter_output><empty_output><jupyter_text>Google Vision API does not have options for differents modes of page segmentation, however there's still some degree of flexibility provided. There are two models that are tuned on two different tasks:* Text Detection model -- detects and extracts text from any image* Document Text Detection model -- performs the same operation, but is tuned for dense texts and documentsThe choice of the model depends on the nature and structure of the images to be recognised (same as with PSM of Tesseract OCR). We show the results produced by both models in this notebook.<jupyter_code>pages = [
    ("Google Text Detection", google_text_detection(img)),
    ("Google Document Text Detection", google_document_text_detection(img)),
]
def get_symbol_text(symbol):
    text = symbol.text
    if symbol.property.HasField('detected_break'):
        if symbol.property.detected_break == vision.enums.TextAnnotation.DetectedBreak.BreakType.LINE_BREAK:
            text += "\n"
        else:
            text += " "
    return text

def get_word_text(word):
    return "".join([get_symbol_text(x) for x in word.symbols])

def get_paragraph_text(para):
    return "".join([get_word_text(x) for x in para.words])

def get_block_text(block):
    return "".join([get_paragraph_text(x) for x in block.paragraphs])
def add_bounding_box(img, bb):
    cv2.polylines(img, [np.array([[point.x, point.y] for point in bb])], True, (255, 0, 0), 2)
    

def get_google_words(page, img):
    _img = img.copy()
    texts = []
    for block in page.blocks:
        for para in block.paragraphs:
            for word in para.words:
                texts += [get_word_text(word)]
                add_bounding_box(_img, word.bounding_box.vertices)
    return texts, _img
    
def get_google_paragraphs(page, img):
    _img = img.copy()
    texts = []
    for block in page.blocks:
        for para in block.paragraphs:
            texts += [get_paragraph_text(para)]
            add_bounding_box(_img, para.bounding_box.vertices)
    return texts, _img
    
def get_google_blocks(page, img):
    _img = img.copy()
    texts = []
    for block in page.blocks:
        texts += [get_block_text(block)]
        add_bounding_box(_img, block.bounding_box.vertices)
    return texts, _img
    
def display_google_results(pages, img, func):
    n = len(pages)
    fig, axs = plt.subplots(1,n, figsize=(5*n, 88))
    for (name, page), ax in zip(pages, axs):
        texts, _img = func(page, img)
        print(name)
        print(texts)
        print()
        ax.imshow(_img)
        ax.set_title(name)
    plt.show()<jupyter_output><empty_output><jupyter_text>Similar to Tesseract OCR, Google Vision also provides results in a hierarchical manner. Hence, they can be browsed at different levels. As an example, iteration over text blocks, paragraphs and individual words are shown below. By text block<jupyter_code>display_google_results(pages, img, get_google_blocks)<jupyter_output><empty_output><jupyter_text>By paragraph<jupyter_code>display_google_results(pages, img, get_google_paragraphs)<jupyter_output><empty_output><jupyter_text>By word<jupyter_code>display_google_results(pages, img, get_google_words)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09509269356597601, 'fraction_numerical': 0.0056706652126499455, 'mean_word_length': 4.360023378141438, 'pattern_counts': {'":': 2, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '468444', 'license': 'Apache-2.0', 'n_tokens_mistral': 2962, 'n_tokens_neox': 2903, 'n_words': 833, 'path': 'vision/ocr.ipynb', 'repo_name': 'fuzzylabs/computational-delicacies-jupyter', 'size': '9170'}