<jupyter_start><jupyter_text>OCR (Optical Character Recognition) from Images with Transformers---[Github](https://github.com/eugenesiow/practical-ml/) | More Notebooks @ [eugenesiow/practical-ml](https://github.com/eugenesiow/practical-ml)--- Notebook to recognise text automaticaly from an input image with either handwritten or printed text.[Optical Character Recognition](https://paperswithcode.com/task/optical-character-recognition) is the task of converting images of typed, handwritten or printed text into machine-encoded text, whether from a scanned document, a photo of a document, a scene-photo (for example the text on signs and billboards in a landscape photo, license plates in cars...) or from subtitle text superimposed on an image (for example: from a television broadcast).The [transformer models used](https://malaya-speech.readthedocs.io/en/latest/tts-singlish.html) are from Microsoft's TrOCR. The TrOCR models are encoder-decoder models, consisting of an image Transformer as encoder, and a text Transformer as decoder. We utilise the versions hosted on [huggingface.co](https://huggingface.co/models?search=microsoft/trocr) and use the awesome transformers library, for longevity and simplicity.The notebook is structured as follows:* Setting up the Environment* Using the Model (Running Inference)  Setting up the Environment  Dependencies and RuntimeIf you're running this notebook in Google Colab, most of the dependencies are already installed and we don't need the GPU for this particular example. If you decide to run this on many (>thousands) images and want the inference to go faster though, you can select `Runtime` > `Change Runtime Type` from the menubar. Ensure that `GPU` is selected as the `Hardware accelerator`. We need to install huggingface `transformers` for this example to run, so execute the command below to setup the dependencies. We use the version compiled directly from the latest source (at the time of writing this is the only way to access the transforemrs TrOCR model code).<jupyter_code>!pip install -q git+https://github.com/huggingface/transformers.git<jupyter_output>Installing build dependencies ... [?25l[?25hdone
  Getting requirements to build wheel ... [?25l[?25hdone
    Preparing wheel metadata ... [?25l[?25hdone
[K     |████████████████████████████████| 3.3 MB 5.1 MB/s 
[K     |████████████████████████████████| 596 kB 58.0 MB/s 
[K     |████████████████████████████████| 895 kB 35.8 MB/s 
[K     |████████████████████████████████| 56 kB 4.4 MB/s 
[?25h  Building wheel for transformers (PEP 517) ... [?25l[?25hdone<jupyter_text>Using the Model (Running Inference) Let's define a function for us to get images from the web. We execute this function to download an image with a line of handwritten text and display it.<jupyter_code>import requests
from IPython.display import display
from PIL import Image

def show_image(url):
  img = Image.open(requests.get(url, stream=True).raw).convert("RGB")
  display(img)
  return img

handwriting1 = show_image('https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg')<jupyter_output><empty_output><jupyter_text>Now we want to load the model to recognise handwritten text.Specifically we are running the following steps:* Load the processor, `TrOCRProcessor`, which processes our input image and converts it into a sequence of fixed-size patches (resolution 16x16), which are linearly embedded. The processor also adds absolute position embeddings and this sequence is fed to the layers of the Transformer encoder.* Load the model, `VisionEncoderDecoderModel`, which consists of the image encoder and the text decoder.* Define `ocr_image` function - We define the function for inferencing which takes our `src_img`, the input image we have downloaded. It will then run both the processor and the model inference and produce the output OCR text that has been recognised from the image.<jupyter_code>import transformers
from transformers import TrOCRProcessor, VisionEncoderDecoderModel

processor = TrOCRProcessor.from_pretrained('microsoft/trocr-base-handwritten')
model = VisionEncoderDecoderModel.from_pretrained('microsoft/trocr-base-handwritten')

def ocr_image(src_img):
  pixel_values = processor(images=src_img, return_tensors="pt").pixel_values
  generated_ids = model.generate(pixel_values)
  return processor.batch_decode(generated_ids, skip_special_tokens=True)[0]<jupyter_output><empty_output><jupyter_text>We now run our `ocr_image` function on the line of handwritten text in the image we have downloaded previously (and stored in `handwriting1`).<jupyter_code>ocr_image(handwriting1)<jupyter_output><empty_output><jupyter_text>Lets try on another image with handwritten text.<jupyter_code>ocr_image(show_image('https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSoolxi9yWGAT5SLZShv8vVd0bz47UWRzQC19fDTeE8GmGv_Rn-PCF1pP1rrUx8kOjA4gg&usqp=CAU'))
import transformers
from transformers import TrOCRProcessor, VisionEncoderDecoderModel

print_processor = TrOCRProcessor.from_pretrained('microsoft/trocr-base-printed')
print_model = VisionEncoderDecoderModel.from_pretrained('microsoft/trocr-base-printed')

def ocr_print_image(src_img):
  pixel_values = print_processor(images=src_img, return_tensors="pt").pixel_values
  generated_ids = print_model.generate(pixel_values)
  return print_processor.batch_decode(generated_ids, skip_special_tokens=True)[0]<jupyter_output><empty_output><jupyter_text>We download an image with noisy printed text, a scanned receipt.<jupyter_code>receipt = show_image('https://github.com/zzzDavid/ICDAR-2019-SROIE/raw/master/data/img/000.jpg')<jupyter_output><empty_output><jupyter_text>As the model processes a line of text, we crop the image to include on of the lines of text in the receipt and send it to our model.<jupyter_code>receipt_crop = receipt.crop((0, 80, receipt.size[0], 110))
display(receipt_crop)
ocr_print_image(receipt_crop)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10360512129380053, 'fraction_numerical': 0.014487870619946091, 'mean_word_length': 6.205097087378641, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 30, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '4058', 'license': 'MIT', 'n_tokens_mistral': 1946, 'n_tokens_neox': 1749, 'n_words': 584, 'path': 'notebooks/OCR_from_Images_with_Transformers.ipynb', 'repo_name': 'eugenesiow/practical-ml', 'size': '5936'}