<jupyter_start><jupyter_text>Testing Tesseract OCR performance on custom fontsThe interactive notebook and necessary files are available on GitHub:`https://github.com/byewokko/tesseract-font-test`<jupyter_code>import matplotlib.pyplot as plt
import matplotlib.axes
import numpy as np
import os
import pandas as pd
%matplotlib inline

from utils import *<jupyter_output><empty_output><jupyter_text>IntroductionHumans are able to read text written in various typefaces or scripts and usually they don't even realize the differences between them. For a computer vision however, those small differences are crucial and a single decorative curl can cause a letter to be completely unrecognizable.In this task I perform two experiments in order to evaluate Tesseract software on text set in different typefaces.To keep this notebook less bulky, I have removed all my custom functions to `utils.py`.  Sample experimentBecause of licensing issues I cannot freely redistribute many of the font files I use in the main experiments below. Here I provide a sample experiment to demonstrate my pipeline (without the data visualization) using the _Gentium Plus_ fonts which are licensed under SIL Open Font License and are included in this repository. Hence this experiment is fully reproducible.<jupyter_code>preview_fonts(get_font_names("."))<jupyter_output><empty_output><jupyter_text>For this experiment, we use both _Gentium_ fonts found in cwd to render file `en-eisenhower.txt` in two sizes (20 and 30 points) and save them in `test` folder. The we pass them to the Tesseract engine and evaluate its output. The steps are shown in the following snippet. Both main experiments follow the same procedure.<jupyter_code>outdir = "test" # directory for storing generated files
fontdir = "." # directory with font files
ptsizes = [20,30] # list of font sizes to generate
textfile = "en-eisenhower.txt" # file with sample text
language = "eng" # Tesseract language model

# Read example text file
with open(textfile, "r") as fin:
    rawtext = fin.read()[:500]
    
# Break long lines
text = breaklines(rawtext)

# Read font data of files in fontdir
fontdata = get_font_names(fontdir)

# Setup the experiment, store metadata in a dataframe
setup = make_experiment_setup(fontdata, ptsizes, outdir)

# Render text as images
batch_render_text(text, setup, fontdata)

# OCR the images
batch_ocr_images(setup, language)

# Evaluate OCR output
results = batch_evaluate(setup, fontdata, text)
results<jupyter_output><empty_output><jupyter_text>Experiment 1 OCRing different sizes, weights and widths of one typeface We explore the OCR performance on different fonts of the same typeface superfamily _Helvetica Neue_ by Linotype foundry. It classifies as a transitional modern sans-serif, similarly to Helvetica or Arial. The family uses convenient number coding for different fonts: weights from _Ultra Light_ to _Black_ are coded with numbers from 2 to 9. Similarly the widths _Extended_, _Regular_ and _Condensed_ are coded with 3, 5 and 9 respectively. So for example  47 represents _Light Condensed_. Here we use 6 different weights and 3 widths.  We render each of them in 5 sizes: 10, 16, 22, 28 and 34 points.We use an excerpt from [Czech news article](https://denikn.cz/31964) as our source text. Accordingly, we use the `ces` language model for OCR in Tesseract.<jupyter_code>outdir = "helvetica" # directory for storing generated files
fontdir = "fonts/helvetica" # directory with font files
ptsizes = [10,16,22,28,34] # list of font sizes to generate
textfile = "cz-news.txt" # file with sample text
language = "ces" # Tesseract language model
# Prepare data and preview fonts
# This step requires having the font files in fontdir!

fontdata = get_font_names(fontdir)
setup = make_experiment_setup(fontdata, ptsizes, outdir)
fontdata.to_csv(os.path.join(outdir, "fontdata.csv"))
setup.to_csv(os.path.join(outdir, "setup.csv"))

preview_fonts(fontdata)
# Batch process all files, write results
# This step requires having the font files in fontdir!

with open(textfile, 'r') as file:
    orig_text = file.read()
orig_text = breaklines(orig_text, 60)

batch_render_text(orig_text, setup, fontdata)
batch_ocr_images(setup, "ces")
results = batch_evaluate(setup, fontdata, orig_text)

results.to_csv(os.path.join(outdir, "results.csv"))<jupyter_output>Wrote helvetica/img/00.10.png
Wrote helvetica/img/00.16.png
Wrote helvetica/img/00.22.png
Wrote helvetica/img/00.28.png
Wrote helvetica/img/00.34.png
Wrote helvetica/img/01.10.png
Wrote helvetica/img/01.16.png
Wrote helvetica/img/01.22.png
Wrote helvetica/img/01.28.png
Wrote helvetica/img/01.34.png
Wrote helvetica/img/02.10.png
Wrote helvetica/img/02.16.png
Wrote helvetica/img/02.22.png
Wrote helvetica/img/02.28.png
Wrote helvetica/img/02.34.png
Wrote helvetica/img/03.10.png
Wrote helvetica/img/03.16.png
Wrote helvetica/img/03.22.png
Wrote helvetica/img/03.28.png
Wrote helvetica/img/03.34.png
Wrote helvetica/img/04.10.png
Wrote helvetica/img/04.16.png
Wrote helvetica/img/04.22.png
Wrote helvetica/img/04.28.png
Wrote helvetica/img/04.34.png
Wrote helvetica/img/05.10.png
Wrote helvetica/img/05.16.png
Wrote helvetica/img/05.22.png
Wrote helvetica/img/05.28.png
Wrote helvetica/img/05.34.png
Wrote helvetica/img/06.10.png
Wrote helvetica/img/06.16.png
Wrote helvetica/img/06.22.png
Wrote helv[...]<jupyter_text>Visualising the resultsFirst we extract the width and weight numbers from font names and save them as new variables in the dataframe.<jupyter_code>outdir = "helvetica"
fontdata = pd.read_csv(os.path.join(outdir, "fontdata.csv"))
results = pd.read_csv(os.path.join(outdir, "results.csv"))
results["Weight"] = pd.to_numeric(results["Font"].str[22])
results["Width"] = pd.to_numeric(results["Font"].str[23])
results = results.set_index(["Weight", "Width", "Point size"])[["WER","CER","Font"]]<jupyter_output><empty_output><jupyter_text>Now we plot average error rates per point size, weight and width.<jupyter_code>fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(16,5), sharey=True)
fig.suptitle("Average error rates")
results.groupby("Point size").mean().plot.bar(y=["WER","CER"], ax=axes[0])
results.groupby("Weight").mean().plot.bar(y=["WER","CER"], ax=axes[1])
results.groupby("Width").mean().plot.bar(y=["WER","CER"], ax=axes[2])

for x in axes:
    x.grid(True, axis="y")
    x.set_axisbelow(True)

plt.show()<jupyter_output>/home/m17/hruska/.local/lib/python3.4/site-packages/pandas/plotting/_core.py:1716: UserWarning: Pandas doesn't allow columns to be created via a new attribute name - see https://pandas.pydata.org/pandas-docs/stable/indexing.html#attribute-access
  series.name = label<jupyter_text>In all three cases, WER and CER follow the same curve. It shows that Tesseract is very sensitive to point size and weight, but much less to font width. Not surprisingly it performs better at bigger sizes. Similarly it works better with fonts of thicker weights. There is also a slight preference of extended over condensed weights.<jupyter_code>fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(16,5), sharey=True)
fig.suptitle("Average word error rate")

results.groupby(["Width", "Weight"]).mean()["WER"].unstack(0).plot.bar(ax=axes[0]);
results.groupby(["Point size", "Weight"]).mean()["WER"].unstack(0).plot.bar(ax=axes[1]);
results.groupby(["Width", "Point size"]).mean()["WER"].unstack(0).plot.bar(ax=axes[2]);

for x in axes:
    x.grid(True, axis="y")
    x.set_axisbelow(True)<jupyter_output><empty_output><jupyter_text>Here we plotted average CER by grouped parameters. The middle plot shows that thinner weights are not legible at all. This is mainly a rendering problem, because even in large sizes some strokes are too thin to be displayed.<jupyter_code>results.nsmallest(5,"CER")<jupyter_output><empty_output><jupyter_text>The three best results are tied at the same WER and CER and they are all extended weights. Interestingly, they are all also sizes 28. I have no explanation why the same fonts in size 34 yielded worse results.  Experiment 2 OCRing different typefaces In this experiment we compare OCR error rates on 16 different typefaces rendered at 2 point sizes. This can give us a crude idea about which typefaces work better in smaller sizes and which ones work well generally.The typefaces we chose can be seen below in the preview.<jupyter_code>outdir = "assorted" # directory for storing generated files
fontdir = "fonts/assorted" # directory with font files
ptsizes = [16,32] # list of font sizes to generate
textfile = "cz-news.txt" # file with sample text
language = "ces" # Tesseract language model
# Prepare data and preview fonts

fontdata = get_font_names(fontdir)
setup = make_experiment_setup(fontdata, ptsizes, outdir)
setup.to_csv(os.path.join(outdir, "setup.csv"))

preview_fonts(fontdata)<jupyter_output><empty_output><jupyter_text>We chose mostly different types of common "non-decorative" text typefaces plus three "decorative" ones with substantially different letterforms. Even though we preview all of them in the same point size, some of them are apparently smaller, because of other design parameters, such as x-height.We add some basic classification to the font data:<jupyter_code>fontdata["Proportional"] = "proportional"
fontdata["Serif"] = "sans"
fontdata["Slanted"] = "upright"
fontdata["Decorative"] = "normal"

fontdata.loc[[1,4,11], "Proportional"] = "monospaced"
fontdata.loc[[1,2,3,6,7,15], "Serif"] = "serif"
fontdata.loc[[3,7,9,11,13], "Slanted"] = "slanted"
fontdata.loc[[0,12,13], "Decorative"] = "decorative"

fontdata.to_csv(os.path.join(outdir, "fontdata.csv"))

fontdata
# Batch process all files, write results
# This takes several minutes and requires the font files!

with open(textfile, 'r') as file:
    orig_text = file.read()
orig_text = breaklines(orig_text, 60)

batch_render_text(orig_text, setup, fontdata)
batch_ocr_images(setup, "ces")
results = batch_evaluate(setup, fontdata, orig_text)

results.to_csv(os.path.join(outdir, "results.csv"))<jupyter_output>Wrote assorted/img/00.16.png
Wrote assorted/img/00.32.png
Wrote assorted/img/01.16.png
Wrote assorted/img/01.32.png
Wrote assorted/img/02.16.png
Wrote assorted/img/02.32.png
Wrote assorted/img/03.16.png
Wrote assorted/img/03.32.png
Wrote assorted/img/04.16.png
Wrote assorted/img/04.32.png
Wrote assorted/img/05.16.png
Wrote assorted/img/05.32.png
Wrote assorted/img/06.16.png
Wrote assorted/img/06.32.png
Wrote assorted/img/07.16.png
Wrote assorted/img/07.32.png
Wrote assorted/img/08.16.png
Wrote assorted/img/08.32.png
Wrote assorted/img/09.16.png
Wrote assorted/img/09.32.png
Wrote assorted/img/10.16.png
Wrote assorted/img/10.32.png
Wrote assorted/img/11.16.png
Wrote assorted/img/11.32.png
Wrote assorted/img/12.16.png
Wrote assorted/img/12.32.png
Wrote assorted/img/13.16.png
Wrote assorted/img/13.32.png
Wrote assorted/img/14.16.png
Wrote assorted/img/14.32.png
Wrote assorted/img/15.16.png
Wrote assorted/img/15.32.png
Wrote assorted/img/16.16.png
Wrote assorted/img/16.32.png
Wrote assorted[...]<jupyter_text>Visualising the results<jupyter_code>outdir = "assorted"

fontdata = pd.read_csv(os.path.join(outdir, "fontdata.csv"))
results = pd.read_csv(os.path.join(outdir, "results.csv"))

# Merge results with fontdata
results = pd.merge(results, fontdata[["id","Proportional","Serif","Slanted","Decorative"]], 
                   left_on="Font id", right_on="id", how="left").drop(columns=[list(results)[0], "id", "txt"])
results.groupby(["Font"])["WER","CER"].mean().sort_values("WER").plot.barh(title="Average error rates");
plt.grid(True, axis="both")<jupyter_output><empty_output><jupyter_text>This plot show the error rates for each font averaged over the two sizes. It is sorted by WER. At the top, with the highest error rates there are two of our decorative typefaces: _Plagwitz_, a modern adaptation of German blackletter, and _Petit Formal Script_, which is a connected script. It is followed by the italic weight of _Fell DW Pica_, _Courier New_ and the last decorative font _Orange Kid_. All of them have very distinct letterforms, which explains why they are hard to OCR correctly.On the other end we find _Consolas_, a monospaced sans with occasional serifs on critical narrow characters like `I, l` or `i`. It is followed by _Museo Slab_ and _Sans_, _Helvetica Neue_ from the previous experiment and the _Times New_.<jupyter_code>results.groupby(["Point size","Font"])["WER"].mean().unstack(0).sort_values(32).plot.barh(title="Word error rate by point size");
plt.grid(True, axis="both")<jupyter_output><empty_output><jupyter_text>This figure shows that fonts that work in large sizes do not always work as well in small sizes. This is the case of _Times New Roman Italic_. On the other hand _Comic Sans MS_ does not work as good as Times New in the large size, but outperforms it in the small size.<jupyter_code>fig, axes = plt.subplots(nrows=1, ncols=4, figsize=(16,4), sharey=True)
fig.suptitle("Average word error rate by font types")

results.groupby(["Point size", "Serif"]).mean()["WER"].unstack(0).plot.bar(ax=axes[0]);
results.groupby(["Point size", "Slanted"]).mean()["WER"].unstack(0).plot.bar(ax=axes[1]);
results.groupby(["Point size", "Proportional"]).mean()["WER"].unstack(0).plot.bar(ax=axes[2]);
results.groupby(["Point size", "Decorative"]).mean()["WER"].unstack(0).plot.bar(ax=axes[3]);

for x in axes:
    x.grid(True, axis="y")
    x.set_axisbelow(True)
    x.tick_params("x",labelrotation=0)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11293314535875937, 'fraction_numerical': 0.029976997848185798, 'mean_word_length': 5.552260573650948, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 50, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '511065', 'license': 'MIT', 'n_tokens_mistral': 4711, 'n_tokens_neox': 4431, 'n_words': 1369, 'path': 'tesseract-fonts.ipynb', 'repo_name': 'byewokko/tesseract-font-test', 'size': '13477'}