<jupyter_start><jupyter_text>OCRWe show how the pipeline performs OCR and how to inspect the results.Reference: [ocr](https://among.github.io/fusus/fusus/ocr.html).<jupyter_code>%load_ext autoreload
%autoreload 2
from fusus.book import Book
B = Book(cd="~/github/among/fusus/example")
# cd to the book directory
!cd `pwd`<jupyter_output><empty_output><jupyter_text>Task runnerThe following function runs the pipeline for one or more pages.If all pages are selected, batch mode is on.<jupyter_code>def checkOcr(pg, quiet=True, **kwargs):
    if pg is None:
        for pg in B.allPagesList:
            page = B.process(
                batch=False,
                pages=pg,
                doOcr=True,
                quiet=quiet,
                **kwargs,
            )
            page.show(stage="proofword,proofchar")
    return B.process(
        batch=False,
        pages=pg,
        doOcr=True,
        quiet=quiet,
        **kwargs,
    )<jupyter_output><empty_output><jupyter_text>Check a single pageOur first example is page 132.The ocr result is shown in data form and as a proof page.<jupyter_code>page = checkOcr(132)<jupyter_output>0.00s Batch of 1 pages: 132
  0.00s Start batch processing images
   |     1.35s Loading for Kraken: ~/github/among/fusus/model/arabic_generalized.mlmodel
   |     8.86s model loaded
   |       15s     1 132.jpg                                 
    15s all done<jupyter_text>Here is the page image with the detected layout:<jupyter_code>page.show(stage="layout")<jupyter_output><empty_output><jupyter_text>We can inspect the results of the OCR by calling for a proof:<jupyter_code>page.show("proofword")<jupyter_output><empty_output><jupyter_text>There is a proof page at character level as well.<jupyter_code>page.show(stage="proofchar")<jupyter_output><empty_output><jupyter_text>**N.B.:** If you work with a file in an online GitHub repo, and if your local files are in a directoryunder `~/github/`, then the links above will show those files in NB-Viewer, provided the repo in question has beenpushed to Github.If you are experimenting locally, you can use the paths to the local files to open themin your browser yourself. We can also ask for the OCR data. Here are the line, word, and character data:<jupyter_code>page.show(stage="line")<jupyter_output><empty_output><jupyter_text>For the terms *stripe*, *column* and *line*,see: [layout](https://among.github.io/fusus/fusus/layout.html).<jupyter_code>page.show(stage="word")<jupyter_output><empty_output><jupyter_text>Here is the OCR data at character level:<jupyter_code>page.show(stage="char")<jupyter_output><empty_output><jupyter_text>StatisticsHere is some statistics about the general quality of the OCR for this page.See [measureOcr.ipynb](measureOcr.ipynb) for more.<jupyter_code>B.measureQuality(132)<jupyter_output>33s Batch of 1 pages: 132
    33s Start measuring ocr quality of these images
   |     0.00s word-confidences of OCR results for 1 pages<jupyter_text>**N.B.:** If you work with a file in an online GitHub repo, and if your local file is in a directoryunder `~/github/`, then the links above refer to files in NB-Viewer, provided the repo in question has beenpushed to Github.If you are experimenting locally, you see the path to the local file when you hover over it, and you have to open itin your browser yourself. **For developers**If you are tweaking the generation and formatting of the proofpages, you do not need to perform the costly OCR process over and over again.The following method regenerates the proof pages on the basis of the existing OCR data.<jupyter_code>page.proofing()<jupyter_output><empty_output><jupyter_text>If you need to regenerate all proofing pages, this will work:```B.measureQuality(None, updateProofs=True)```  Check all pagesNow we check all example pages in batch<jupyter_code>page = checkOcr(None)<jupyter_output>0.00s Batch of 1 pages: 47
  0.00s Start batch processing images
   |     3.21s     1 047.tif                                 
  3.22s all done<jupyter_text>Measure the quality of all these pages (and update all proof pages)<jupyter_code>B.measureQuality(updateProofs=True)<jupyter_output>37s Batch of 18 pages: 47-48,58-59,63,67,101-102,111-113,121-122,131-132,200,300,400
    37s Start measuring ocr quality of these images
    37s   end regenrating proof files
   |     0.10s word-confidences of OCR results for 18 pages<jupyter_text>LegendHere is how we color the degrees of confidence reported by the Kraken OCR engine.We translate a confidence (a number between 0 and 100 including) into a HSL color:<jupyter_code>from fusus.ocr import getProofColor
for i in range(101):
    clr = getProofColor(100 - i, test=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08565989847715735, 'fraction_numerical': 0.028130287648054145, 'mean_word_length': 4.112432432432432, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '85607', 'license': 'MIT', 'n_tokens_mistral': 1597, 'n_tokens_neox': 1473, 'n_words': 495, 'path': 'notebooks/example/ocr.ipynb', 'repo_name': 'annotation/fusus', 'size': '4728'}