<jupyter_start><jupyter_text>HRIS Invoice Recognition Project<jupyter_code># string extraction tools
import re
from Levenshtein import distance

# opencv
import cv2

# images 
from wand.image import Image
from wand.color import Color
try:
    from PIL import Image as P_image
except ImportError:
    import Image as P_image

# ocr engine
import pytesseract

# others
from datetime import datetime
import os
import numpy as np
import pandas as pd
from tabulate import tabulate
import warnings
warnings.filterwarnings("ignore")
!jt -t onedork -f roboto -fs 10 -tfs 11 -T<jupyter_output>'jt' 不是内部或外部命令，也不是可运行的程序
或批处理文件。<jupyter_text>Please change your directory to computer_vision  Intro  Prerequisite:- Text extraction process:    1. pdfminer.six pip    2. regex    3. pip install python-Levenshtein  - OCR process    1. PDF to Image:        - ImageMagick windows, put it into path and name it as MAGICK_HOME        - ghostscript windows        - wand pip    2. Opencv    3. tesseract windows    4. pytesseract pip          Process:- if it's a PDF file?    - If yes, can we extract the text through pdfminer without error?        - if yes, use pdfminer        - if no, we convert it into PNGs, do some opencv processing and start the OCR process- We have the text file- We use regex and Levenshtein distance to extract text- We score the output- We go back to the OCR & pdfminder stepes and use another way trying to get a higher score----------  OCR1. PDF to Image2. Opencv preprocessing3. Tesseract<jupyter_code>def ocr_process(filename, resolution=450):
    """ Convert a PDF into images, 
        preprocess them using opencv, 
        and then feed them into Tesseract ocr engine.
    """
    txt = ""
    all_pages = Image(filename=filename, resolution=resolution)
    for i, page in enumerate(all_pages.sequence):
        with Image(page) as img:
            img.format = 'png'
            img.background_color = Color('white')
            img.alpha_channel = 'remove'

            image_filename = os.path.splitext(os.path.basename(filename))[0]
            image_filename = '{}-{}.png'.format(image_filename, i)
            path_filename = os.path.join('converted_image', image_filename)
            
            try:
                os.mkdir('converted_image')
            except:
                pass
            
            img.save(filename=path_filename) # save it to the output path
            
            # 1. 转化为灰度图
            im = cv2.imread(path_filename)
            im_gray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY)
            
            # 这个在Invoice上用不多。当我们需要做Receipt时，需要这个。
            
#             # 2. 用adaptive threshold对图像进行二值化处理
#             im_inv = cv2.adaptiveThreshold(im_gray,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C,cv2.THRESH_BINARY,9,2)

#             # 3. 进行降噪处理
#             kernel = 1/16*np.array([[1,2,1],[2,4,2],[1,2,1]])
#             im_blur = cv2.filter2D(im_inv, -1, kernel)
            
            try:
                os.mkdir('preprocessed_image')
            except:
                pass

            # save it to preprocessed_image
            path_filename2 = os.path.join('preprocessed_image', image_filename)
            
            cv2.imwrite(path_filename2,im_gray)
            
            txt = "".join([txt,pytesseract.image_to_string(P_image.open(path_filename2),lang="eng")])
            
    return txt<jupyter_output><empty_output><jupyter_text>OpenCV这里有个问题，字体需要是黑色，背景是白色，反过来是不行的。这里简单的threshold不太好。最好的方法是locally adaptive thresholding<jupyter_code># We put the function here for future use...
# We do not use it in INVOICE process.

def pre_processing(filename):
    # 1. 转化为灰度图
    im = cv2.imread(directory + 'converted_image/' + filename)
    im_gray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY)
    # 2. 用adaptive threshold对图像进行二值化处理
    im_inv = cv2.adaptiveThreshold(im_gray,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C,cv2.THRESH_BINARY,9,2)

    # 3. 进行降噪处理
    kernel = 1/16*np.array([[1,2,1],[2,4,2],[1,2,1]])
    im_blur = cv2.filter2D(im_inv, -1, kernel)

    try:
        os.mkdir('preprocessed_image')
    except:
        pass
    
    # save it to preprocessed_image
    cv2.imwrite('preprocessed_image/' + filename,im_inv)<jupyter_output><empty_output><jupyter_text>Tesseract ocrTesseract result is really satisfying. All we need to do is preprocessing the image and run one line of code.<jupyter_code># test codes
print(ocr_process('test_image/03-19 AvePoint Inc. Inv 106203.pdf'))<jupyter_output>The Web Benefits Design Technology Center

Web Benefits Design Corporation WEB BENEFITS DESIGN ~

4725 West Sand Lake Road, Suite 300 SN aL) Qesicey
Orlando, Florida 32819 I

 

Bill To: Invoice No.
| me 106203
AvePoint, Inc.
525 Washington Blvd., Suite 1400
Jersey City, NJ 07310 3/4/2019 3/19/2019
Attn: <NAME>hut Payments
§ 1,240.35 $ -1,240.35

Item Amount
Previous Balance 1,240.35
Payments / Credits *Thank You* -1,240.35
Past Due Balance 0.00

 

Services for AvePoint, Inc.: 1,241.35
Customized Benefits Website - Fee Waived $200/month $ 0.00
Benefits Administration [211 @ $4.00] $ 844.00
Call Center Year-Round [211 @ $0.85] $ 179.35
Discrepancy Report Management [218 @ $0.25] $ 54.50
COBRA Administration [218 @ $0.75] $ 163.50

Balance Due: |$ 1,241.35

Thank you for your business!

Please Remit Payment to:

Web Benefits Design Corporation
4725 West Sand Lake Road, Suite 300
Orlando, Florida 32819<jupyter_text>PDF Text ExtractionThe layout is not as good as the OCR output, but the accuracy for words is 100%.**The next task is to know how to get a nice layout using pdfminer.****IMPORTANT** Change char_margin to 30 if you want to match vat<jupyter_code>from pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter
from pdfminer.converter import TextConverter
from pdfminer.layout import LAParams
from pdfminer.pdfpage import PDFPage
from io import StringIO

def convert_pdf_to_txt(path, line_overlap=0.5, char_margin=5, line_margin=0.5, boxes_flow=0.5):
    rsrcmgr = PDFResourceManager()
    retstr = StringIO()
    codec = 'utf-8'
    laparams = LAParams(line_overlap=line_overlap, char_margin=char_margin, line_margin=line_margin, word_margin=0.1, boxes_flow=boxes_flow)
    device = TextConverter(rsrcmgr, retstr, codec=codec, laparams=laparams)
    fp = open(path, 'rb')
    interpreter = PDFPageInterpreter(rsrcmgr, device)
    password = ""
    maxpages = 0
    caching = True
    pagenos=set()

    for page in PDFPage.get_pages(fp, pagenos, maxpages=maxpages, password=password,caching=caching, check_extractable=True):
        interpreter.process_page(page)

    text = retstr.getvalue()

    fp.close()
    device.close()
    retstr.close()
    return text<jupyter_output><empty_output><jupyter_text>Text Extraction<jupyter_code>def amount_checker(regex_findall,distance_str):
    '''
    This is the function to check for amount,
    based on different criteria.
    
    1. regex criteria
    2. levenshtein rating criteria
    
    return a dataframe
    '''
    counter = 0
    
    df = pd.DataFrame(columns=['string','amount','rating'])
    
    for ind, item in enumerate(regex_findall):
        if ('tax' in item.lower()) or ('last' in item.lower()):
            del regex_findall[ind]
        else:
            amount = re.search('[0-9]{1,15}.{1,15}[0-9]{2}',item)
            
            if amount is not None:
            
                rating = distance(distance_str, item.lower())

                amount = amount.group(0).replace(',','')
                
                # record it in the dataframe
                df.loc[counter] = [item,amount,rating]
                df[['amount','rating']] = df.loc[:,['amount','rating']].astype(float)
                
                # if the amount is 0, we drop it. It's false.
                df = df[df['amount'] != 0]
                counter += 1

# testing code below:
#     if len(regex_findall) > 0:
#         print(tabulate(df.sort_values(by='rating',ascending=True),headers=('string','amount','rating'),tablefmt='psql'))
#                 print('The string is: {}'.format(item))
#                 print('The amount is: {}'.format(amount))
#                 print('The rating is: {}'.format(rating))
#                 print('-'*20)
    return df
def leven_amount(txt):
    '''
    This is a warpper for amount_checker.
    In here, three kinds of amount are checked.
    '''
    amount_str_ls = re.findall('(?<!Tax )(?<!Sub)(?<!Sub )(Total[^0-9]{1,30}[0-9,]*\.\d\d)', txt, re.IGNORECASE)
    amount_df = amount_checker(amount_str_ls,"Total: USD \$%d.%d".lower())
    
    balance_str_ls = re.findall('(?<!Previous )(?<!Prior )(?<!Ending )(?<!Past Due )(Balance[^0-9]{1,30}[0-9,]*\.\d\d)', txt, re.IGNORECASE)
    balance_df = amount_checker(balance_str_ls,"Balance due: USD \$%d.%d".lower())

    due_str_ls = re.findall('(Amount Due[^0-9]{1,30}[0-9,]*\.\d\d)', txt, re.IGNORECASE)
    due_df = amount_checker(due_str_ls,"Amount due: USD \$%d.%d".lower())

    # add a column for each one above
    if len(amount_df) !=0:
        amount_df.loc[:,"Criteria"] = "Amount"
    if len(balance_df) !=0:
        balance_df.loc[:,"Criteria"] = "Balance"
    if len(due_df) !=0:
        due_df.loc[:,"Criteria"] = "Due"
    
    # append them all
    df = amount_df.append([balance_df,due_df])
    
#     print("amount analysis end")
    
    return df<jupyter_output><empty_output><jupyter_text>Total Amount  Amount<jupyter_code>def most_common(lst):
    return max(set(lst), key=lst.count)

def reg_amount(txt):

    # prepare three kinds of amount
    amount_due = re.findall('Due[^0-9]*\$\s*[0-9,]*\.\d\d', txt, re.IGNORECASE)
    amount_pay = re.findall('Pay[^0-9]*\$\s*[0-9,]*\.\d\d', txt, re.IGNORECASE)
    amount_total = re.findall('Total[^0-9]*\$\s*[0-9,]*\.\d\d', txt, re.IGNORECASE)
    pure_amount = re.findall('\$\s*[0-9,]*\.\d\d', txt, re.IGNORECASE)

    # BlaBlaBla
    amount_ls = []
    counter = 0
    
    try:
        for i in (amount_due, amount_pay, amount_total, pure_amount):
            for amount in i:
                amount_ls.append(float(amount.replace(',','').split('$')[1]))
                counter += 1

        print(amount_ls)
        print(''*20)

        print('The most common amount in the list is: {}'.format(most_common(amount_ls)))
        print('The maximum amount in the list is: {}'.format(max(amount_ls)))
    except:
        print("parse amount error")
    
#     return max(amount_ls), most_common(amount_ls)<jupyter_output><empty_output><jupyter_text>Net & Vat还没有好例子呢。需要继续搞！EMEA需要  Date- Due date- Invoice date- Bill date- Statement date<jupyter_code>from dateutil.parser import parse


def regex_date(txt):
    
    bill_date = re.findall('(?<=Bill Date)[^a-zA-Z]*.*\d\d\d\d.*(?=\s)', txt, re.IGNORECASE)
    invoice_date = re.findall('(?<=Invoice Date)[^a-zA-Z]*.*\d\d\d\d.*(?=\s)', txt, re.IGNORECASE)
    try:
        for ind, i in enumerate(bill_date):
            bill_date[ind] = parse(i.replace(':','').strip()).strftime("%m/%d/%Y")
        for ind, i in enumerate(invoice_date):
            invoice_date[ind] = parse(i.replace(':','').strip()).strftime("%m/%d/%Y")
    except:
        print("parse date error!")

    print(bill_date)
    print(invoice_date)

    if len(bill_date) != 0:
        print('The most common Bill Date in the list is: {}'.format(most_common(bill_date)))
    if len(invoice_date) != 0:
        print('The most common Invoice Date in the list is: {}'.format(most_common(invoice_date)))
        
#     return most_common(bill_date), most_common(invoice_date)<jupyter_output><empty_output><jupyter_text>Invoice  or Account<jupyter_code>def regex_invoice_number(txt):
    invoice_number = re.findall('INVOICE[^0-9]*\d\d\d[^a-zA-Z]*', txt, re.IGNORECASE)
    account_number = re.findall('Account[^0-9]*\d\d\d[^a-zA-Z]*', txt, re.IGNORECASE)
    customer_number = re.findall('Customer[^0-9]*\d\d\d[^a-zA-Z]*', txt, re.IGNORECASE)
    ref_number = re.findall('Reference[^0-9]*\d\d\d[^a-zA-Z]*', txt, re.IGNORECASE)
    sales_order_number = re.findall('Sales order[^0-9]*\d\d\d[^a-zA-Z]*', txt, re.IGNORECASE)


    def number_checker(regex,name):
        for ind, i in enumerate(regex):
            regex[ind] = re.search("\d.*\d", regex[ind]).group(0)

        print('{} list: {}'.format(name,regex))

        if len(regex) != 0:
            print('The most common {} in the list is: {}'.format(name,most_common(regex)))

        print('-'*20)
        
#         return most_common(regex)

    number_checker(invoice_number,'invoice number')
    number_checker(account_number,'account number')
    number_checker(customer_number,'customer number')
    number_checker(ref_number,'reference number')
    number_checker(sales_order_number,'sales order number')<jupyter_output><empty_output><jupyter_text>Vendor Name- Vendor Name- Vendor Address  Vendor Address<jupyter_code>def regex_vendor_address(txt):
    vendor_address = re.findall('.{1,30}\n\d{1,30}.{1,30}\n.{1,30}\n.{1,15}\n.{1,30}', txt, re.IGNORECASE)
    if len(vendor_address) != 0:
        common_address = most_common(vendor_address)
        print('-'*20)
        print(common_address)
        print('-'*20)

        vendor_name = common_address.splitlines()[0]
        print(vendor_name)<jupyter_output><empty_output><jupyter_text>Vendor Remittance Address<jupyter_code>def regex_remittance(txt):
    for i in re.findall('To.*\n.{1,30}\n.{1,30}\n.{1,30}\n.{1,15}', txt, re.IGNORECASE):
        print(i)
        print('-'*20)<jupyter_output><empty_output><jupyter_text>NS Vendor Name<jupyter_code># import pandas as pd
# from fuzzywuzzy import fuzz
# from fuzzywuzzy import process

#     df = pd.read_excel('reference/vendor.xlsx')
#     df.dropna(inplace=True)
#     df['Full Name'] = df.apply(lambda x: str(x['ID']) + ' ' + x['Name'], axis=1)
#     df.set_index('Name',inplace=True)

#     fuzz_result = process.extract('Harvard Services Group, Inc.',df.index,limit=3)
#     result_ls = []

#     for i in fuzz_result:
#         result_ls.append(df.loc[i[0],'Full Name'])

#     for i in result_ls:
#         print(i)<jupyter_output><empty_output><jupyter_text>AvePoint Address  Description...  Main MEMO for pdfminer:char_margin = 5 ~ 100 (100几乎就顶到pdf的头头了)char_margin 的调节可以解决，行内，非box隔开字符的问题，e.g. Total amountline_margin .5 ~ 1，box_flow<=1 可以更好的识别addressline_margin = 5 也可以把行都去掉了，把它变小到0.5可以解决box问题boxes_flow = 0.5 ~ 5boxes_flow = 5 可以把所有字符挤在一起box > 1.5 line > 1.5 所有的字儿就变成一行了<jupyter_code>def regex_extraction(txt):
    '''
    This controls what functions will be run.
    You can use this to test each function independently.
    '''
    return leven_amount(txt)
#     reg_amount(txt)
#     regex_date(txt)
#     regex_vendor_address(txt)
#     regex_remittance(txt)
def balance_rating_up(x):
    if x['Criteria'] == 'Balance':
        return x['rating'] + 2
    else:
        return x['rating']
    
def ocr_rating_down(x):
    if x['Process'] == 'OCR process':
        return x['rating'] - 3
    else:
        return x['rating']
    
    
def agg_dfs(df):
    if len(df) > 1:
        df.loc[:,['amount','rating']].astype(float)
        agg_df = df.copy()
        agg_df['rating'] = df.apply(balance_rating_up,axis=1)
        agg_df['rating'] = df.apply(ocr_rating_down, axis=1)

        agg_df = (df.groupby('amount')
              .aggregate({'string':len,'rating':np.mean})
              .sort_values(by='rating',ascending=True)
              .reset_index())
        agg_df.loc[:,'string'].astype(float)
        agg_df['final rating'] = agg_df['rating'] - (agg_df['string'] - 1)
        agg_df.drop('rating',axis=1,inplace=True)
        return agg_df
    else:
        return df
df = pd.DataFrame({
    'Process':['2','2','2','2','2', 'OCR process'],
    'Criteria':['Balance',2,3,4,5,6],
    'string':[1,2,3,4,5,6],
    'amount':[66,66,44,44,44,33],
    'rating':[1,2,3,4,5,6]
})
df
df['rating'] = df.apply(balance_rating_up,axis=1)
df['rating'] = df.apply(ocr_rating_down, axis=1)
df
df = (df.groupby('amount')
      .aggregate({'string':len,'rating':np.mean})
      .sort_values(by='rating',ascending=True)
      .reset_index())
df['final rating'] = df['rating'] - (df['string'] - 1)
df.drop('rating',axis=1,inplace=True)

print(tabulate(df,tablefmt='psql',showindex=False,headers=('amount','frequency','final rating')))
# Main Loop
startTime = datetime.now()

# set your working directory
directory = 'D:/git/Invoice-Receipt-OCR/'
dir_test_img = directory + 'test_image/'

# reference for pdfminer looping. This matters a lot!
argu = [(5, 0.5, 5), (100, 1, 5),(5, 1.5, 1.5)]

# main loop
for filename in os.listdir(dir_test_img): 
    dir_file = "".join([dir_test_img, filename])
    print('Analysing pdf {}...'.format(filename))
    
    # If it's a pdf file, then...
    if filename.endswith(".pdf"): 
        # **************************************pdfminer process starts
        print('\nStarting PDFminer process...')
        
        counter = 1
        df = pd.DataFrame()
        
        # Perform PDF Miner process:
        # Looping three times with different settings
        for i, j, k in argu:
            print('Performing option {} for pdfminer'.format(counter))
            txt = convert_pdf_to_txt(dir_file, char_margin=i, line_margin=j, boxes_flow=k)
            tem_df = regex_extraction(txt)
            
            # Add a column to tem_df to show it's from PDFminer process
            if (tem_df is not None) and (len(tem_df) != 0):
                tem_df.loc[:,"Process"] = "PDF Miner option {}".format(counter)
            
            # Append all DataFrames together
            df = df.append(tem_df)
            counter += 1
        # **************************************pdfminer process ends
        
        # **************************************ocr process starts
        print('\nStarting ocr process...')
        txt = ocr_process(dir_file)
        tem_df = regex_extraction(txt)
                       
        if (tem_df is not None) and (len(tem_df) != 0):
            tem_df.loc[:,"Process"] = "OCR process"
        
        # Append all DataFrames from the above processes together
        df = df.append(tem_df)
        
        # print initial rating
        df = df.loc[:,['Process','Criteria','string','amount','rating']]
        print('\nThe initial rating is:')
        print(tabulate(df.sort_values(by='rating'),tablefmt='psql',headers=('Process','Criteria','string','amount','rating')))
        
        # aggregate ratings based on amount
        # do some ajustment on ratings
        # print final aggregated rating
        agg_df = agg_dfs(df)
                       
        print('\nThe final aggregated rating is:')
        print(tabulate(agg_df,tablefmt='psql',showindex=False,headers=('amount','frequency','final rating')))
        # **************************************ocr process ends

        
        
        
    else: # if it's not a pdf file
#         txt = ocr_process('test_image/' + filname) # We use a OCR process
#         regex_extraction(txt)
        pass
        
    print("-"*20 + "\n")


print(datetime.now() - startTime)<jupyter_output>Analysing pdf 03-19 AvePoint Inc. Inv 106203.pdf...

Starting PDFminer process...
Performing option 1 for pdfminer
Performing option 2 for pdfminer
Performing option 3 for pdfminer

Starting ocr process...

The initial rating is:
+----+-------------+------------+--------------------------+----------+----------+
|    | Process     | Criteria   | string                   |   amount |   rating |
|----+-------------+------------+--------------------------+----------+----------|
|  0 | OCR process | Balance    | Balance Due: |$ 1,241.35 |  1241.35 |       10 |
+----+-------------+------------+--------------------------+----------+----------+

The final aggregated rating is:
+-------------+---------+--------------------------+-------------+----------------+
|             |         | amount                   |   frequency |   final rating |
|-------------+---------+--------------------------+-------------+----------------|
| OCR process | Balance | Balance Due: |$ 1,241.35 |     1241.35 |    [...]
{'fraction_non_alphanumeric': 0.1617925940712646, 'fraction_numerical': 0.024952590078850184, 'mean_word_length': 2.7323523933693425, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 55, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '291723', 'license': 'MIT', 'n_tokens_mistral': 7190, 'n_tokens_neox': 6793, 'n_words': 1486, 'path': 'zDONOTUSEComputer Vision Project_updated.ipynb', 'repo_name': 'air-yan/Invoice-Receipt-OCR', 'size': '20046'}