<jupyter_start><jupyter_text>PLSE ETL - Part 2: Scraping PDFs using _pdfminer_ libraryNote: Using Sample Data Set (count=199)<jupyter_code># @note Python 3 req. ```pip install pdfminer.six``` - https://buildmedia.readthedocs.org/media/pdf/pdfminer-docs/latest/pdfminer-docs.pdf
import subprocess
from os import makedirs, listdir
from os.path import exists, isfile, join, splitext
import re
def getGenericDatum(a, b):
    c = re.search(a, b)
    if c and c.group():
        #print(c)
        #print(len(c.group()))
        print(c.group())
    return
path_to_src = '../tmp/sample_dockets'
path_to_dest = '../tmp/extracted'
if not exists(path_to_dest):
    makedirs(path_to_dest)
dockets = [f for f in listdir(path_to_src) if isfile(join(path_to_src, f))]<jupyter_output><empty_output><jupyter_text>the loop below converts each sample docket from the given PDF format to a __single__ text fileNote: dockets can span multiple pages<jupyter_code>for docket in dockets:
    try:
        filename = splitext(docket)[0]
        cmd = 'pdf2txt.py -o {}/{}.txt {}/{}.pdf'.format(path_to_dest, filename, path_to_src, filename)

        subprocess.call(cmd, shell=True)

        assert isfile('{}/{}.txt'.format(path_to_dest, filename)) is True, "Text file failed to generate: %r" % dest_file
    except:
        print("Something else went wrong...")
dockets = [f for f in listdir(path_to_dest) if isfile(join(path_to_dest, f))]
docket_ref_a = 'CP-51-CR-0100031-2003.txt'
otn = re.compile('(?<=OTN\:\s{2})\w\s\d{6}\-\d+')
odn = re.compile('(?<=Docket\sNumber\:\s)\w{2}\-\d{2}\-\w{2}\-\d{7}\-\d{4}')
df = re.compile('(?<=Date\sFiled\:\s{2})\d{2}\/\d{2}\/\d{4}')

fullname = re.compile('(?<=Commonwealth\sof\sPennsylvania\n\nv\.\n\n)(\w|\s)+(?=\nCASE\sINFORMATION)')
ja_test_strings = ['Judge Assigned: Temin, <NAME>']
ja = re.compile('(?<=Judge\sAssigned\:\s{2})\w+\,\s\w+(\s\w\.)?')
dispo_events = re.compile('[0-9]\s\/\s[a-zA-Z0-9\s\/\-]+') # OffenseDescription
statues = re.compile('[0-9]+\s\u00A7\s[0-9]+')
def getEvents(a):
    b = re.findall(dispo_events, a)
    if len(b):
        print(len(b), b)
        
    c = re.findall(statues, a)
    if len(c):
        print(len(c), c)      
    return
#demo_docket = '{}/{}'.format(path_to_dest, docket_ref_a)#dockets[23])
demo_docket = '{}/{}'.format(path_to_dest, dockets[103])
print(demo_docket)
with open(demo_docket, 'r') as f:
    #lines = f.readlines()
    
    data = f.read()
    
    print('\n======= Scrapped Values\n')    
    
    print('Generic:')
    getGenericDatum(otn, data)
    getGenericDatum(odn, data)
    getGenericDatum(df, data)
    getGenericDatum(ja, data)
    getGenericDatum(fullname, data)
    
    print('Event:')
    getEvents(data)

    print('\n======= Source\n\n', data)<jupyter_output>======= Scrapped Values

Generic:
MC-51-SU-0001345-1978
11/29/1978
<NAME>
<NAME>
Event:
1 ['1 / Retail Theft-Take Mdse \n\nDisposition Date\n\nOffense Disposition\n\nSentence Date\n\nFinal Disposition\n\nGrade\n\nSection\n\nCredit For Time Served\n\nIncarceration/Diversionary Period\n\nStart Date\n\n12/13/1978\n\nDismissed\n\nFinal Disposition\n\nS\n\n18 ']
2 ['18 § 3929', '18 § 3929']

======= Source

 MUNICIPAL COURT OF PHILADELPHIA COUNTY  

DOCKET

Docket Number: MC-51-SU-0001345-1978
SUMMARY DOCKET
Non-Traffic

Commonwealth of Pennsylvania

v.

<NAME>
CASE INFORMATION

Judge Assigned:  <NAME>
OTN:  
Initial Issuing Authority:  
Arresting Agency:  Philadelphia Pd
Complaint/Incident #:  780606801201
Case Local Number Type(s)

LOTN:  

District Control Number
Legacy Docket Number

Case Status:

Closed

Date Of Birth:

10/13/1957

Participant Type
Defendant

Date Filed:  11/29/1978
Originating Docket No:  
Final Issuing Authority:  
Arresting Officer: [...]
{'fraction_non_alphanumeric': 0.17073170731707318, 'fraction_numerical': 0.03618332886625569, 'mean_word_length': 3.796915167095116, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '460372', 'license': 'MIT', 'n_tokens_mistral': 1573, 'n_tokens_neox': 1454, 'n_words': 302, 'path': 'etl/notebooks/etl_part2_pdfminer.ipynb', 'repo_name': 'Philadelphia-Lawyers-for-Social-Equity/etl-db-env', 'size': '3770'}