<jupyter_start><jupyter_text>Demonstration of `pdfplumber`'s table-extraction optionsThis notebook uses a [report from the FBI's National Instant Criminal Background Check System](../pdfs/background-checks.pdf).  Import `pdfplumber`<jupyter_code>import pdfplumber<jupyter_output><empty_output><jupyter_text>Load the PDF<jupyter_code>pdf = pdfplumber.open("../pdfs/background-checks.pdf")<jupyter_output><empty_output><jupyter_text>Get the first page<jupyter_code>p0 = pdf.pages[0]
im = p0.to_image()
im<jupyter_output><empty_output><jupyter_text>What data would we get if we used the default settings?We can check by using `PageImage.debug_tablefinder()`:<jupyter_code>im.reset().debug_tablefinder()<jupyter_output><empty_output><jupyter_text>The default settings correctly identify the table's vertical demarcations, but don't capture the horizontal demarcations between each group of five states/territories. So:  Using custom `.extract_table`'s settings- Because the columns are separated by lines, we use `vertical_strategy="lines"`- Because the rows are, primarily, separated by gutters between the text, we use `horizontal_strategy="text"`- And because the left and right-hand extremities of the text aren't quite flush with the vertical lines, we use `"intersection_tolerance": 15`<jupyter_code>table_settings = {
    "vertical_strategy": "lines",
    "horizontal_strategy": "text",
    "intersection_x_tolerance": 15
}
im.reset().debug_tablefinder(table_settings)
table = p0.extract_table(table_settings)
for row in table[:5]:
    print(row)<jupyter_output>['November - 2015', None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]
['Pre-Pawn Redemption Returned/Disposition Rentals Private Sale Return to Seller - Private Sale', None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]
['State / Territory', 'Permit', 'Handgun', 'Long Gun', '*Other', '**Multiple', 'Admin', 'Handgun', 'Long Gun', '*Other', 'Handgun', 'Long Gun', '*Other', 'Handgun', 'Long Gun', '*Other', 'Handgun', 'Long Gun', 'Handgun', 'Long Gun', '*Other', 'Handgun', 'Long Gun', '*Other', 'Totals']
['Alabama', '18,870', '23,022', '22,650', '859', '1,178', '0', '14', '15', '0', '2,179', '2,307', '11', '0', '0', '0', '', '', '13', '14', '0', '3', '2', '0', '71,137']
['Alaska', '209', '3,062', '3,209', '191', '184', '0', '9', '3', '0', '100', '100', '0', '18', '9', '1', '', '', '0', '0', '0', '0', '0', '[...]<jupyter_text>Cleaning up the data`.extract_table` worked with our custom settings, but the table it detected contains extraneous headers and footers. Since we know that the `Alabama` row is the first, and that there are 56 rows we care about (50 states + DC + 4 territories + the "Totals" row), we can slice away the rest:<jupyter_code>core_table = table[3:3+56]<jupyter_output><empty_output><jupyter_text>The first row:<jupyter_code>" • ".join(core_table[0])<jupyter_output><empty_output><jupyter_text>The last:<jupyter_code>" • ".join(core_table[-1])<jupyter_output><empty_output><jupyter_text>Now, let's turn those rows into dictionaries, and also convert strings-representing-numbers to the numbers themselves, e.g., `"18,870" -> 18870`:<jupyter_code>COLUMNS = [
    "state",
    "permit",
    "handgun",
    "long_gun",
    "other",
    "multiple",
    "admin",
    "prepawn_handgun",
    "prepawn_long_gun",
    "prepawn_other",
    "redemption_handgun",
    "redemption_long_gun",
    "redemption_other",
    "returned_handgun",
    "returned_long_gun",
    "returned_other",
    "rentals_handgun",
    "rentals_long_gun",
    "private_sale_handgun",
    "private_sale_long_gun",
    "private_sale_other",
    "return_to_seller_handgun",
    "return_to_seller_long_gun",
    "return_to_seller_other",
    "totals"
]
def parse_value(i, x):
    if i == 0: return x
    if x == "": return None
    return int(x.replace(",", ""))
from collections import OrderedDict
def parse_row(row):
    return OrderedDict((COLUMNS[i], parse_value(i, cell))
        for i, cell in enumerate(row))
data = [ parse_row(row) for row in core_table ]<jupyter_output><empty_output><jupyter_text>Now here's the first row, parsed:<jupyter_code>data[0]<jupyter_output><empty_output><jupyter_text>Sort the dataFor demonstration purposes, let's list the rows with the highest number of handgun-only background checks:<jupyter_code>for row in list(reversed(sorted(data, key=lambda x: x["handgun"])))[:6]:
    print("{state}: {handgun:,d} handgun-only checks".format(**row))<jupyter_output>Totals: 671,330 handgun-only checks
Pennsylvania: 62,752 handgun-only checks
Texas: 56,941 handgun-only checks
Florida: 50,796 handgun-only checks
California: 41,181 handgun-only checks
Ohio: 34,878 handgun-only checks<jupyter_text>Use `extract_text` to extract the report monthIt looks like the month of the report is listed in an area 35px to 65px from the top of the page. But there's also some other text directly above and below it. So when we crop for that area, we'll use `.within_bbox` instead of `.crop` to select only characters (and other objects) that are fully within the bounding box.<jupyter_code>month_crop = p0.within_bbox((0, 35, p0.width, 65))
month_crop.to_image()
month_chars = month_crop.extract_text()
month_chars<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14679748241392077, 'fraction_numerical': 0.031469825990373936, 'mean_word_length': 5.9716129032258065, 'pattern_counts': {'":': 5, '<': 47, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '197824', 'license': 'MIT', 'n_tokens_mistral': 1975, 'n_tokens_neox': 1823, 'n_words': 532, 'path': 'examples/notebooks/extract-table-nics.ipynb', 'repo_name': 'mrkom1/pdfplumber', 'size': '5402'}