<jupyter_start><jupyter_text>Contents1&nbsp;&nbsp;Words in the filing2&nbsp;&nbsp;Pages in the filing3&nbsp;&nbsp;Count images4&nbsp;&nbsp;Find the sections5&nbsp;&nbsp;Top 5 Salaries6&nbsp;&nbsp;Extracting information<jupyter_code>import re
import os
from collections import Counter
from IPython.display import HTML<jupyter_output><empty_output><jupyter_text>Using the filing at `cik=1652044`, `accession=0001308179-17-000170` ([link](https://www.sec.gov/Archives/edgar/data/1652044/000130817917000170/lgoog2017_def14a.htm)), read in the HTML and answer the following questions:  1. How many words are in the filing?  1. How many pages are in the filing?  1. How many images are included in the filing?  1. What are the different sections of the proxy statement? (hint: see the table of contents we found above).  1. What are the top 5 people (by salary) paid at Google?  1. *WITHOUT CODING*: Describe how you would go about extracting this information programatically.     1. What format is this information in?     1. Do you think it's repeatable for not-Google?     1. Would you use HTML or plain text to get this information?     1. Extra credit: Write out some [pseudo-code](https://www.vikingcodeschool.com/software-engineering-basics/what-is-pseudo-coding) for your approach, or just the steps in plain english.<jupyter_code># Import the filing
import html2text
from bs4 import BeautifulSoup
from pyedgar import Filing

# Load the filing
filing = Filing(1652044, '0001308179-17-000170')

# Get the HTML copy
html = filing.documents[0]['full_text']

# Get the beautifulsoup object of the HTML
soup = BeautifulSoup(html, 'lxml')

# Get the plaintext version of the html
h = html2text.HTML2Text()
h.ignore_links = False
h.ignore_tables = False
plaintext = h.handle(html)<jupyter_output><empty_output><jupyter_text>Words in the filing<jupyter_code># Using the word finder from last homework
re_word_finder = re.compile(
    r'\b'                 # word-boundry, so match beginning or ends of words
    r"(?:[^\W\d_]|[-'])+" # (?: is a non-capture group, just means we can use | for or.
                          # ^\W\d_ means not (^) a non-word, or a digit or underscore
    r'\b'                 # ending word-boundry
    , re.IGNORECASE
)
# Counting words in the plaintext
num_words = len(re_word_finder.findall(plaintext))
print(f"Length of text: {len(plaintext):,d}")
print(f"Number of words: {num_words:,d}")
# Counting words in the plaintext
num_words = len(re_word_finder.findall(soup.get_text()))
print(f"Length of text: {len(soup.get_text()):,d}")
print(f"Number of words: {num_words:,d}")<jupyter_output><empty_output><jupyter_text>That's odd, I would think that plaintext (which leaves more formatting in) would be longer.Let's take a look at why.<jupyter_code>ex_sec = plaintext.index("Post-Employment and Change in Control Payments")
print(plaintext[ex_sec: ex_sec+2000])
ex_sec = soup.get_text().index("Post-Employment and Change in Control Payments")
print(soup.get_text()[ex_sec: ex_sec+2000])<jupyter_output><empty_output><jupyter_text>Let's do a wordcount to see the top words<jupyter_code>num_pt = Counter(map(lambda x: x.lower(), re_word_finder.findall(plaintext)))
num_pt.most_common(10)
num_soup = Counter(map(lambda x: x.lower(), re_word_finder.findall(soup.get_text())))
num_soup.most_common(10)<jupyter_output><empty_output><jupyter_text>What words are in one and not the other?<jupyter_code>in_pt_not_soup = set(num_pt.keys()) - set(num_soup.keys())
in_pt_not_soup
in_soup_not_pt = set(num_soup.keys()) - set(num_pt.keys())
in_soup_not_pt<jupyter_output><empty_output><jupyter_text>Do those words alone explain the discrepancy?<jupyter_code>sum(1 for w in re_word_finder.findall(plaintext) if w in in_pt_not_soup)
sum(1 for w in re_word_finder.findall(soup.get_text()) if w in in_soup_not_pt)
dif_sum = 0
for k in num_pt.keys():
    dif = num_pt[k] - num_soup[k]
    if dif != 0:
        print(k, num_pt[k], num_soup[k])
    dif_sum += dif
print(dif_sum)
dif_sum = 0
for k in num_soup.keys():
    dif = num_soup[k] - num_pt[k]
    if dif != 0:
        print(k, num_soup[k], num_pt[k])
    dif_sum += dif
print(dif_sum)<jupyter_output><empty_output><jupyter_text>I notice that alphabet seems to be pretty different:<jupyter_code>num_pt['alphabet'], num_soup['alphabet']<jupyter_output><empty_output><jupyter_text>Let's look through and see what matches from each text:<jupyter_code>_ = [w for w in re_word_finder.findall(plaintext) if 'alphabet' in w.lower()]
len(_)
_ = [w for w in re_word_finder.findall(soup.get_text()) if 'alphabet' in w.lower()]
len(_)<jupyter_output><empty_output><jupyter_text>If we look through, we see that plaintext gets lots of apostrophies, but soup.get_text() doesn't.<jupyter_code>_txt = soup.get_text()
for match in re_word_finder.finditer(_txt):
    if 'alphabet' not in match.group(0).lower():
        continue
    print(_txt[match.start()-10:match.end()+10].replace('\n', '\t'))
for match in re_word_finder.finditer(plaintext):
    if 'alphabet' not in match.group(0).lower():
        continue
    print(plaintext[match.start()-10:match.end()+10].replace('\n', '\t'))
# Using the word finder from last homework
re_word_finder = re.compile(
    r'\b'                  # word-boundry, so match beginning or ends of words
    r"(?:[^\W\d_]|[-'’])+" # Add ’, which is not just single quote, but single end quote
    r'\b'                  # ending word-boundry
    , re.IGNORECASE
)
# Counting words in the plaintext
num_words = len(re_word_finder.findall(plaintext))
print(f"Length of text: {len(plaintext):,d}")
print(f"Number of words: {num_words:,d}")
# Counting words in the plaintext
num_words = len(re_word_finder.findall(soup.get_text()))
print(f"Length of text: {len(soup.get_text()):,d}")
print(f"Number of words: {num_words:,d}")<jupyter_output><empty_output><jupyter_text>Magnifique.See, regexes take lots of work, most of it iterative like this.  Pages in the filing Most EDGAR filings are broken up into pages. Maybe not all, but a lot.They look like this:<jupyter_code>soup.findAll(style='font: 9pt Arial, Helvetica, Sans-Serif; margin: 0pt 0; text-align: right', limit=2)[-1]<jupyter_output><empty_output><jupyter_text>It's just a break between pages, so we can look for those and count them.Looking at the page numbers will help us verify that we're doing it right.<jupyter_code># re.DOTALL is needed to let .* match newlines
alpha_to_number = re.compile(r'^\s*alphabet.*[^\d]+\d{1,3}\s*$', re.IGNORECASE | re.DOTALL)

pages = []
for match in soup.findAll('p'):
    txt = match.get_text()
    if not alpha_to_number.search(txt):
        continue
    print(txt.strip().replace('\n', ' '))
    pages.append(txt.strip().split()[-1])
print(pages)
print(f"Number of pages: {len(pages)}")<jupyter_output><empty_output><jupyter_text>Obviously, we started at 4, so really there's 84 pages.But close enough.  Count images<jupyter_code>num_images = len(soup.findAll('img'))
print(f"Number of images: {num_images:,d}")<jupyter_output><empty_output><jupyter_text>What are these images?<jupyter_code>soup.findAll('img')<jupyter_output><empty_output><jupyter_text>Find the sections From `3_Scraping.ipynb`, You might remember we found the table of contents by looking for headers.Let' try that again.<jupyter_code>soup.findAll(text=re.compile('table.{0,20}contents', re.I))
toc = soup.findAll(text=re.compile('table.{0,20}contents', re.I))[0]
toc.parent
toc.parent.parent
if toc.parent.parent.parent.name == 'body':
    print("We've gone far enough.")<jupyter_output><empty_output><jupyter_text>So let's see if the TOC is the first table after that header we found above.<jupyter_code>HTML(toc.parent.parent.findNext('table').prettify())<jupyter_output><empty_output><jupyter_text>Because we always look at our data when we develop algorithms, we instantly notice that the CD&A is missing.Why?Stupid pages.<jupyter_code>html = toc.parent.parent.findNext('table')
html.findNext('table')<jupyter_output><empty_output><jupyter_text>Ug, time to go look at the filing again.[Here's the link again](https://www.sec.gov/Archives/edgar/data/1652044/000130817917000170/lgoog2017_def14a.htm).<jupyter_code>table1 = toc.parent.parent.findNext('table')
table2 = (table1.findNext(text=re.compile('compensation\s+discussion\s+and\s+analysis', re.I))
                .findParent('table'))
HTML(table1.prettify() + table2.prettify())<jupyter_output><empty_output><jupyter_text>How can we be sure that we're done?That's the hard part.You might think of looking for tables that just have words and one number, which sequentially increase.Or you could think of classifying each heading you can think of and then looking for all of them, grabbing the tables that enclose them.Hopefully you now see that parsing HTML is hard, and requires a lot of iteration on documents to see what is and isn't working.<jupyter_code>i = 1
for tab in [table1, table2]:
    for h1 in tab.findAll(attrs={'style':re.compile('font: 12pt Arial, Helvetica, Sans-Serif;')}):
        text = h1.get_text().strip()
        if len(text) <= 5:
            # These are the page numbers
            continue
        text = re.sub('\s+', ' ', text)
        print("{})".format(i), text)
        i += 1<jupyter_output><empty_output><jupyter_text>Top 5 Salaries I always start by finding it myself.For this, the inspector in chrome is absolutely invaluable.Learn it well, it will be your friend.[Documentation](https://developers.google.com/web/tools/chrome-devtools/inspect-styles/)I'm going to start by finding the "Summary Compensation Table", and taking the first table after that with salary in the header.Well that's my plan, we'll see if it works.<jupyter_code>sum_comp = soup.findAll(text='Summary Compensation Table')
sum_comp<jupyter_output><empty_output><jupyter_text>We've been here before, we probably want the second one.If we're in a table, that's the wrong one (TOC).<jupyter_code># The carat means the start of the text, so not in the middle of a paragraph
for sum_comp in soup.findAll(text=re.compile('^summary\s+compensation\s*table', re.I)):
    print(sum_comp)
    if sum_comp.findParent('table'):
        print("We're in a table, move on.")
        continue
    
    print("We're not in a table. Go with this one.")
    break
sum_comp<jupyter_output><empty_output><jupyter_text>Now let's find the next tables.<jupyter_code>sum_table = (sum_comp.findNext('table', text=re.compile('salary', re.I))
                     .findParent('table'))<jupyter_output><empty_output><jupyter_text>Hmm, I wonder why not.Let's try some things:<jupyter_code>sum_comp.findNext('table', text=re.compile('salary', re.I))
sum_comp.findNext('table', text=re.compile('salary', re.I))<jupyter_output><empty_output><jupyter_text>Hmm, nothing.Don't ask why I'd try this, but what about row instead of table?<jupyter_code>sum_comp.findNext('tr', text=re.compile('salary', re.I))
sum_comp.findNext('td', text=re.compile('salary', re.I))<jupyter_output><empty_output><jupyter_text>Okay... odd.Well this works I guess.<jupyter_code>sum_table = (sum_comp.findNext('td', text=re.compile('salary', re.I))
                     .findParent('table'))
HTML(sum_table.prettify())<jupyter_output><empty_output><jupyter_text>That's the table we want.Now let's grab all the first columns with names in them.<jupyter_code>for row in sum_table.findAll('tr'):
    cell = row.find('td') # the first cell
    print(cell.get_text())<jupyter_output><empty_output><jupyter_text>Hmm, don't like all that noise.A quick look shows that the names are bolded.Let's just keep the bold then.<jupyter_code>for row in sum_table.findAll('tr'):
    # The named individuals are in bold.
    bold_name_in_cell = row.find('td').find('b')
    print(bold_name_in_cell.get_text())<jupyter_output><empty_output><jupyter_text>Oh, some are none if 'b' isn't found. Ignore those.<jupyter_code>for row in sum_table.findAll('tr'):
    # The named individuals are in bold.
    bold_name_in_cell = row.find('td').find('b')
    if not bold_name_in_cell:
        continue
    print(bold_name_in_cell.get_text())<jupyter_output><empty_output><jupyter_text>Lastly, we just want that first name in the cell, not the super-script (6):<jupyter_code>for row in sum_table.findAll('tr'):
    # The named individuals are in bold.
    bold_name_in_cell = row.find('td').find('b')
    if not bold_name_in_cell:
        continue
    print(bold_name_in_cell.contents[0])<jupyter_output><empty_output><jupyter_text>So now we want to drop that first one.We know that these tables will have the hear associated with the salary in them, so let's check for that.<jupyter_code>for row in sum_table.findAll('tr'):
    # The named individuals are in bold.
    if not row.find(text=re.compile('(?:19|20)\d\d')):
        continue
    bold_name_in_cell = row.find('td').find('b')
    if not bold_name_in_cell:
        continue
    print(bold_name_in_cell.contents[0])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12593795930997137, 'fraction_numerical': 0.015780923648178232, 'mean_word_length': 5.092365692742695, 'pattern_counts': {'":': 0, '<': 122, '<?xml version=': 0, '>': 121, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '30', 'dir': 'jupyter-structured-clean-dedup', 'id': '171522', 'license': 'MIT', 'n_tokens_mistral': 4612, 'n_tokens_neox': 4423, 'n_words': 1180, 'path': 'notebooks/3a_Scraping-answers.ipynb', 'repo_name': 'gaulinmp/edgar_shortcourse', 'size': '12927'}