<jupyter_start><jupyter_text>Scraping non-tabular dataLook at the first section of CEO data:```https://sandeepmj.github.io/scrape-example-page/organized```  This won't be as simple as scraping a table:* Where and how is the content held on the page?* How can we access it?* Is there a pattern?* Is there anything that breaks the pattern?<jupyter_code>## import libraries
from bs4 import BeautifulSoup
import pandas as pd
import requests ## a library that returns information from websites.
import re ## Regular expressions to help clean our headers
# url to scrape
url = "https://sandeepmj.github.io/scrape-example-page"
## We use the request library to grab page content
## get contents of URL and store in object called page.
## Let's print page and see what we have.

## what type of object is it?

## We'll used BeautifulSoup to convert content into content we recognize

## prettify the print so it's easier to see the HTML tree.<jupyter_output><empty_output><jupyter_text>Non-tabular data in classes<jupyter_code>## soup captures the entire page.
## we narrow it down to capture our target section and store in object called dis

def sanitize_label(label):
    """This function takes a string and lowercases it; strips it of trailing and leading spaces; and puts underscores between words"
    """
    value = label.lower().replace(":", "").strip()
    value = re.sub(r'[^A-Za-z0-9]+', '_', value)
    return value
## find the labels (column headers)
## using for loop and store in list called labels_fl


## find the labels (column headers)
## using list comprehension save in a list called labels_lc

## find class ceo and store in object called ceos

## GET RID OF SPAN DATA<jupyter_output><empty_output><jupyter_text>BUILDING A DICTIONARY  USING { }<jupyter_code>## let's store ceo vitals stats in a list of dicts called all_ceos_dict_list<jupyter_output><empty_output><jupyter_text>BUILDING A DICTIONARY USING DICT AND ZIP<jupyter_code>## let's store ceo vitals stats in a list of dicts called all_ceos_dict_list


def export2csv(a_list, filename):
    '''
    provide list name first
    provide filename as a string
    '''
    df = pd.DataFrame(a_list)
    df.to_csv(filename, encoding='utf-8', index=False)
    print(f"{filename} is in your project folder!")
## export to csv
export2csv(all_ceos_dict_list, "ceo_ranks.csv")<jupyter_output><empty_output><jupyter_text>Scrape the non-tabular data in id="disorganized"<jupyter_code>## soup captures the entire page.
## we narrow it down to capture our target section and store in object called dis

## find the labels (column headers)
## using for loop and store in list called labels_fl


## find the labels (column headers)
## using list comprehension save in a list called labels_lc

## find class ceo and store in object called ceos



## we only printed them out. let's store in a list of dicts called ceo_dict_list

## we only printed them out. let's store in a list of dicts called ceo_dict_list

## use pandas to write to csv file
filename = "disorganized.csv" ## what are file name is
df = pd.DataFrame(ceo_dict_list) ## we turn our list of dicts into a dataframe which we're call df
df.to_csv(filename, encoding='utf-8', index=False) ## export to csv as utf-8 coding (it just has to be this)

print(f"{filename} is in your project folder!") ## a print out that tells us the file is ready<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08586603717910889, 'fraction_numerical': 0.002360578341693715, 'mean_word_length': 4.726351351351352, 'pattern_counts': {'":': 1, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '78763', 'license': 'MIT', 'n_tokens_mistral': 1043, 'n_tokens_neox': 1006, 'n_words': 439, 'path': 'week_04/week-4-inclass-non-tabular-scrape_BLANK.ipynb', 'repo_name': 'sandeepmj/fall20-student-practical-python', 'size': '3389'}