<jupyter_start><jupyter_text>Introduction to Web ScrapingOften we are interested in getting data from a website.  Modern websites are often built using a [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) framework that has an Application Programming Interface ([API](https://en.wikipedia.org/wiki/Application_programming_interface)) to make [HTTP](https://www.tutorialspoint.com/http/http_requests.htm) requests to retrieve structured data in the form of [JSON](https://en.wikipedia.org/wiki/JSON) or XML.However, when there is not a clear API, we might need to perform web scraping by directly grabbing the data ourselves<jupyter_code>try:
    import requests
except:
    !pip install requests
try:
    from bs4 import BeautifulSoup
except:
    !pip install bs4<jupyter_output><empty_output><jupyter_text>Getting data using requests[Requests](http://docs.python-requests.org/en/master/) is an excellent library for performing HTTP requests.  In this simple example, we will scrape data from the PBS faculty webpage.<jupyter_code>page = requests.get("http://pbs.dartmouth.edu/people")
print(page)<jupyter_output><empty_output><jupyter_text>Here the response '200' indicates that the get request was successful.  Now let's look at the actual text that was downloaded from the webpage.<jupyter_code>print(page.content)<jupyter_output><empty_output><jupyter_text>Here you can see that we have downloaded all of the data from the PBS faculty page and that it is in the form of HTML. HTML is a markup language that tells a browser how to layout content.  HTML consists of elements called tags.  Each tag indicates a beginning and end.  Here are a few examples:  - `` - indicates hyperlink - `` - indicates paragraph - `` - indicates a division, or area, of the page. - `` - bolds any text inside. - `` - italicizes any text inside. - `` - indicates a header - `` - creates a table. - `` - ordered list - `` - unordered list - `` - list item  Parsing HTML using Beautiful SoupThere are many libraries that can be helpful for quickly parsing structured text such as HTML.  We will be using Beautiful Soup as an example.<jupyter_code>soup = BeautifulSoup(page.content, 'html.parser')
print(soup.prettify())<jupyter_output><empty_output><jupyter_text>Here we are going to find the unordered list tagged with the id 'faculty-container'.  We are then going to look for any nested tag that use the 'h4' header tag.  This should give us all of the lines with the faculty names as a list.<jupyter_code>names_html = soup.find_all('ul',id='faculty-container')[0].find_all('h4')
names = [x.text for x in names_html]
print(names)<jupyter_output><empty_output><jupyter_text>What if we wanted to get all of the faculty email addresses?<jupyter_code>email_html = soup.find_all('ul',id='faculty-container')[0].find_all('span',{'class' : 'contact'})
email = [x.text for x in email_html]
print(email)<jupyter_output><empty_output><jupyter_text>Parsing string dataWhat if we wanted to grab the name from the list of email addresses?<jupyter_code>print([x.split('@')[0] for x in email])<jupyter_output><empty_output><jupyter_text>One thing we might do with this data is create a dictionary with names and emails of all of the professors in the department.  This could be useful if we wanted to send a bulk email to them.<jupyter_code>email_dict = dict([(x.split('@')[0],x) for x in email])
print(email_dict)<jupyter_output><empty_output><jupyter_text>You can see that every name also includes an initial.  Let's try to just pull out the first and last name.<jupyter_code>for x in email_dict.keys():
    old = x.split('.')
    email_dict[" ".join([i for i in old if len(i) > 2])] = email_dict[x]
    del email_dict[x]

print(email_dict)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09736632083000799, 'fraction_numerical': 0.0031923383878691143, 'mean_word_length': 5.225165562913907, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 38, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '352337', 'license': 'MIT', 'n_tokens_mistral': 1138, 'n_tokens_neox': 1100, 'n_words': 420, 'path': 'Notebooks/7_Introduction_to_Scraping.ipynb', 'repo_name': 'ljchang/psyc63', 'size': '3759'}