<jupyter_start><jupyter_text>[Link to this jupyter notebook](./Files/0204-Web_Scraping.ipynb)  In-Class Assignment: Web ScrapingToday we will be exploring some of the extensive datasets available at the National Oceanic and Atmospheric Administration (NOAA). Work as a team to try to get as many of todays activities done.  We will meet as a class again around 3:30pm to discuss what you learned. Image From: https://www.ncdc.noaa.gov/data-access/land-based-station-data/land-based-datasets/us-climate-reference-network-uscrn  Agenda for today's class (80 minutes)1. [(10 minutes) NOAA Example](NOAA_Example)1. [(5 minutes) Installing Beautiful Soup](Installing_Beautiful_Soup)2. [(20 minutes) Presidential data example](Presidential_data_example)4. [(20 minutes) Dynamic Website example](DynamicWebsites)5. [(25 minutes) wrap-up Discussion](Wrapup) ---- 1. NOAA Example and Coding Standards.We are going to start today's activity by doing a code review of a **_web spider_** program. &9989; **DO THIS:**  Download the [noaa_scrapper.py](./Files/noaa_scrapper.py) and [this jupyter notebook](./Files/0204-Web_Scraping.ipynb') annd put them in the same directory. Run the file via the following command:<jupyter_code>%matplotlib inline 
import matplotlib.pyplot as plt

from noaa_scraper import get_noaa_temperatures<jupyter_output>running as noaa_scraper<jupyter_text>&9989; **DO THIS:** Run the ```get_noaa_temperatures``` function as follows:<jupyter_code>air_temperatures = get_noaa_temperatures('http://www1.ncdc.noaa.gov/pub/data/uscrn/products/subhourly01/', 'Gaylord', 100)
plt.plot(air_temperatures)
# plt.axis([0,1000,-20,80])<jupyter_output><empty_output><jupyter_text>&9989; **DO THIS:** With your group, do a code review of the contents of the **noaa_scraper.py** file and figure out what it does. What are the main part of this module and what do they do? Be prepared to discuss this with the class.  Put your notes here ---- 2. Installing Beautiful SoupFor this class we will be trying out [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/), a Python web parsing module. &9989; **DO THIS:** Install the ```beautifulsoup4``` library on your computer (the following will work on jupyterhub but should work anywhere).  When you are done, help your neighbor and raise your hand if you need help.<jupyter_code>#!mkdir packages
#!pip install -t ./packages/ beautifulsoup4
import sys
sys.path.append('./packages/')<jupyter_output><empty_output><jupyter_text>---- 3. Presidential data exampleThis second example is a **_web scraper_** program. Found this idea by reading the following blog post: https://blog.exploratory.io/scraping-us-presidents-list-from-web-and-transforming-it-to-be-useful-fff534470bb6&9989; **DO THIS:** Click on the following link and review the page source with your teams.  Discuss which tags you need to look for to try and isolate the table data only.  Ideally we want to create a ```pandas table``` of this data:https://www.loc.gov/rr/print/list/057_chron.html Put notes on what you find here.  Download and view htmlThe following code should download the above website and parse read it into a ```beautifulsoup``` object:<jupyter_code>#The following library downloads the data and stores it in a page variable
import requests
page = requests.get("https://www.loc.gov/rr/print/list/057_chron.html")
#Import and run beautifule should html.parser
from bs4 import BeautifulSoup
soup = BeautifulSoup(page.content, 'html.parser')<jupyter_output><empty_output><jupyter_text>&9989; **DO THIS:** explore the ```soup``` variable using python functions such as; ```type```, ```dir``` and ```help```.<jupyter_code>#Put your answer to the above here
##ANSWER##
type(soup)
##ANSWER##
##ANSWER##
dir(soup)
##ANSWER##
##ANSWER##
help(soup)
##ANSWER##
#Print out the raw html using "pretty print" 
print(soup.prettify())<jupyter_output><empty_output><jupyter_text>Find the TablesNext, the following code finds all of the ```table``` sections in the website:<jupyter_code>tables = soup.find_all('table')
type(tables)
len(tables)<jupyter_output><empty_output><jupyter_text>According to the above the results show that there are 9 ```table``` objects in the document.  We are just looking for the one that has our data in it.  &9989; **DO THIS:** Find the table from the nine tables that has only the data we want. Make a variable ```table``` that only includes the information we want. Hint, it is not the first table which we can see by using the following code.<jupyter_code>table = tables[0]
print(table.prettify())
##ANSWER##
table = tables[3]
print(table.prettify())
##ANSWER##<jupyter_output><empty_output><jupyter_text>Parse out all the rowsThe rows of a table are determined by the ```tr``` (table row) tag and the columns are determined by the ```td```. The following code can find all of the rows in the table:<jupyter_code>rows = table.find_all('tr')
rows<jupyter_output><empty_output><jupyter_text>Get the column labelsThe first row is the column header row as can be seen by running the following code:<jupyter_code>rows[0]
labels = []
for c in rows[0].find_all('th'):
    labels.append(c.get_text())
labels<jupyter_output><empty_output><jupyter_text>Parse Rows&9989; **DO THIS:** The next step is to loop though the remaining rows and save the data as a list of lists<jupyter_code>#put your code here
##ANSWER##

data = []    
for row in rows[1:]:
    myrow = []
    for c in row.find_all('td'):
        myrow.append(c.get_text())
    data.append(myrow)

##ANSWER##<jupyter_output><empty_output><jupyter_text>Convert list of list to Pandas DataframeAssuming the above works, we can convert the list of lists and labels to a Pandas Dataframe<jupyter_code>import pandas as pd  
    
# Create the pandas DataFrame  
df = pd.DataFrame(data, columns=labels)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11109206240356592, 'fraction_numerical': 0.01714383679067375, 'mean_word_length': 5.56981981981982, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 5, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '662624', 'license': 'MIT', 'n_tokens_mistral': 1915, 'n_tokens_neox': 1832, 'n_words': 653, 'path': 'Files/0204-Web_Scraping.ipynb', 'repo_name': 'bowerm37/cmse495-SS22', 'size': '5833'}