<jupyter_start><jupyter_text>Web scraping**Date: 28 March 2017**@author: <NAME>  Preliminaries Import usual packages.<jupyter_code>import pandas as pd             # data package
import matplotlib.pyplot as plt # graphics 
import datetime as dt           # date tools, used to note current date  

%matplotlib inline<jupyter_output><empty_output><jupyter_text>We have seen how to input data from `csv` and `xls` files -- either online or from our computer and through APIs. Sometimes the data is only available as specific part of a website.We want to access the source code of the website and systematically extract the relevant information. Again, use Google fu to find useful links. Here are a couple:* [link 1](https://www.dataquest.io/blog/web-scraping-tutorial-python/)* [link 2](http://blog.danwin.com/examples-of-web-scraping-in-python-3-x-for-data-journalists/)* [link 3](https://www.analyticsvidhya.com/blog/2015/10/beginner-guide-web-scraping-beautiful-soup-python/)  Structure of web pages (very simplistic)`Hypertext Markup Language` (HTML) specifies the structure and main content of the site -- tells the browser how to layout content. Think of `Markdown`.It is structured using tags.```html            (Meta) Information about the page.                            This is a paragraph.                            This is a table            ``` `Tag`s determine the content and layout depending on their relation to other tags. Useful terminology:* `child` -- a child is a tag inside another tag. The `p` tag above is a child of the `body` tag.* `parent` --  a parent is the tag another tag is inside. The `body` tag above is a parent of the `p` tag.* `sibling` -- a sibling is a tag that is nested inside the same parent as another tag. The `head` and `body` tags above are siblings.There are many different tags -- take a look at a [reference list](https://developer.mozilla.org/en-US/docs/Web/HTML/Element). You won't and shouldn't remember all of them but it's useful to have a rough idea about them.And take a look at a real example -- open page, then right click:  "View Page Source" In the real example you will see that there is more information after the tag, most commanly a `class` and an `id`. Something similar to the following:```html            (Meta) Information about the page.                            This is a paragraph.            Learn Data Science Online                            This is a table            ```The `class` and `id` information will help us in locating the information we are looking for in a systematic way. (Originally, classes and ids are used by CSS to determine which HTML elements to apply certain styles to)Useful way to explore the `html` and the corresponding website is right clicking on the web page and then clicking on `Inspect element` -- interpretation of the html by the browser Suppose we want to check prices for renting a room in Manhattan in Craigslist. Let's check for example the `rooms & shares` section for the [East Village](https://newyork.craigslist.org/search/roo?query=east+village&availabilityMode=0).  Accessing web pages We have to download the content of the webpage -- i.e. get the contents structured by the HTML. This we can do with the `requests` library, which is a human readable HTTP (HyperText Transfer Protocol) library for python. You cna find the Quickstart Documentation [here](http://docs.python-requests.org/en/master/user/quickstart/).<jupyter_code>import requests                              # you might have to install this
url = 'https://newyork.craigslist.org/search/roo?query=east+village&availabilityMode=0'

cl = requests.get(url)
cl<jupyter_output><empty_output><jupyter_text>After running our request, we get a Response object. This object has a status_code property, which indicates if the page was downloaded successfully. > A status_code of 200 means that the page downloaded successfully.    * status code starting with a 2 generally indicates success    * a code starting with a 4 or a 5 indicates an error.<jupyter_code>cl.status_code<jupyter_output><empty_output><jupyter_text>You might want to query for different things and download information for all of them. * You can pass this as extra information (defined as a dictionary with `keys` and `values`). * Best way to learn about the available keys is by "changing" things on the site and see how the url changes<jupyter_code>url = 'https://newyork.craigslist.org/search/roo'
keys = {'query' : 'east village', 'availabilityMode' : '0'}
cl_extra = requests.get(url, params=keys)
# see if the URL was specified successfully
cl_extra.url<jupyter_output><empty_output><jupyter_text>Check tab completion<jupyter_code>cl.url<jupyter_output><empty_output><jupyter_text>To print out the content of the html file, use the `content` or `text` properties This is going to be ugly and unreadable<jupyter_code>cl.text[:300]
cl.content[:500] # this works also for information which is not purely text<jupyter_output><empty_output><jupyter_text>Extracting information from a web page Now that we have the content of the web page we want to extraxt certain information. `BeautifulSoup` is a Python package which helps us in doing that. See the [documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) for more information.We first have to import the library, and create an instance of the BeautifulSoup class to parse our document:<jupyter_code>from bs4 import BeautifulSoup
BeautifulSoup?
cl_soup = BeautifulSoup(cl.content, 'html.parser')<jupyter_output><empty_output><jupyter_text>Print this out in a prettier way.<jupyter_code>#print(cl_soup.prettify())
print('Type:', type(cl_soup))
# we can access a tag 
print('Title: ', cl_soup.title)
# or only the text content
print('Title: ', cl_soup.title.text)          # or
print('Title: ', cl_soup.title.get_text())<jupyter_output>Title:  new york rooms for rent & shares available "east village" - craigslist
Title:  new york rooms for rent & shares available "east village" - craigslist<jupyter_text>We can find all tags of certain type with the `find_all` method. This **returns a list**.<jupyter_code>cl_soup.find_all?<jupyter_output><empty_output><jupyter_text>To get the first paragraph in the html write<jupyter_code>cl_soup.find_all('p')[0]<jupyter_output><empty_output><jupyter_text>This is a lot of information and we want to extract some part of it. Use the `text` or `get_text()` method to get the text content.<jupyter_code>cl_soup.find_all('p')[0].get_text()<jupyter_output><empty_output><jupyter_text>This is still messy. We will need a smarter search. As all the tags are nested, we can move through the structure one level at a time. We can first select all the elements at the top level of the page using the `children` property of soup. For example here are the children of the first paragraph tag. **Note:** `children` returns a list iterator, so we need to call the list function on it.<jupyter_code>list(cl_soup.find_all('p')[0].children)<jupyter_output><empty_output><jupyter_text>Look for tags based on their class. This is extremely useful for efficiently locating information.<jupyter_code>cl_soup.find_all('span', class_='result-price')[0].get_text()
cl_soup.find_all('span', class_='result-price')[:10]
prices = cl_soup.find_all('span', class_='result-price')
price_data = [price.get_text() for price in prices]
price_data[:10]
len(price_data)<jupyter_output><empty_output><jupyter_text>We are getting more cells than we want -- there were only 120 listings on the page. Check the ads with "Inspect Element". There are duplicates. We need a different tag level (``)<jupyter_code>cl_soup.find_all('li', class_='result-row')[0]
ads = cl_soup.find_all('li', class_='result-row')
# we can access values of the keys by using a dictionary like syntax
ads[5].find('a', class_='result-title hdrlnk')
ads[5].find('a', class_='result-title hdrlnk')['href']
data = [[ad.find('a', class_='result-title hdrlnk').get_text(), 
         ad.find('a', class_='result-title hdrlnk')['data-id'], 
         ad.find('span', class_='result-price').get_text()] for ad in ads ]<jupyter_output><empty_output><jupyter_text>What's going wrong? Some ads don't have a price listed, so we can't retrieve it.<jupyter_code># if it exists then the type is
type(ads[0].find('span', class_='result-price'))<jupyter_output><empty_output><jupyter_text>If it does not find the price, it returns a NoneType. We might exploit this fact to select only the valid links.<jupyter_code>import bs4

data = [[ad.find('a', class_='result-title hdrlnk').get_text(), 
         ad.find('a', class_='result-title hdrlnk')['data-id'], 
         ad.find('span', class_='result-price').get_text()] for ad in ads 
            if type(ad.find('span', class_='result-price'))==bs4.element.Tag]
data[:10]
df = pd.DataFrame(data)
df.head(10)
df.shape<jupyter_output><empty_output><jupyter_text>We only have 118 listing because 2 listings did not have a price.<jupyter_code>df.columns = ['Title', 'ID', 'Price']
df.head()<jupyter_output><empty_output><jupyter_text>We could do text anaylsis and see what words are common in ads which has a relatively higher price. This approach is not really efficient because it only gets the first page of the search results. We see on the top of the CL page the total number of listings. In the `Inspection` mode we can pick an element from the page and check how it is defined in the `html` -- this is useful to get tags and classes efficiently.For example, the total number of ads is a `span` tag with a 'totalcount' `class`.<jupyter_code>cl_soup.find('span', class_='totalcount')<jupyter_output><empty_output><jupyter_text>We can see if we start clicking on the 2nd nd 3rd pages of the results that there is a structure in how they are definedFirst page:https://newyork.craigslist.org/search/roo?query=east+village&availabilityMode=0Second page:https://newyork.craigslist.org/search/roo?s=120&availabilityMode=0&query=east%20villageThird page:https://newyork.craigslist.org/search/roo?s=240&availabilityMode=0&query=east%20villageThe number after `roo?s=` in the domain specifies where the listings are starting from (not inclusive). In fact, if we modify it ourselves we can fine-tune the page starting from the corresponding listing and then showing 120 listings. Try it!We can also define the first page by puttig `s=0&` after `roo?` like this:https://newyork.craigslist.org/search/roo?s=0&availabilityMode=0&query=east%20village<jupyter_code># First we get the total number of listings in real time
url = 'https://newyork.craigslist.org/search/roo?query=east+village&availabilityMode=0'
cl = requests.get(url)

cl_soup = BeautifulSoup(cl.content, 'html.parser')
total_count = int(cl_soup.find('span', class_='totalcount').get_text())
print(total_count)<jupyter_output>488<jupyter_text>We have the total number of listings with the given search specification. Breaking down the steps:1) Specify the url of each page we want to scrape2) For each page scrape the data -- we will reuse the code what we already have for one page3) Save the data into one dataframe -- we can use the `append` method for DataFrames or the `extend` method for lists<jupyter_code># 1) Specify the url
for page in range(0, total_count, 120):
    print('https://newyork.craigslist.org/search/roo?s={}&availabilityMode=0&query=east%20village'.format(page))

# Next we write a loop to scrape all pages

df = pd.DataFrame({'Title' : [], 'ID' : [], 'Price' : []})

for page in range(0, total_count, 120):
    url = 'https://newyork.craigslist.org/search/roo?s={}&availabilityMode=0&query=east%20village'.format(page)
    
    cl = requests.get(url)
    cl_soup = BeautifulSoup(cl.content, 'html.parser')
    
    ads = cl_soup.find_all('li', class_='result-row')
    data = pd.DataFrame([[ad.find('a', class_='result-title hdrlnk').get_text(), 
                          ad.find('a', class_='result-title hdrlnk')['data-id'], 
                          ad.find('span', class_='result-price').get_text()] for ad in ads 
                                 if type(ad.find('span', class_='result-price'))==bs4.element.Tag], 
                        columns=['Title', 'ID', 'Price'])
    
    df = df.append(data, ignore_index=True)
df.head()
# Do the same using the `extend` method

data = []
for page in range(0, total_count, 120):
    url = 'https://newyork.craigslist.org/search/roo?s={}&availabilityMode=0&query=east%20village'.format(page)
    cl = requests.get(url)
    cl_soup = BeautifulSoup(cl.content, 'html.parser')
    ads = cl_soup.find_all('li', class_='result-row')
    data_page = [[ad.find('a', class_='result-title hdrlnk').get_text(), 
         ad.find('a', class_='result-title hdrlnk')['data-id'], 
         ad.find('span', class_='result-price').get_text()] for ad in ads 
        if type(ad.find('span', class_='result-price'))==bs4.element.Tag]
    data.extend(data_page)
    
df = pd.DataFrame(data, columns=['Title', 'ID', 'Price'])
df.head()
df.shape
df.tail()<jupyter_output><empty_output><jupyter_text>We have scraped all the listings from CL in section "Rooms and Shares" for the East Village.  ExerciseSuppose you have a couple of destinations in mind and you want to check the weather for each of them for this Friday. You want to get it from the [National Weather Service](http://www.weather.gov/).These are the places I want to check (suppose there are many more and you want to automate it):```pythonlocations = ['Bozeman, Montana', 'White Sands National Monument', 'Stanford University, California']```It seems that the NWS is using latitude and longitude coordinates in its search.i.e. for White Sandshttp://forecast.weather.gov/MapClick.php?lat=32.38092788700044&lon=-106.4794398029997Would be cool to pass these on as arguments.After some Google fu (i.e. "latitude and longitude of location python") find a post by [<NAME>](https://chrisalbon.com/python/geocoding_and_reverse_geocoding.html) which describes exactly what we want.> "Geocoding (converting a phyiscal address or location into latitude/longitude) and reverse geocoding (converting a lat/long to a phyiscal address or location)[...] Python offers a number of packages to make the task incredibly easy [...] use pygeocoder, a wrapper for Google's geo-API, to both geocode and reverse geocode.Install `pygeocoder` through `pip install pygeocoder` (from `conda` only the OSX version is available).<jupyter_code>from pygeocoder import Geocoder
# check for one of the locations how it's working
# some addresses might not be valid -- it goes through Google's API

loc = Geocoder.geocode('Bozeman, Montana')
loc.coordinates
Geocoder.geocode('Stanford, California').coordinates<jupyter_output><empty_output><jupyter_text>We can check whether it's working fine at http://www.latlong.net/<jupyter_code>locations = ['Bozeman, Montana', 'White Sands National Monument', 'Stanford University, California']

coordinates = [Geocoder.geocode(location).coordinates for location in locations]
coordinates
for location, coordinate in zip(locations, coordinates):
    print('The coordinates of {} are:'.format(location), coordinate)<jupyter_output>The coordinates of Bozeman, Montana are: (45.6769979, -111.0429339)
The coordinates of White Sands National Monument are: (32.7872403, -106.3256816)
The coordinates of Stanford University, California are: (37.4274745, -122.169719)<jupyter_text>Define a dictionary for the parameters we want to pass to the GET request for NWS server.<jupyter_code>keys = {}
for location, coordinate in zip(locations, coordinates):
    keys[location] = {'lat' : coordinate[0], 'lon' : coordinate[1]}
keys<jupyter_output><empty_output><jupyter_text>Recall the format of the url associated with a particular locationhttp://forecast.weather.gov/MapClick.php?lat=32.38092788700044&lon=-106.4794398029997<jupyter_code>keys[locations[0]]
url = 'http://forecast.weather.gov/MapClick.php'    
nws = requests.get(url, params=keys[locations[0]])
nws.status_code
nws.url
nws.content[:300]
nws_soup = BeautifulSoup(nws.content, 'html.parser')
seven = nws_soup.find('div', id='seven-day-forecast-container')
seven.find(text='Friday')
seven.find(text='Friday').parent
seven.find(text='Friday').parent.parent
seven.find(text='Friday').parent.parent.find('p', class_='temp temp-high').get_text()
data = []

for location in locations:
    
    nws = requests.get(url, params=keys[location])
    nws_soup = BeautifulSoup(nws.content, 'html.parser')
    
    seven = nws_soup.find('div', id='seven-day-forecast-container')
    temp = seven.find(text='Friday').parent.parent.find('p', class_='temp temp-high').get_text()
    data.append([location, temp])
df_weather = pd.DataFrame(data, columns=['Location', 'Friday weather'])
df_weather
df_weather['high_temp'] = df_weather['Friday weather'].str.rsplit().str.get(1).astype(float)
df_weather['high_temp'].std()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11146006162597771, 'fraction_numerical': 0.01422137947380896, 'mean_word_length': 4.823671497584541, 'pattern_counts': {'":': 0, '<': 96, '<?xml version=': 0, '>': 98, 'https://': 16, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '400577', 'license': 'MIT', 'n_tokens_mistral': 5372, 'n_tokens_neox': 5091, 'n_words': 1756, 'path': 'Code/notebooks/bootcamp_adv_scraping.ipynb', 'repo_name': 'kunal-mulki/Materials', 'size': '16887'}