<jupyter_start><jupyter_text>Web Scraping with BeautifulSoupLet's look at how to install beautiful soup and then this will be our mission:Which presidential candidate is mentioned the most in the politics section of the NY Times? Maybe even a word cloud will be made... First install beautiful soup:`pip install bs4` or`conda install bs4` (probably)<jupyter_code>from bs4 import BeautifulSoup

import re ## For parsing

import requests ## For getting the HTML

from wordcloud import WordCloud  ## For fun

import backoff

import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Intro to BeautifulSoup BeautifulSoup takes the html from webpage and turns into an object that you can work with. All the tags and classes that a webpage have become attributes in a `Soup` object.<jupyter_code>html_doc = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc)
soup
soup.html
soup.head
soup.title
soup.p
soup.a
soup.find_all('a')
for tag in soup.find_all(True):
    print(tag.name)<jupyter_output>html
head
title
body
p
b
p
a
a
a
p<jupyter_text>Now real website! Let's look at the New york times politics section:https://www.nytimes.com/news-event/2020-electionSeems like a nice look to work from. Let's go to the browser and check what we can see?<jupyter_code>## Request the url

url = "https://www.nytimes.com/news-event/2020-election"

r= requests.get(url)
r
soup = BeautifulSoup(r.text, 'lxml')
soup
url_list = []

needed_divs = soup.find_all('div', class_ ="css-1l4spti")
for div in needed_divs:
    url_list.append(div.a.get('href'))
    
url_list<jupyter_output><empty_output><jupyter_text>Now let's go to each website and do our searches for each candidate. But wait!  A brief foray into decorators and the `backoff` module Oftentimes, when you loop through many webpages, the loop might break because you're making too many requests at once. That's what `backoff` is for. If the website gives an error, `backoff` will catch the exception and make the request again with some pause. The more times the exception is thrown the longer `backoff` will pause before making another one. `backoff` works as a decorator function. What's that? It's basically a function takes a function as an argument, but returns some "wrapper" for that function that references a function in it... What does that mean?<jupyter_code>def add():
    print('1+2=3') 

add()<jupyter_output>1+2=3<jupyter_text>But now you wanted to make sure that people knew that this was a function about addition when they called it. You can write something like this:<jupyter_code>def i_want_everyone_to_understand(func):
    
    def that_this_is_addition():
        print("just in case you didn't know, this is addition")
        func()
    return that_this_is_addition

add = i_want_everyone_to_understand(add)

add()<jupyter_output>just in case you didn't know, this is addition
1+2=3<jupyter_text>Instead of writing out the whole function, we can do:<jupyter_code>@i_want_everyone_to_understand
def add():
    print("1+2=3")
    
add()<jupyter_output>just in case you didn't know, this is addition
1+2=3<jupyter_text>So `backoff` does this samething, but it catches exceptions of your function. So let's make out requests getter a function so we can use `backoff` with it.<jupyter_code>@backoff.on_exception(backoff.expo,
                      requests.exceptions.RequestException)
def requester(url):
    return requests.get(url)<jupyter_output><empty_output><jupyter_text>Now let's make our loop that gets the content from the webpage:<jupyter_code>base_url = "https://www.nytimes.com"

articles = []

for relative_link in url_list:
    site = requester(base_url + relative_link).text
    articles.append(site)
    print(f"Accessing {base_url + relative_link}")<jupyter_output>Accessing https://www.nytimes.com/2020/02/13/us/politics/elizabeth-warren-bloomberg.html
Accessing https://www.nytimes.com/2020/02/13/us/politics/nevada-culinary-union-endorsement.html
Accessing https://www.nytimes.com/2020/02/13/climate/trump-fuel-economy-rollback.html
Accessing https://www.nytimes.com/2020/02/13/us/politics/trump-bloomberg.html
Accessing https://www.nytimes.com/2020/02/13/us/politics/hope-hicks-white-house-return.html
Accessing https://www.nytimes.com/2020/02/13/us/politics/on-politics-sanders-nevada.html
Accessing https://www.nytimes.com/2020/02/13/us/russian-propaganda-radio.html
Accessing https://www.nytimes.com/2020/02/13/us/politics/nevada-caucuses.html
Accessing https://www.nytimes.com/2020/02/13/us/politics/voting-smartphone-app.html
Accessing https://www.nytimes.com/2020/02/13/style/michael-bloomberg-memes-jerry-media.html
Accessing https://www.nytimes.com/2020/02/12/us/politics/new-hampshire-primary-turnout.html
Accessing https://www.nytimes.com/2020/02/12/u[...]<jupyter_text>Now let's sift through and see what we can find.<jupyter_code>name_list = []

for i, article in enumerate(articles):
    for name in ['Sanders', 'Trump', 'Warren', 'Buttigieg', 'Klobuchar', 'Yang']:
        name_list.extend(re.findall(name,article))
    print(f"Going through article {i}")
name_list
%matplotlib inline

name_text = ' '.join(name_list)

wordcloud = WordCloud(background_color="white", collocations=False).generate(name_text)

plt.imshow(wordcloud)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11513043478260869, 'fraction_numerical': 0.02208695652173913, 'mean_word_length': 5.092161016949152, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 52, 'https://': 15, 'lorem ipsum': 0, 'www.': 15, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '38012', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2040, 'n_tokens_neox': 1938, 'n_words': 540, 'path': 'Notes/Beautiful Soup Web Scraping.ipynb', 'repo_name': 'amichuda/Python_Boot_Camp', 'size': '5750'}