<jupyter_start><jupyter_text>Web Scraping ChallengesFor these exercises, we will be parsing information from [Quotes to Scrape](http://quotes.toscrape.com/) and [Books to Scrape](http://books.toscrape.com/). These sites are built and offered as web scraping testing grounds to learn different web scraping techniques. For this, we will focus on using [Requests](https://requests.readthedocs.io/en/master/) and [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/).If you are looking for an extra challenge, here are a few things you can try:* Only parse content with:    * BeautifulSoup [`find()`](https://www.crummy.com/software/BeautifulSoup/bs4/doc/find) and [`find_all()`](https://www.crummy.com/software/BeautifulSoup/bs4/doc/find-all) methods    * [css selectors](https://www.w3.org/TR/selectors-4/) (BeautifulSoup)    * [path selectors](https://www.w3.org/TR/2017/REC-xpath-31-20170321/) (lxml)* Use [Selenium](https://selenium.dev/selenium/docs/api/py/) to scrape a [javascript version of Quotes to Scrape](http://quotes.toscrape.com/js/)* Use [Urllib](https://docs.python.org/3/library/urllib.html) from the Python Standard Library instead of Requests Getting startedTo get started, we need to import the necessary modules. Unless you're attempting one of the challenges, we'll give you this one for free. However, you might find it useful to import some other modules later to help answer some questions.<jupyter_code>import requests
from bs4 import BeautifulSoup
from IPython.display import display, Image

books_url = 'http://books.toscrape.com'
quotes_url = 'http://quotes.toscrape.com'
# Additional Imports
from collections import Counter
from datetime import datetime
from itertools import chain
from pathlib import Path
from slugify import slugify
from tabulate import tabulate
from urllib.parse import urljoin<jupyter_output><empty_output><jupyter_text>Getting Page ContentThe first challenge is to [get](https://requests.readthedocs.io/en/master/user/quickstart/make-a-request) the page content. For now, let's just look at the Quotes site.<jupyter_code>response = requests.get(quotes_url)
response<jupyter_output><empty_output><jupyter_text>MetadataWhat kind of information was returned in the response header?<jupyter_code>print(tabulate(response.headers.items()))<jupyter_output><empty_output><jupyter_text>What kind of information did we send in the request header?<jupyter_code>print(tabulate(response.request.headers.items()))<jupyter_output><empty_output><jupyter_text>Actual DataHow many quotes are on the first page?<jupyter_code>soup = BeautifulSoup(response.content, 'html.parser')

items = soup.select('div.quote')
print(f"The first page has {len(items)} quotes")<jupyter_output><empty_output><jupyter_text>PaginationHow many pages of quotes are there to scrape?<jupyter_code>next_page = '/page/1/'
while next_page:
    response = requests.get(urljoin(quotes_url, next_page))
    soup = BeautifulSoup(response.content, 'html.parser')
    try:
        next_page = soup.select_one('li.next a')['href']
    except TypeError:
        break

print(f"There are {next_page.split('/')[2]} pages to scrape.")<jupyter_output><empty_output><jupyter_text>How many quotes are there to scrape? Store them in a list for further processing (reduces number of slow http requests)<jupyter_code>quotes = []

next_page = '/page/1/'
while next_page:
    response = requests.get(urljoin(quotes_url, next_page))
    soup = BeautifulSoup(response.content, 'html.parser')
    quotes.extend(soup.select('div.quote'))
    try:
        next_page = soup.select_one('li.next a')['href']
    except TypeError:
        break
        
print(f'There are {len(quotes)} quotes to process.')<jupyter_output><empty_output><jupyter_text>TagsHow many tags are there?<jupyter_code>tags = Counter(
    chain.from_iterable([
        quote.select('a.tag') 
        for quote in quotes
    ])
)

print(f'There are {len(tags)} tags')<jupyter_output><empty_output><jupyter_text>What are the top 20 tags? How many quotes does each tag have? What is it's url?<jupyter_code>print(tabulate([
    {
        'tag': tag.text,
        'num': count,
        'url': tag['href'],
    }
    for tag, count in tags.most_common(20)
]))<jupyter_output><empty_output><jupyter_text>AuthorsHow many authors are there?<jupyter_code>authors = Counter([
    quote.select_one('small.author').text
    for quote in quotes
])
    
print(f'There are {len(authors)} authors')<jupyter_output><empty_output><jupyter_text>What are the top 20 authors? How many quotes does each author have? What is their url?<jupyter_code>authors_dict = {name: {'count': count} for name, count in authors.most_common()}
for quote in quotes:
    authors_dict[quote.select_one('small.author').text]['url'] = quote.select_one('span a')['href']

for auth in authors_dict:
    authors_dict[auth]['name'] = auth

authors_list = list(authors_dict.values())

print(tabulate(authors_list[:20]))<jupyter_output><empty_output><jupyter_text>Who is the oldest author? Who is the youngest?<jupyter_code>for author in authors_list:
    response = requests.get(urljoin(quotes_url, author['url']))
    soup = BeautifulSoup(response.content, 'html.parser')
    author['birthday'] = datetime.strptime(
        soup.select_one('.author-born-date').text,
        '%B %d, %Y'
    )
    author['home'] = soup.select_one('.author-born-location').text[3:]
by_birthday = sorted(authors_list, key=lambda i: i['birthday'])

print(f"The oldest author is {by_birthday[0]['name']} born on {by_birthday[0]['birthday'].strftime('%B %d, %Y')}")
print(f"The youngest author is {by_birthday[-1]['name']} born on {by_birthday[-1]['birthday'].strftime('%B %d, %Y')}")<jupyter_output><empty_output><jupyter_text>Where were the most authors born (by country)? Which countries have the most?<jupyter_code>print(tabulate(
    Counter([
        auth['home'].split(', ')[-1] 
        for auth in authors_list]
    ).most_common()
))<jupyter_output><empty_output><jupyter_text>Parsing JSONHow many pages does it take to scrape the whole api? Use [params](https://requests.readthedocs.io/en/master/user/quickstart/passing-parameters-in-urls) to build the query string.The api can be found at http://quotes.toscrape.com/api/quotes<jupyter_code>quotes_url_json = 'http://quotes.toscrape.com/api/quotes'

next_page = 1
while next_page:
    response = requests.get(quotes_url_json, params={'page': next_page})
    data = response.json()
    if data['has_next']:
        next_page += 1
    else:
        break
        
print(f"There are {next_page} pages to scrape.")<jupyter_output><empty_output><jupyter_text>How many quotes are available on the api?<jupyter_code>quotes = []

next_page = 1
while next_page:
    response = requests.get(quotes_url_json, params={'page': next_page})
    data = response.json()
    quotes.extend(data['quotes'])
    if data['has_next']:
        next_page += 1
    else:
        break
        
print(f'There are {len(quotes)} quotes to process.')<jupyter_output><empty_output><jupyter_text>How many unique tags are in the api?<jupyter_code>tags = Counter(
    chain.from_iterable([
        quote['tags']
        for quote in quotes
    ])
)

print(f'There are {len(tags)} tags')<jupyter_output><empty_output><jupyter_text>What are the top 20 tags? Do they match the previous result?<jupyter_code>print(tabulate(
    tags.most_common(20)
))<jupyter_output><empty_output><jupyter_text>How many unique authors are in the api?<jupyter_code>authors = Counter([
    quote['author']['name']
    for quote in quotes
])
    
print(f'There are {len(authors)} authors')<jupyter_output><empty_output><jupyter_text>What are the top 20 authors? Do they match the previous result?<jupyter_code>print(tabulate(
    authors.most_common(20)
))<jupyter_output><empty_output><jupyter_text>Sessions and Logging InStarting with the `quotes_url`, do the following:* Navigate to the login page* Login (any credentials work)* Return to the home page and verify you are still logged in. The login link should be different<jupyter_code>with requests.Session() as s:

    # Find the login page from the home page
    response = s.get(quotes_url)
    soup = BeautifulSoup(response.content, 'html.parser')
    login_page = [
        a['href'] 
        for a in soup.select('.header-box a') 
        if 'login' in a.text.lower()
    ][0]
    
    # Parse the login page for where to submit login info and the date needed
    response = s.get(urljoin(quotes_url, login_page))
    soup = BeautifulSoup(response.content, 'html.parser')
    login_form = soup.select_one('form')
    action = login_form['action']
    data = {
        input['name']: input.get('value', 'any_value_works_for_this_site')
        for input in login_form.select('input[name]')
    }
    
    # Submit login data with csrf_token
    s.post(urljoin(quotes_url, action), data=data)
    
    # Verify the login link has changed
    response = s.get(quotes_url)
    assert login_page not in response.text.lower()
    
    # Have a look at what the link now says
    soup = BeautifulSoup(response.content, 'html.parser')
    link = [
        a['href'] 
        for a in soup.select('.header-box a')
        if len(a['href']) > 1
    ][0]
    print(link)<jupyter_output><empty_output><jupyter_text>Downloading ImagesFor this, you should look at [Books to Scrape](http://books.toscrape.com/). Download all the images of book covers from the home page. Rename the file so you can know what each file is. Use `display(Image(filename))` to display all the book covers in your notebook.<jupyter_code>response = requests.get(books_url)
soup = BeautifulSoup(response.content, 'html.parser')
images = {
    f"{slugify(img['alt'])}{Path(img['src']).suffix}" : img['src']
    for img in soup.select('article.product_pod img')
}

for image, uri in images.items():
    response = requests.get(urljoin(books_url, uri))
    with open(image, 'wb') as f:
        for chunk in response:
            f.write(chunk)
            
for filename in Path('.').glob('*jpg'):
    display(Image(filename))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12591857000993048, 'fraction_numerical': 0.005263157894736842, 'mean_word_length': 4.6546883773161145, 'pattern_counts': {'":': 0, '<': 85, '<?xml version=': 0, '>': 86, 'https://': 10, 'lorem ipsum': 0, 'www.': 5, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '363000', 'license': 'MIT', 'n_tokens_mistral': 3345, 'n_tokens_neox': 3257, 'n_words': 838, 'path': 'web-scraping-intro/Answers.ipynb', 'repo_name': 'djfurman/coding-night', 'size': '10070'}