<jupyter_start><jupyter_text>Course on webscraping*By <NAME>, <NAME>, <NAME>*<jupyter_code># Imports:
import requests                  # for issueing HTTP requests
from bs4 import BeautifulSoup    # for parsing and navigating HTML results
import time                      # for sleeping between multiple requests
headers = {'user-agent': 'scrapingCourseBot'}<jupyter_output><empty_output><jupyter_text>Documentation:- [Requests.py](http://docs.python-requests.org)- [Beautifulsoup.py](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)- [CSS selectors](https://www.w3schools.com/cssref/css_selectors.asp)  Soup with CSS selectors: (select):<jupyter_code># Get page in soup:
r5 = requests.get('https://www.cbs.nl/en-gb', headers=headers)
print(r5.url, r5.status_code)
soup = BeautifulSoup(r5.text, 'lxml')
# Find all elements with a tag:

# find_all:
#print(soup.find_all("h2"))
#print("")
#print(soup.find_all("h3"))

#CSS:
print(soup.select("h2"))
print("")
print(soup.select("h3"))
# Find all elements belonging to a class:

# find_all:
#print(soup.find_all("a", class_="thumbnail"))

# CSS:
print(soup.select("a.thumbnail"))
# find all elements with an id:

# find_all:
#print(soup.find_all("section", id="aside-main"))

# CSS:
print(soup.select("section#aside-main"))
# The power of CSS is in traversing the tree in one statement:
# Get all headlines via a CSS selector:
headlines = soup.select("a.thumbnail h3")
for headline in headlines:
    print(headline.text)
# Get all links of thumbnails via CSS:
thumbnails = soup.select("a.thumbnail")
for thumbnail in thumbnails:
    print(thumbnail['href'])
# Get all h3s in the aside section:
aside_H3s = soup.select("section#aside-main h3")
for a in aside_H3s:
    print(a.text)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14804310833806011, 'fraction_numerical': 0.009075439591605219, 'mean_word_length': 5.211267605633803, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 3, 'lorem ipsum': 0, 'www.': 3, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '579623', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 638, 'n_tokens_neox': 621, 'n_words': 147, 'path': '20210913/3_1___CSS.ipynb', 'repo_name': 'SNStatComp/CBSAcademyBD', 'size': '1790'}