<jupyter_start><jupyter_text>Presentation, webscraping-- * By <NAME> and <NAME> *<jupyter_code># Retrieving the English home page of Statistics Netherlands:
import requests
from bs4 import BeautifulSoup
import time

r1 = requests.get('https://www.cbs.nl/en-gb')

print(r1.status_code, r1.headers['content-type'], r1.encoding)
# Retrieving document:
print(r1.text)
# Issue an request with parameters:

pars = {'products': 2, 'years': 2}
r2 = requests.get('http://testing-ground.scraping.pro/table?', params=pars)
        
print(r2.url)
# Retrieving document:
print(r2.text)
# Retrieving English home page of Statistics Netherlands:
r3 = requests.get('https://www.cbs.nl/en-gb')
print(r3.text)

soup3 = BeautifulSoup(r3.text, 'html.parser')
print(soup3.title.text)
# Get the URLS to all news articles of CBS:
articles = soup3.find_all("div", class_='thumbnail')
for article in articles:
    link = article.find("a")['href']
    print(link)
headlines = soup3.select("div.thumbnail h3")

print(headlines)
# Get all texts of news articles of CBS:
articles = soup3.find_all("div", class_='thumbnail')
links3 = []
for article in articles:
    links3.append(article.find("a")['href'])

for link in links3:
    r = requests.get('https://www.cbs.nl'+link)
    soup = BeautifulSoup(r.text, 'html.parser')
    leadtext = soup.find('section', class_='leadtext').text
    print(leadtext)
    time.sleep(1) # in robots.txt CBS advises a delay of 1 second<jupyter_output>According to Statistics Netherlands (CBS), producer confidence among Dutch manufacturers has hardly changed. Confidence stood at 5.7 in September, down from 5.9 in August. In September, manufacturers were less positive about their future output than in the previous month.
            

                One-fifth of all exports by Dutch manufacturers consist of products that are not self-manufactured, but supplied along with their own products, a phenomenon referred to as carry-along trade. This is reported by Statistics Netherlands (CBS) in the latest edition of the Internationalisation Monitor, based on new research into this phenomenon.
            

                Almost half of the total Dutch adult population are moderately or severely overweight. In the period 2015-2017, two in five severely overweight (obese) people indicated that they were dissatisfied with their body weight, while one in five said that they were satisfied. This is reported by Statistics Nether[...]
{'fraction_non_alphanumeric': 0.10151453131395825, 'fraction_numerical': 0.01514531313958248, 'mean_word_length': 4.324618736383442, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 3, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '85122', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 727, 'n_tokens_neox': 694, 'n_words': 261, 'path': '.ipynb_checkpoints/Examples-checkpoint.ipynb', 'repo_name': 'SNStatComp/CBSAcademyBD', 'size': '2460'}