<jupyter_start><jupyter_text>Webscraping Wikipedia  Scraping rules- You should check a site's terms and conditions before you scrape them. It's their data and they likely have some rules to govern it.- Be nice - A computer will send web requests much quicker than a user can. Make sure you space out your requests a bit so that you don't hammer the site's server.- Scrapers break - Sites change their layout all the time. If that happens, be prepared to rewrite your code.- Web pages are inconsistent - There's sometimes some manual clean up that has to happen even after you've gotten your data. Import necessary modules<jupyter_code>import requests
from bs4 import BeautifulSoup
import json
import os<jupyter_output><empty_output><jupyter_text>requests- requests executes HTTP requests, like GET- The requests object holds the results of the request. This is page content and other items like HTTP status codes and headers.- requests only gets the page content without any parsing.- Beautiful Soup does the parsing of the HTML and finding content within the HTML.<jupyter_code>url = 'https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Main+Page'
login = 'https://en.wikipedia.org/w/index.php?title=Special:UserLogin&action=submitlogin&type=login'
watchlist = 'https://en.wikipedia.org/wiki/Special:Watchlist'
url, login, watchlist<jupyter_output><empty_output><jupyter_text>Login via session Store your credentials in a encrypted/protected file (line1 = name, line2 = pwd)<jupyter_code># Create credentials file:
# CRED_KEY = '....'
# !touch ../_credentials/wiki_credentials.txt
# !echo "$CRED_KEY" > ../_credentials/wiki_credentials.txt
# !chmod 400 ../_credentials/wiki_credentials.txt
# !cat ../_credentials/wiki_credentials.txt
with open('../_credentials/wiki_credentials.txt') as f:
    contents = f.read().split('\n')
    username = contents[0]
    password = contents[1]<jupyter_output><empty_output><jupyter_text>Construct object that contains requested login dataInspect the login-form in your browser get the value of the login token<jupyter_code>def get_login_token(response):
    soup = BeautifulSoup(response.text, 'lxml')
    token = soup.find('input', {'name': "wpLoginToken"}).get('value')
    return token
payload = {
    'wpName': username,
    'wpPassword': password,
    'wploginattempt': 'Log in',
    'wpEditToken': '+\\',
    'title': 'Special:UserLogin',
    'authAction': 'login',
    'force': '',
    'wpForceHttps': '1',
    'wpFromhttp': '1',
    'wpLoginToken': 'get_login_token(session.response)'
    }<jupyter_output><empty_output><jupyter_text>Setup a session, login, and get data<jupyter_code>with requests.session() as s:
    
    response = s.get(url)
    
    # Set login token
    payload['wpLoginToken'] = get_login_token(response)
    
    # Send the login request
    response_post = s.post(login, data=payload)
    
    # Get another page and check if we’re still logged in
    response = s.get(watchlist)
    data = BeautifulSoup(response.content, 'lxml')
data;
print(data.find('div', class_='watchlistDetails').get_text())
s.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11036468330134357, 'fraction_numerical': 0.003198976327575176, 'mean_word_length': 4.79074074074074, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 22, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '386787', 'license': 'MIT', 'n_tokens_mistral': 956, 'n_tokens_neox': 927, 'n_words': 296, 'path': '_webscraping/Web_scraping_wikipedia.ipynb', 'repo_name': 'aixpact/data-science', 'size': '3126'}