<jupyter_start><jupyter_text>> **DO YOU USE GITHUB?**  If True: print('Remember to make your edits in a personal copy of this notebook')  Else: print('You don't have to understand. Continue your life.')  Module 7: Web Scraping 2In module_6 your learned some powerful tricks. Tricks that will work when the data is already shipped in a neat format. However this is often not the case. In this session we shall learn the art of parsing unstructured text and a more principled and advanced method of parsing HTML.This will help you build ***custom datasets*** within just a few hours or days work, that would have taken ***months*** to curate and clean manually.Readings for `module 6+7+8`:- [Python for Data Analysis, chapter 6](https://bedford-computing.co.uk/learning/wp-content/uploads/2015/10/Python-for-Data-Analysis.pdf)- [A Practical Introduction to Web Scraping in Python](https://realpython.com/python-web-scraping-practical-introduction/)- [An introduction to web scraping with Python](https://towardsdatascience.com/an-introduction-to-web-scraping-with-python-a2601e8619e5)- [Introduction to Web Scraping using Selenium](https://medium.com/the-andela-way/introduction-to-web-scraping-using-selenium-7ec377a8cf72)Video materiale from `ISDS 2020`:- [Web Scraping 1](https://bit.ly/ISDS2021_6)- [Web Scraping 2](https://bit.ly/ISDS2021_7)- [Web Scraping 3](https://bit.ly/ISDS2021_8)Other ressources:- [Nicklas Webpage](https://nicklasjohansen.netlify.app/)- [Data Driven Organizational Analysis, Fall 2021](https://efteruddannelse.kurser.ku.dk/course/2021-2022/ASTK18379U)- [Master of Science (MSc) in Social Data Science](https://www.socialdatascience.dk/education)  Introduction to HTML[What is HTML?](https://www.w3schools.com/whatis/whatis_html.asp)  HTML has a Tree structure. Each node in the tree has:- Children, siblings, parents, descendants. - Ids and attributes  Important syntax and patterns_______________```html The p tag indicates a paragraph ```_______________```html The b tag makes the text bold, giving us a clue to its importance ```output: The b tag makes the text bold, giving us a clue to its importance ```html The em tag emphasize the text, giving us a clue to its importance```output: The em tag makes emphasize the text, giving us a clue to its importance___________```html h1h2h3Headers give similar clues```output:h1h2h3Headers give similar clues    ```html The a tag creates a hyperlink ```output: The a tag creates a hyperlink   How do we find our way around this tree?1. Regex: Extracting string patterns using .split and regular expresssions2. CSS-selectors: Specifying paths using css-selectors, xpath syntax.3. ```BeautifulSoup```: A more powerful, principled and readable way to parse data and navigate HTML<jupyter_code>import requests
from bs4 import BeautifulSoup
import re
import selenium
import time
import pandas as pd<jupyter_output><empty_output><jupyter_text>Regex- [What is regex?](https://en.wikipedia.org/wiki/Regular_expression)- The brute force way is to parse by convering your downloded matriale into a large string- Now you can create standard string operations- And apply smart regex to identify the data you are looking for e.g. links.<jupyter_code>url = 'https://www.theguardian.com/us-news/2019/aug/14/taco-eating-contest-death-fresno-california'
response = requests.get(url)
html = response.text
#html.split('\n')
#re.findall("(?P<url>https?://[^\s]+)", html)[0]<jupyter_output><empty_output><jupyter_text>CSS Selectors - [What is a CSS Selector?](https://www.w3schools.com/css/css_selectors.asp)- Another way to browse through the HTML tree- Define a unique path to an element in the HTML tree.- It is quick but has to be hardcoded and also more likely to break.- [Nicklas recommend using this free Google Chrome CSS Selector](https://chrome.google.com/webstore/detail/selectorgadget/mhjhnkcfbdhnjickkkdbjoemdmbfginb)<jupyter_code>url = 'https://www.theguardian.com/us-news/2019/aug/14/taco-eating-contest-death-fresno-california'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
soup.select('.dcr-125vfar')[0].text<jupyter_output><empty_output><jupyter_text>Parsing HTML with BeautifulSoupBeautifulSoup makes the html tree navigable. It allows you to:    * Search for elements by tag name and/or by attribute.    * Iterate through them, go up, sideways or down the tree.    * Furthermore it helps you with standard tasks such as extracting raw text from html,    which would be a very tedious task if you had to hardcode it using `.split` commands and using your own regular expressions will be unstable.<jupyter_code>url = 'https://www.theguardian.com/us-news/2019/aug/14/taco-eating-contest-death-fresno-california'
response = requests.get(url)
html = response.text
soup = BeautifulSoup(html,'lxml') # parse the raw html using BeautifoulSoup
# example: finding hyperlinks
links = soup.find_all('a') # find all a tags -connoting a hyperlink.
[link['href'] for link in links if link.has_attr('href')][0:5] # unpack the hyperlink from the a nodes.
# example: finding headline
headline = soup.find('h1') # search for the first headline: h1 tag. 
name = headline['class'][0].strip() # use the class attribute name as column name.
value = headline.text.strip() # extract text using build in method.
print(name,':',value)
# example: finding article_text
article_text = soup.findAll('div', {'class':'dcr-185kcx9'})[0]
article_text.text<jupyter_output><empty_output><jupyter_text>Say we are interested in how articles cite sources to back up their story i.e. their hyperlink behaviour within the article, and we want to see if the media has changed their behaviour.We know how to search for links. But the cool part is that we can search from anywhere in the HTML tree. This means that once we have located the article content node - as above - we can search from there. This results in hyperlinks used within the article text.<jupyter_code># example: finding citation links
citations = article_text.findAll('a')

citation_links = [] # define container to the hyperlinks
for citation in citations: # iterate through each citation node
    if citation.has_attr('data-link-name'): # check if it has the right attribute
        if citation['data-link-name'] =='in body link': # and if the value of that attribute is correct
            print(citation['href'])
            citation_links.append(citation['href']) #  add link to the container

citation_links<jupyter_output>https://www.theguardian.com/us-news/california
https://www.theguardian.com/us-news/gallery/2016/jul/04/nathans-famous-hotdog-eating-contest-in-pictures<jupyter_text>Creating a dataset from www.bold.dk Let's put together some of the stuff we have learned so far1. **Investigate:** In this example we will try to investigate the website to uderstand its structure. 2. **Mapping:** Then we will try to collect all the urls and save them into a list3. **Parsing:** At last, we will try to collect the information in each url in a simpel loop. First, we pay around with the site trying to understand its structure<jupyter_code># define our URL
url = 'https://www.bold.dk/' 

# connects to site
response = requests.get(url)

# parse data with BeautifulSoup
soup = BeautifulSoup(response.text,'lxml') # parse the HTML

#identify articles to scrape by inspecting site
articles = soup.find_all('div',{'class':'news_list_item'}) # search for the ul node

# checking if articles match website
for i in range(1):
    print(articles[i].text.strip())

# identifying how to find an url from an article
article_url = articles[0].attrs['data-vr-contentbox-url']
print(article_url)
articles[0]<jupyter_output><empty_output><jupyter_text>Second, we create a list of urls that we want to scrape<jupyter_code>url = 'https://www.bold.dk/' 
response = requests.get(url)
soup = BeautifulSoup(response.text,'lxml')
articles = soup.find_all('div',{'class':'news_list_item'})

#create an empty list
list_of_article_urls = []

# creating a loop that appends the article url to the list above
for i in range(len(articles)):
    list_of_article_urls.append(articles[i].attrs['data-vr-contentbox-url'])

#printing the list
#list_of_article_urls

#printing one example
print(list_of_article_urls[0])<jupyter_output>https://www.bold.dk/fodbold/nyheder/psv-anfoerer-vil-ikke-kalde-fcm-en-walkover/<jupyter_text>Third, we scrape each site from the url list<jupyter_code># this step usually reuqiere a new step of investigation
# to figure out what information you want to download
# in this example we want the title, the lead and time posted

# creatig empty list for the infomation we want to extract for every article
h1_list = []
lead = []
time_posted = []

for i in range(10): # 10 #len(list_of_article_urls)
    
    # this time we scrape for each news article in the url list we created before
    url = list_of_article_urls[i]
    response = requests.get(url)
    soup = BeautifulSoup(response.text,'lxml')
    
    # pedagogical way of append title to list
    temp_1 = soup.find_all('h1')
    temp_1 = temp_1[1]
    temp_1 = temp_1.text.strip()
    h1_list.append(temp_1)
    
    # how I would actually do it
    lead.append(soup.find_all('div',{'class':'lead'})[0].text.strip())
    
    # sometimes you make wierd things - that works
    temp_3 = soup.find_all('time')
    temp_3 = temp_3[0]
    temp_3 = str(temp_3)[16:32]
    time_posted.append(temp_3)
# h1 
soup.find_all('h1')
soup.find_all('h1')[1]
soup.find_all('h1')[1].text.strip()
# lead
soup.find_all('div',{'class':'lead'})[0].text.strip()
# time_posted
soup.find_all('time')[0]<jupyter_output><empty_output><jupyter_text>Lastly, we put our collected information into a dataframe<jupyter_code>import pandas as pd
df = pd.DataFrame({'title':h1_list, 'lead':lead, 'time':time_posted})
df
# saving df
df.to_csv('df_bold.dk.csv')

# loading df
pd.read_csv('df_bold.dk.csv', index_col=0)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10722211014726649, 'fraction_numerical': 0.016340528545491225, 'mean_word_length': 5.0054512416717145, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 37, 'https://': 22, 'lorem ipsum': 0, 'www.': 12, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '231247', 'license': 'MIT', 'n_tokens_mistral': 3212, 'n_tokens_neox': 3048, 'n_words': 1043, 'path': 'teaching_material/module_7/module_7.ipynb', 'repo_name': 'Todden96/isds2021', 'size': '9914'}