<jupyter_start><jupyter_text>> **Note:** In most sessions you will be solving exercises posed in a Jupyter notebook that looks like this one. Because you are cloning a Github repository that only we can push to, you should **NEVER EDIT** any of the files you pull from Github. Instead, what you should do, is either make a new notebook and write your solutions in there, or **make a copy of this notebook and save it somewhere else** on your computer, not inside the `sds` folder that you cloned, so you can write your answers in there. If you edit the notebook you pulled from Github, those edits (possible your solutions to the exercises) may be overwritten and lost the next time you pull from Github. This is important, so don't hesitate to ask if it is unclear.  Exercise Set 9: Parsing and Information Extraction*Morning, August 17, 2018*In this Exercise Set we shall develop our webscraping skills even further by practicing navigating html trees using BeautifoulSoup and furthermore train extracting information from raw text with no html tags to help, using regular expressions.   Exercise Section 9.1: Parsing a Table from HTML using BeautifulSoup.Yesterday I showed you a neat little prepackaged function in pandas that did all the work. However today we should learn the mechanics of it. *(It is not just for educational purposes, sometimes the package will not do exactly as you want.)*We hit the Basketball stats page from yesterday again: https://www.basketball-reference.com/leagues/NBA_2018.html. > **Ex. 9.1.1:** Here we practice simply locating the table node of interest using the `find` method build into BeautifoulSoup. But first we have to fetch the HTML using the `requests` module. Parse the tree using `BeautifulSoup`. And then use the **>Inspector<** tool (* right click on the table < press inspect element *) in your browser to see how to locate the Eastern Conference table node - i.e. the *tag* name of the node, and maybe some defining *attributes*.<jupyter_code># [Answer to Ex. 9.1.1]
import requests
from bs4 import BeautifulSoup
import bs4
import pandas as pd

url ='https://www.basketball-reference.com/leagues/NBA_2018.html'
response = requests.get(url)
html = response.text
html

soup = BeautifulSoup(html, 'lxml')
soup
table = soup.find('table', {'id':'confs_standings_E'})
print(table)

# pd.read_html trækker df ud af html koden.
df = pd.read_html(response.content,attrs = {'id': 'confs_standings_E'})[0]
print(df)<jupyter_output><table class="suppress_all sortable stats_table" data-cols-to-freeze="1" id="confs_standings_E"><caption>Conference Standings Table</caption>
<colgroup><col/><col/><col/><col/><col/><col/><col/><col/></colgroup>
<thead>
<tr>
<th aria-label="Eastern Conference" class=" poptip sort_default_asc left" data-stat="team_name" scope="col">Eastern Conference</th>
<th aria-label="Wins" class=" poptip right" data-stat="wins" data-tip="Wins" scope="col">W</th>
<th aria-label="Losses" class=" poptip right" data-stat="losses" data-tip="Losses" scope="col">L</th>
<th aria-label="Win-Loss Percentage" class=" poptip right" data-stat="win_loss_pct" data-tip="Win-Loss Percentage" scope="col">W/L%</th>
<th aria-label="Games Behind" class=" poptip sort_default_asc right" data-stat="gb" data-tip="Games Behind" scope="col">GB</th>
<th aria-label="Points Per Game" class=" poptip right" data-stat="pts_per_g" data-tip="Points Per Game" scope="col">PS/G</th>
<th aria-label="Opponent Points Per Game" class=" popt[...]<jupyter_text>You have located the table should now build a function that starts at a "table node" and parses the information, and outputs a pandas DataFrame. Inspect the element either within the notebook or through the **>Inspector<** tool and start to see how a table is written in html. Which tag names can be used to locate rows? How will you iterate through columns. Were is the header located?> **Ex. 9.1.2:** First you parse the header which can be found in the canonical tag name: thead. Next you use the `find_all` method to search for the tag, and iterate through each of the elements extracting the text, using the `.text` method builtin to the the node object. Store the header values in a list container.<jupyter_code># [Answer to Ex. 9.1.2]
import pandas as pd

name = table.caption.text
print(name)

header = table.thead.find_all('th')
#print(header)
header = [col['aria-label'] for col in header]
print(header)<jupyter_output>Conference Standings Table
['Eastern Conference', 'Wins', 'Losses', 'Win-Loss Percentage', 'Games Behind', 'Points Per Game', 'Opponent Points Per Game', 'Simple Rating System']<jupyter_text>> **Ex. 9.1.3:** Next you locate the rows, using the canonical tag name: tbody. And from here you search for all rows tags. Fiugre out the tag name yourself, inspecting the tbody node in python or using the **Inspector**.<jupyter_code># [Answer to Ex. 9.1.3]
row = table.tbody.find_all('tr')
print(row[0:5])<jupyter_output>[<tr class="full_table"><th class="left " data-stat="team_name" scope="row"><a href="/teams/TOR/2018.html">Toronto Raptors</a>* <span class="seed">(1) </span></th><td class="right " data-stat="wins">59</td><td class="right " data-stat="losses">23</td><td class="right " data-stat="win_loss_pct">.720</td><td class="right " data-stat="gb">—</td><td class="right " data-stat="pts_per_g">111.7</td><td class="right " data-stat="opp_pts_per_g">103.9</td><td class="right " data-stat="srs">7.29</td></tr>, <tr class="full_table"><th class="left " data-stat="team_name" scope="row"><a href="/teams/BOS/2018.html">Boston Celtics</a>* <span class="seed">(2) </span></th><td class="right " data-stat="wins">55</td><td class="right " data-stat="losses">27</td><td class="right " data-stat="win_loss_pct">.671</td><td class="right " data-stat="gb">4.0</td><td class="right " data-stat="pts_per_g">104.0</td><td class="right " data-stat="opp_pts_per_g">100.4</td><td class="right " data-stat="srs">3.23</td></tr>[...]<jupyter_text>> **Ex. 9.1.4:** Next run through all the rows and extract each value, similar to how you extracted the header. However here is a slight variation: Since each value node can have a different tag depending on whether it is a digit or a string, you should use the `.children` method instead of the `.find_all` - (or write compile a regex that matches both the td tag and the th tag.) >Once the value nodes of each row has been located using the `.children` method you should extract the value. Store the extracted rows as a list of lists: ```[[val1,val2,...valk],...]```<jupyter_code># [Answer to Ex. 9.1.4]
children = table.children('a')<jupyter_output><empty_output><jupyter_text>> **Ex. 9.1.5:** Convert the data you have collected into a pandas dataframe. _Bonus:_ convert the code you've written above into a function which scrapes the page for you and returns a dataframe.<jupyter_code>#[Answer 9.1.5]<jupyter_output><empty_output><jupyter_text>> **Ex. 9.1.6:** Now locate all tables from the page, using the `.find_all` method searching for the table tag name. Iterate through the table nodes and apply the function created for parsing html tables. Store each table in a dictionary using the table name as key. The name is found by accessing the id attribute of each table node, using dictionary-style syntax - i.e. `table_node['id']`.<jupyter_code># [Answer to Ex. 9.1.6]<jupyter_output><empty_output><jupyter_text>> ** 9.1.extra.:** Compare your results to the pandas implementation.   Exercise Section 9.2: Practicing Regular Expressions.This exercise is about developing your experience with designing your own regular expressions.Remember you can always consult the regular expression reference page [here](https://www.regular-expressions.info/refquick.html), if you need to remember or understand a specific symbol. You should practice using *"define-inspect-refine-method"* described in the lectures to systematically ***explore*** and ***refine*** your expressions, and save all the patterns tried. You can download the small module that I created to handle this in the following way: ``` pythonimport requestsurl = 'https://raw.githubusercontent.com/snorreralund/explore_regex/master/explore_regex.py'response = requests.get(url)with open('explore_regex.py','w') as f:    f.write(response.text)import explore_regex as e_re```Remember to start ***broad*** to gain many examples, and iteratively narrow and refine.We will use a sample of the trustpilot dataset that you practiced collecting yesterday.You can load it directly into python from the following link: https://raw.githubusercontent.com/snorreralund/scraping_seminar/master/english_review_sample.csv > **Ex. 9.2.0:** Load the data used in the exercise using the `pd.read_csv` function. (Hint: path to file can be both a url or systempath). >Define a variable `sample_string = '\n'.join(df.sample(2000).reviewBody)` as sample of all the reviews that you will practice on.  (Run it once in a while to get a new sample for potential differences).Imagine we were a company wanting to find the reviews where customers are concerned with the price of a service. They decide to write a regular expression to match all reviews where a currencies and an amount is mentioned.<jupyter_code># [Answer to Ex. 9.2.0]<jupyter_output><empty_output><jupyter_text>> **Ex. 9.2.1:** > Write an expression that matches both the dollar-sign (\$) and dollar written literally, and the amount before or after a dollar-sign. Remember that the "$"-sign is a special character in regular expressions. Explore and refine using the explore_pattern function in the package I created called explore_regex. ```pythonimport explore_regex as e_reexplore_regex = e_re.Explore_Regex(sample_string)  Initaizlie the Explore regex Class.explore_regex.explore_pattern(pattern)  Use the .explore_pattern method.```Start with exploring the context around digits ("\d") in the data.<jupyter_code># [Answer to exercise 9.2.1]<jupyter_output><empty_output><jupyter_text>> **Ex.9.2.3** Use the .report() method. e_re.report(), and print the all patterns in the development process using the .pattern method - i.e. e_re.patterns<jupyter_code># [Answer 9.2.3]<jupyter_output><empty_output><jupyter_text>>** Ex. 9.2.4** Finally write a function that takes in a string and outputs if there is a match. Use the .match function to see if there is a match (hint if does not return a NoneType object - `re.match(pattern,string)!=None`).> Define a column 'mention_currency' in the dataframe, by applying the above function to the text column of the dataframe. *** You should have approximately 310 reviews that matches. - but less is also alright***> **Ex. 9.2.5** Explore the relation between reviews mentioning prices and the average rating. > **Ex. 9.2.extra** Define a function that outputs the amount mentioned in the review (if more than one the largest), define a new column by applying it to the data, and explore whether reviews mentioning higher prices are worse than others by plotting the amount versus the rating.<jupyter_code>#[Answer to 9.2.4-5]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11311030741410488, 'fraction_numerical': 0.013562386980108499, 'mean_word_length': 5.316961736150771, 'pattern_counts': {'":': 0, '<': 113, '<?xml version=': 0, '>': 129, 'https://': 5, 'lorem ipsum': 0, 'www.': 3, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '637623', 'license': 'MIT', 'n_tokens_mistral': 3350, 'n_tokens_neox': 3191, 'n_words': 1350, 'path': 'Test_karl/material/Exercises/karl_exercise_9.ipynb', 'repo_name': 'karlbindslev/sds_group29', 'size': '11060'}