<jupyter_start><jupyter_text>Section 9 - Web Scraping and Web CrawlingData from websites are not always neatly organized. We call this **unstructured data**. Most webpages are rendered in **HTML**, a programming language designed for layout and formatting. But in order to understand how to work with unstructured data from websites we need to know a little bit of HTML. CodeAcademy has a great free [HTML tutorial](https://www.codecademy.com/learn/learn-html), which covers even more than what we need for this section. We are also going to use a little bit of **regular expressions** (or RegEx), which is a special string describing a search pattern. And here is a great [RegEx tutorial](https://regexone.com/).Now that we know the basics of HTML, you should also learn how to **open the source code panel** of your web browser. If you are using Google Chrome, just press F12. If you are using Safari on a Mac, you need to **enable the developer menu**:1. Click on *Safari Menu* > *Preferences* > *Advanced*1. Check the *"show Develop menu in the menu bar"*1. Now there a is a drop down menu called *Develop*, with the *view source* option.In this notebook we are going to use some new libraries:* **`requests`**: Allows to grab the source code and other characteristics of a webpage.* **`BeaultifulSoup4`**: Interprets text as sorce code and lets you grab the parts you want.* **`re`**: The RegEx module that is already built in python.<jupyter_code>import requests
from bs4 import BeautifulSoup as bs
import re
import pandas as pd
from tqdm import tqdm<jupyter_output><empty_output><jupyter_text>For now, our target is going to the **G1 website**.https://g1.globo.com/We can use the requests library to `get` the contents of the page.<jupyter_code>response = requests.get('http://g1.globo.com')
type(response)<jupyter_output><empty_output><jupyter_text>notice that the `get` method returns a specific object of the `requests` library. Let's take a look at the atributes of the object.<jupyter_code>dir(response)<jupyter_output><empty_output><jupyter_text>Try a few of these attributes on the next cell. The most important for us is going to be `.text` method.<jupyter_code>response.text<jupyter_output><empty_output><jupyter_text>We can use RegEx to search content in this page. For example, let's **grab all the URLs from G1 website**.<jupyter_code>regex_html_url = '''href=["'](.[^"']+)["']'''  # RegEx for URLs
re.findall(regex_html_url, response.text)<jupyter_output><empty_output><jupyter_text>Notice that you can now write a program to visit all the avaible URLs from G1 home page and turn your webscrapper into a webcrawler.Now, let's say we want to **grab the text from all the headlines**.<jupyter_code>response = requests.get('http://g1.globo.com')
headlines = bs(response.content, 'html.parser').find_all('a', class_='feed-post-link gui-color-primary gui-color-hover')

print(len(headlines))

headlines<jupyter_output><empty_output><jupyter_text>To grab the contents inside a HTML tag, use the `.get_text` method<jupyter_code>print(type(headlines[0]), '\n')

print(headlines[0])

headlines[0].get_text()
for i in range(len(headlines)):
    print(headlines[i].get_text(), '\n')<jupyter_output><empty_output><jupyter_text>--- Scraping Emails[Students from the masters of economics in PUC-Rio](http://www.econ.puc-rio.br/pessoas/alunos-mestrado)<jupyter_code>response = requests.get('http://www.econ.puc-rio.br/pessoas/alunos-mestrado')

regExpEmail = '[\w\.-]+@[\w\.-]+' # RegEx for email
re.findall(regExpEmail, response.text)<jupyter_output><empty_output><jupyter_text>If you click on the name of a student, you can see that the URL ends with something that looks like an ID number. The link for the first one is http://www.econ.puc-rio.br/pessoas/perfil/1If you keep adding 1 to this ID number, you will go through everybody that is registered in the department of economics from PUC-Rio.Not all the ID numbers are valid. For example, number 17 returns an errorhttp://www.econ.puc-rio.br/pessoas/perfil/17<jupyter_code>requests.get('http://www.econ.puc-rio.br/pessoas/perfil/17').text<jupyter_output><empty_output><jupyter_text>We are going to use this in our favor. Whenever the string "fatal error" is in the response we are going to skip that ID number.<jupyter_code>re.findall('Fatal error', requests.get('http://www.econ.puc-rio.br/pessoas/perfil/17').text)<jupyter_output><empty_output><jupyter_text>In case the page exists, we use `beatifulsoup` to grab the HTML tag of the person's name and a regular expression search to get the email. We are searching for different information using different methods for a reason. The person's name is always under the `` HTML tag, while several other information are under the same tag ``.<jupyter_code>response = requests.get('http://www.econ.puc-rio.br/pessoas/perfil/18')

soup = bs(response.content.decode('utf-8','ignore'), 'html.parser')
all_a = soup.find_all('a')
for p in all_a:
    print(p.get_text())<jupyter_output><empty_output><jupyter_text>Here is a single example<jupyter_code>print(soup.find_all('h5')[0].get_text())

print(re.findall(regExpEmail , response.text)[0])<jupyter_output><empty_output><jupyter_text>Now, all we have to do is iterate on the ID of the URL and save the name and email, but we have to handle two types of errors:* We skip the iteration if the page is not avilable. To handle this we check if the string "fatal error" is on the page.* If there is no email on the page, we have to save a blank. To handle this problem we use the `try-except` clause.<jupyter_code># Creates empty lists that are going to store the data
Nomes = []
Emails = []

# loops for IDs from 1 to 2000
for i in tqdm(range(1, 3000)):    
    
    #if i%250 == 0:
     #   print('buscando pagina', i)
    
    response = requests.get('http://www.econ.puc-rio.br/pessoas/perfil/' + str(i))
    
    if 'Fatal error' in re.findall('Fatal error', response.text):
        continue  # skips to the next iteration
    else:
        nome_i = bs(response.content.decode('utf-8','ignore'), 'html.parser').find_all('h5')[0].text
        
        try:
            email_i = re.findall(regExpEmail, response.text)[0]
        except IndexError:
            email_i = ''
  
        Nomes.append(nome_i)
        Emails.append(email_i)
        
EmailsPUC = pd.DataFrame(data={'Nomes':Nomes, 'Emails':Emails}).sort_values('Nomes')
EmailsPUC
EmailsPUC[EmailsPUC['Nomes'].str.contains('Gustavo')].dropna().sort_values('Nomes')<jupyter_output><empty_output><jupyter_text>One way to protect yourself from webscraping is to not write your email **explicitly**. Here is an example from <NAME> [website](http://www.quantresearch.info/). --- Query Strings: URLs with ParametersEverything after "?" is the query string and it is meant to contain data that does not fit within a URL’s normal hierarchical path structurehttps://www.google.com.br/search?q=pesquisa+eleitoral+bolsonaro&num=3&as_sitesearch=g1.com.br* query string comes at the end of a URL, starting with a single question mark, “?”.* Parameters are provided as key-value pairs and separated by an ampersand, “&”.* The key and value are separated using an equals sign, “=”.<jupyter_code>url = 'https://www.google.com.br/search?q=pesquisa+eleitoral+bolsonaro&num=3&as_sitesearch=g1.com.br'
res = requests.get(url)
res.text<jupyter_output><empty_output><jupyter_text>We want an easy way to change the parameters of our search. Luckily, the `requests` library already has a functionality for URL queries.<jupyter_code>url = 'https://www.google.com.br/search'

param_dict = {'q': 'programa+de+governo', 
              'num': '3', 
              'as_sitesearch': 'g1.com.br'}

response = requests.get(url, params=param_dict)

response.url<jupyter_output><empty_output><jupyter_text>Practical Example - Derivatives Settlement Prices from B3An applied example of this is going to be relevant for is scraping sttlement prices from the B3 derivatives.http://www2.bmf.com.br/pages/portal/bmfbovespa/lumis/lum-sistema-pregao-enUS.asp?Data=10/01/2018&Mercadoria=DOL<jupyter_code>url = 'http://www2.bmf.com.br/pages/portal/bmfbovespa/lumis/lum-sistema-pregao-enUS.asp'
param_dict = {'Data': '10/01/2018', 'Mercadoria': 'DOL'}
r = requests.get(url, params=param_dict)
r.text<jupyter_output><empty_output><jupyter_text>--- WebdriversFirst we need to install the [**selenium webdriver library**](https://www.seleniumhq.org/projects/webdriver/). This library allows you to command the web browsers. We will use **Google Chrome** as our web browser, so we need the [**Chrome Driver**](https://sites.google.com/a/chromium.org/chromedriver/downloads)<jupyter_code>from selenium import webdriver
import datetime as dt
import time as time
import os
driver_options = webdriver.ChromeOptions()

browser = webdriver.Chrome(r'C:\Users\gamarante\Desktop\chromedriver', 
                           chrome_options=driver_options)

browser.get(r'https://www3.bcb.gov.br/expectativas/publico/consulta/serieestatisticas')<jupyter_output><empty_output><jupyter_text>select the indicator<jupyter_code>xpath = r'//*[@id="indicador"]/option[5]'
browser.find_element_by_xpath(xpath).click()<jupyter_output><empty_output><jupyter_text>Select the price index<jupyter_code>xpath = r'//*[@id="grupoIndicePreco:opcoes_5"]'
browser.find_element_by_xpath(xpath).click()<jupyter_output><empty_output><jupyter_text>select the metric<jupyter_code>xpath = r'//*[@id="calculo"]/option[3]'
browser.find_element_by_xpath(xpath).click()<jupyter_output><empty_output><jupyter_text>Select the periodicity<jupyter_code>xpath = r'//*[@id="periodicidade"]/option[3]'
browser.find_element_by_xpath(xpath).click()<jupyter_output><empty_output><jupyter_text>Fill the dates<jupyter_code>initial_date = dt.date(2018, 1, 10)
xpath = r'//*[@id="tfDataInicial1"]'
browser.find_element_by_xpath(xpath).send_keys(initial_date.strftime('%d/%m/%Y'))

end_date = dt.date(2018, 11, 16)
xpath = r'//*[@id="tfDataFinal2"]'
browser.find_element_by_xpath(xpath).send_keys(end_date.strftime('%d/%m/%Y'))<jupyter_output><empty_output><jupyter_text>Fill initial year and final year<jupyter_code>xpath = r'//*[@id="form4"]/div[2]/table/tbody[3]/tr/td[2]/select/option[2]'
browser.find_element_by_xpath(xpath).click()

# trick - always grabs the last element
xpath = r'//*[@id="form4"]/div[2]/table/tbody[3]/tr/td[4]/select'
selection = browser.find_element_by_xpath(xpath)
selection.click()
options = selection.find_elements_by_tag_name('option')
options[len(options) - 1].click()<jupyter_output><empty_output><jupyter_text>click the download button<jupyter_code>xpath = r'//*[@id="btnXLSa"]'
browser.find_element_by_xpath(xpath).click()

download_save_time = dt.datetime.now()  # saves the time the file was downloaded<jupyter_output><empty_output><jupyter_text>Now we close the browser<jupyter_code>browser.close()<jupyter_output><empty_output><jupyter_text>Get the default download directoryProbably only works on windows<jupyter_code>username = os.getlogin()
download_path = r'C:\Users\%(user)s\Downloads' % {'user': username}
download_path<jupyter_output><empty_output><jupyter_text>Explain the walk methoddirpathdirnamefilename<jupyter_code>list(os.walk(download_path))<jupyter_output><empty_output><jupyter_text>Look for the files in the download path<jupyter_code>for (dirpath, dirnames, filenames) in os.walk(download_path):
    
    for f in filenames:
    
        if 'Séries de estatísticas' in f:
            file_save_time = os.path.getmtime(dirpath + '\\' + f)
            file_save_time = dt.datetime.fromtimestamp(file_save_time)
            
            if file_save_time > download_save_time:
                file_path = dirpath + '\\' + f
                
print(file_path)<jupyter_output><empty_output><jupyter_text>read the file and clean the dataframe<jupyter_code>df = pd.read_excel(file_path, skiprows=1, na_values=[' '])

df['Data'] = pd.to_datetime(df['Data'], dayfirst=True)
df = df.set_index('Data')
df
import matplotlib.pyplot as plt
df.plot(figsize=(15, 8))
plt.show()<jupyter_output><empty_output><jupyter_text>All together now<jupyter_code>driver_options = webdriver.ChromeOptions()

browser = webdriver.Chrome(r'\\bw2k1201app\groupbrw$\BWGI\Thiago Barros\Drivers\chromedriver', 
                           chrome_options=driver_options)

browser.get(r'https://www3.bcb.gov.br/expectativas/publico/consulta/serieestatisticas')

xpath = r'//*[@id="indicador"]/option[5]'
browser.find_element_by_xpath(xpath).click()

xpath = r'//*[@id="grupoIndicePreco:opcoes_5"]'
browser.find_element_by_xpath(xpath).click()

xpath = r'//*[@id="calculo"]/option[3]'
browser.find_element_by_xpath(xpath).click()

xpath = r'//*[@id="periodicidade"]/option[3]'
browser.find_element_by_xpath(xpath).click()

initial_date = dt.date(2018, 1, 10)
xpath = r'//*[@id="tfDataInicial1"]'
browser.find_element_by_xpath(xpath).send_keys(initial_date.strftime('%d/%m/%Y'))

end_date = dt.date(2018, 11, 16)
xpath = r'//*[@id="tfDataFinal2"]'
browser.find_element_by_xpath(xpath).send_keys(end_date.strftime('%d/%m/%Y'))

xpath = r'//*[@id="form4"]/div[2]/table/tbody[3]/tr/td[2]/select/option[2]'
browser.find_element_by_xpath(xpath).click()

# trick - always grabs the last element
xpath = r'//*[@id="form4"]/div[2]/table/tbody[3]/tr/td[4]/select'
selection = browser.find_element_by_xpath(xpath)
selection.click()
options = selection.find_elements_by_tag_name('option')
options[len(options) - 1].click()

xpath = r'//*[@id="btnXLSa"]'
browser.find_element_by_xpath(xpath).click()

download_save_time = dt.datetime.now()  # saves the time the file was downloaded

time.sleep(6)  # give some time for the download to finish

browser.close()

username = os.getlogin()
download_path = r'C:\Users\%(user)s\Downloads' % {'user': username}

for (dirpath, dirnames, filenames) in os.walk(download_path):
    
    for f in filenames:
    
        if 'Séries de estatísticas' in f:
            file_save_time = os.path.getmtime(dirpath + '\\' + f)
            file_save_time = dt.datetime.fromtimestamp(file_save_time)
            
            if file_save_time > download_save_time:
                file_path = dirpath + '\\' + f
                
df = pd.read_excel(file_path, skiprows=1, na_values=[' '])

df['Data'] = pd.to_datetime(df['Data'], dayfirst=True)
df = df.set_index('Data')

df.plot(figsize=(15, 8))
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13292988313856427, 'fraction_numerical': 0.010781858653311074, 'mean_word_length': 5.231902904204595, 'pattern_counts': {'":': 0, '<': 122, '<?xml version=': 0, '>': 126, 'https://': 10, 'lorem ipsum': 0, 'www.': 18, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 21, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '30', 'dir': 'jupyter-structured-clean-dedup', 'id': '570211', 'license': 'MIT', 'n_tokens_mistral': 5075, 'n_tokens_neox': 4917, 'n_words': 1194, 'path': 'Python Lectures/Section 08 - Webscrapping and Webcrawling.ipynb', 'repo_name': 'Finance-Hub/FinanceHubMaterials', 'size': '14391'}