<jupyter_start><jupyter_text>1.4 Geetting data=============================This notebook will introduce the techniques to access data in the web using services request.Accessing data via services request is the easiest way. The data is requested using an API available at the server.Before getting access to it, the user must get an **api_key** to authenticate the call.The Python's module dealing with the API mode is **requests**<jupyter_code>import requests
import pandas as pd
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>We need the URL where data are located.URL's tend to have information related to the data we are after.<jupyter_code>url = 'https://www.quandl.com/api/v3/datasets/CHRIS/MGEX_IH1.json?api_key=<KEY>'<jupyter_output><empty_output><jupyter_text>Here, **CHRIS** is the dataset, **MGEX_IH1** is the data code and **DotPcR2Bpi7uezRsnS-S** is the api_key.We can use a clever way to create the URL.<jupyter_code>dataset = 'EURONEXT'
code = 'TFA'
apikey = '<KEY>'
url = 'https://www.quandl.com/api/v3/datasets/%s/%s.json?api_key=%s' % (dataset,code,apikey)
print(url)<jupyter_output>https://www.quandl.com/api/v3/datasets/EURONEXT/TFA.json?api_key=DotPcR2Bpi7uezRsnS-S<jupyter_text>To get the data, we simply need to call the **get** method from **requests**<jupyter_code>response = requests.get(url)<jupyter_output><empty_output><jupyter_text>The return is a JSON file (look at the URL...).JSON files are the most frequent file format when accesing data through APIs.We just need to _activate_ the response with the **.json** method<jupyter_code>RESPONSE = response.json()<jupyter_output><empty_output><jupyter_text>Now we are able to print the RESPONSE.AS you can see the JSON format looks very similar to a dictionnary...<jupyter_code>RESPONSE<jupyter_output><empty_output><jupyter_text>Likewise dictionaries, information is accesible through _keys_<jupyter_code>RESPONSE['dataset']['column_names']<jupyter_output><empty_output><jupyter_text>And we can convert the data into a Pandas DataFrame.<jupyter_code>DATA = pd.DataFrame.from_dict(RESPONSE['dataset']['data'])
# Let's rename the columns
DATA.columns = RESPONSE['dataset']['column_names']
DATA.tail()<jupyter_output><empty_output><jupyter_text>Done! Now we can play around with the data.<jupyter_code>plt.plot(DATA['Last'])<jupyter_output><empty_output><jupyter_text>If the information is not available through an API, we do what is called '_web scraping_'. First we need to download the HTML page.<jupyter_code>url = 'https://markets.ft.com/data/funds/tearsheet/historical?s=LU0207025593:EUR'
page = requests.get(url)<jupyter_output><empty_output><jupyter_text>In Python, the module dealing with HTML contents is called **BeautifulSoup**<jupyter_code>from bs4 import BeautifulSoup<jupyter_output><empty_output><jupyter_text>We apply the html parser to the requests return<jupyter_code>soup = BeautifulSoup(page.text, 'html.parser')<jupyter_output><empty_output><jupyter_text>soup looks like a typical web page now<jupyter_code>soup<jupyter_output><empty_output><jupyter_text>Now the hard bit... We need to insvestigate where is the information we are after.<jupyter_code>spansymb = soup.find_all('div',class_ = 'mod-tearsheet-overview__header__symbol')
spansymb[0].span.text
tdpri = soup.find_all('table', class_ ="mod-ui-table mod-tearsheet-historical-prices__results mod-ui-table--freeze-pane")
for row in tdpri:
    for tds in row.find_all('td'):
        
        try:
            print(tds.span.text)
        except:
            print(tds.text)<jupyter_output>Friday, January 25, 2019
274.80
274.80
274.80
274.80
--
Thursday, January 24, 2019
271.93
271.93
271.93
271.93
--
Wednesday, January 23, 2019
270.52
270.52
270.52
270.52
--
Tuesday, January 22, 2019
271.43
271.43
271.43
271.43
--
Monday, January 21, 2019
272.62
272.62
272.62
272.62
--
Friday, January 18, 2019
273.09
273.09
273.09
273.09
--
Thursday, January 17, 2019
269.22
269.22
269.22
269.22
--
Wednesday, January 16, 2019
268.36
268.36
268.36
268.36
--
Tuesday, January 15, 2019
266.39
266.39
266.39
266.39
--
Monday, January 14, 2019
266.92
266.92
266.92
266.92
--
Friday, January 11, 2019
268.56
268.56
268.56
268.56
--
Thursday, January 10, 2019
269.72
269.72
269.72
269.72
--
Wednesday, January 09, 2019
269.36
269.36
269.36
269.36
--
Tuesday, January 08, 2019
265.53
265.53
265.53
265.53
--
Monday, January 07, 2019
260.96
260.96
260.96
260.96
--
Friday, January 04, 2019
259.78
259.78
259.78
259.78
--
Thursday, January 03, 2019
252.58
252.58
252.58
252.58
--
Wednesday, January 02, 2019
[...]
{'fraction_non_alphanumeric': 0.12240437158469945, 'fraction_numerical': 0.10295081967213114, 'mean_word_length': 5.217391304347826, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '308094', 'license': 'Apache-2.0', 'n_tokens_mistral': 2018, 'n_tokens_neox': 1643, 'n_words': 445, 'path': '1.4-Getting_data.ipynb', 'repo_name': 'mlcordoba/DA_workshop', 'size': '4605'}