<commit_before># Pulling data from (open) REST APIs

[Huge source of public APIs](https://www.publicapis.com/)

We have already seen how to use `requests` to fetch a webpage:

```python
r = requests.get('http://www.cnn.com')
print r.text
```

If the URL is to a page that gives you HTML, we would say that we are fetching a webpage. On the other hand, if the URL is returning data in some form, we would say that we are accessing a *REST* api.
 
**REST** is an acronym for *REpresentational State Transfer* and is a very handy way to make something trivial sound very complicated.  Anytime you see the word REST, just think "webpage that gives me data not HTML." There is a massive industry and giant following behind this term but I don't see anything beyond "fetch data from webpage".

Anyway, we are going to pull data from web servers that intentionally provide nice data spigot URLs. Information you need in order to get data is typically:

* Base URL, including machine name, port number, and "file" path
* The names and values of parameters
* What data comes back and in what format (XML, JSON, CSV, ...)

## Historical stock data

*Yahoo's API was taken down in 2017 so we will use Quandl instead.* 

Register for an API key at [Quandl](https://www.quandl.com/account/api) and then you can use this URL to access, for example, Apple's stock data:

```
https://www.quandl.com/api/v3/datatables/WIKI/PRICES.csv?ticker=AAPL&api_key=YOURAPIKEY
```

You have to replace the `AAPL` with whatever stock ticker you want and of course replace `YOURAPIKEY` with your specific API key that you got when you registered.
 
Code: [Quandl finance stock history](notes/code/quandl/history.py):

```python
import sys
import requests

HistoryURL = "https://www.quandl.com/api/v3/datatables/WIKI/PRICES.csv?ticker=%s&api_key=%s"

ticker = sys.argv[1]  # AAPL
APIKEY = sys.argv[2]  # your key

url = HistoryURL % (ticker,APIKEY)
r = requests.get(url)
csvdata = r.text
print csvdata
```

The CSV you get back looks like:

```
ticker,date,open,high,low,close,volume,ex-dividend,split_ratio,adj_open,adj_high,adj_low,adj_close,adj_volume
AAPL,1980-12-12,28.75,28.87,28.75,28.75,2093900.0,0.0,1.0,0.42270591588018,0.42447025361603,0.42270591588018,0.42270591588018,117258400.0
AAPL,1980-12-15,27.38,27.38,27.25,27.25,785200.0,0.0,1.0,0.40256306006259,0.40256306006259,0.40065169418209,0.40065169418209,43971200.0
...
```

```
# csv is easy to handle ourselves:
for row in csvdata.strip().split("\n"):
    cols = row.split(',')
    print ', '.join(cols)
```

**Exercise:** By looking at the [quandl doc](https://docs.quandl.com/docs/in-depth-usage-1) fetch and fetch stock history for TSLA for just 2015 and only get columns `data` and `open`.

```python
QuoteURL = "https://www.quandl.com/api/v3/datatables/WIKI/PRICES.csv?ticker=%s&api_key=YOURAPIKEY&date.gte=%s&date.lt=%s&qopts.columns=%s"
```
<commit_msg>tweak<commit_after># Pulling data from (open) REST APIs

[Huge source of public APIs](https://www.publicapis.com/)

We have already seen how to use `requests` to fetch a webpage:

```python
r = requests.get('http://www.cnn.com')
print r.text
```

If the URL is to a page that gives you HTML, we would say that we are fetching a webpage. On the other hand, if the URL is returning data in some form, we would say that we are accessing a *REST* api.
 
**REST** is an acronym for *REpresentational State Transfer* and is a very handy way to make something trivial sound very complicated.  Anytime you see the word REST, just think "webpage that gives me data not HTML." There is a massive industry and giant following behind this term but I don't see anything beyond "fetch data from webpage".

Anyway, we are going to pull data from web servers that intentionally provide nice data spigot URLs. Information you need in order to get data is typically:

* Base URL, including machine name, port number, and "file" path
* The names and values of parameters
* What data comes back and in what format (XML, JSON, CSV, ...)

## Historical stock data

*Yahoo's API was taken down in 2017 so we will use Quandl instead.* 

Register for an API key at [Quandl](https://www.quandl.com/account/api) and then you can use this URL to access, for example, Apple's stock data:

```
https://www.quandl.com/api/v3/datatables/WIKI/PRICES.csv?ticker=AAPL&api_key=YOURAPIKEY
```

You have to replace the `AAPL` with whatever stock ticker you want and of course replace `YOURAPIKEY` with your specific API key that you got when you registered.
 
Code: [Quandl finance stock history](notes/code/quandl/history.py):

```python
import sys
import requests

HistoryURL = "https://www.quandl.com/api/v3/datatables/WIKI/PRICES.csv?ticker=%s&api_key=%s"

ticker = sys.argv[1]  # AAPL
APIKEY = sys.argv[2]  # your key

url = HistoryURL % (ticker,APIKEY)
r = requests.get(url)
csvdata = r.text
print csvdata
```

The CSV you get back looks like:

```
ticker,date,open,high,low,close,volume,ex-dividend,split_ratio,adj_open,adj_high,adj_low,adj_close,adj_volume
AAPL,1980-12-12,28.75,28.87,28.75,28.75,2093900.0,0.0,1.0,0.42270591588018,0.42447025361603,0.42270591588018,0.42270591588018,117258400.0
AAPL,1980-12-15,27.38,27.38,27.25,27.25,785200.0,0.0,1.0,0.40256306006259,0.40256306006259,0.40065169418209,0.40065169418209,43971200.0
...
```

```
# csv is easy to handle ourselves:
for row in csvdata.strip().split("\n"):
    cols = row.split(',')
    print ', '.join(cols)
```

**Exercise:** By looking at the [quandl usage doc](https://docs.quandl.com/docs/in-depth-usage-1) and [quandl parameter doc](https://docs.quandl.com/docs/parameters-1) fetch and fetch stock history for TSLA for just 2015 and only get columns `data` and `open`.

```python
QuoteURL = "https://www.quandl.com/api/v3/datatables/WIKI/PRICES.csv?ticker=%s&api_key=YOURAPIKEY&date.gte=%s&date.lt=%s&qopts.columns=%s"
```

{'fraction_non_alphanumeric': 0.12230953194396993, 'fraction_numerical': 0.07669969251793646, 'mean_word_length': 4.678952473326867, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 13, 'lorem ipsum': 0, 'www.': 12, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1407308', 'n_tokens_mistral': 2345, 'n_tokens_neox': 2028, 'n_words': 661}