<jupyter_start><jupyter_text>Python Requests(c) 2019, <NAME>.MIT License.Tutorial on using the [Requests](http://docs.python-requests.org/en/master/user/quickstart/) library to access HTTP requests, GET, POST, PUT, DELETE, HEAD, OPTIONS.  This notebook also covers how to use the Python [JSON](https://docs.python.org/3/library/json.html) library to parse values out of a GET response.  If you don't have the requests library installed you can run 'pip install requests' or some equivalent command for your system in the console window.<jupyter_code>import requests
import json
r = requests.get('https://api.github.com/events')
r = requests.post('https://httpbin.org/post', data = {'name':'Joe'})
r = requests.put('https://httpbin.org/put', data = {'name':'Joe'})
r = requests.delete('https://httpbin.org/delete')
r = requests.head('https://httpbin.org/get')
r = requests.options('https://httpbin.org/get')<jupyter_output><empty_output><jupyter_text>GET Requests - Passing Parameters in URLsA URL that returns an HTTP response in JSON format is called an API endpoint.  Here's an example, https://httpbin.org/get  With GET requests we can add parameters onto the URL to retrieve specific data.  We define the params as a dictionary, and add params=payload to the Request.  The Requests library builds the whole URL for us.<jupyter_code>payload = {'key1': 'value1', 'key2': 'value2'}
r = requests.get('https://httpbin.org/get', params=payload)
print(r.url)<jupyter_output>https://httpbin.org/get?key1=value1&key2=value2<jupyter_text>**Passing a List as a parameter**  Still use key:value pairs, with the list as the value.  You can see here all the different attributes included in an HTTP Request response.<jupyter_code>payload = {'key1': 'value1', 'key2': ['value2', 'value3']}
r = requests.get('https://httpbin.org/get', params=payload)
print('URL: ', r.url)
print('ENCODING: ', r.encoding)
print('STATUS_CODE: ', r.status_code)
print('HEADERS: ', r.headers)
print('TEXT: ', r.text)
print('CONTENT: ', r.content)
print('JSON: ', r.json)<jupyter_output>URL:  https://httpbin.org/get?key1=value1&key2=value2&key2=value3
ENCODING:  None
STATUS_CODE:  200
HEADERS:  {'Access-Control-Allow-Credentials': 'true', 'Access-Control-Allow-Origin': '*', 'Content-Encoding': 'gzip', 'Content-Type': 'application/json', 'Date': 'Tue, 26 Feb 2019 18:13:35 GMT', 'Server': 'nginx', 'Content-Length': '229', 'Connection': 'keep-alive'}
TEXT:  {
  "args": {
    "key1": "value1", 
    "key2": [
      "value2", 
      "value3"
    ]
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.21.0"
  }, 
  "origin": "192.168.127.12, 192.168.127.12", 
  "url": "https://httpbin.org/get?key1=value1&key2=value2&key2=value3"
}

CONTENT:  b'{\n  "args": {\n    "key1": "value1", \n    "key2": [\n      "value2", \n      "value3"\n    ]\n  }, \n  "headers": {\n    "Accept": "*/*", \n    "Accept-Encoding": "gzip, deflate", \n    "Host": "httpbin.org", \n    "User-Agent": "python-request[...]<jupyter_text>POST RequestsWe can add parameters to a POST request in Dictionary format, but we use data=payload.  POST requests are used to upload new records to the server.  POST would typically be used to get data from a web form and submit it to the server.<jupyter_code>r = requests.post('https://httpbin.org/post', data = {'name':'Joe'})

payload = {'key1': 'value1', 'key2': 'value2'}
r = requests.post("https://httpbin.org/post", data=payload)
print(r.text)<jupyter_output>{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "key1": "value1", 
    "key2": "value2"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "23", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.21.0"
  }, 
  "json": null, 
  "origin": "192.168.127.12, 192.168.127.12", 
  "url": "https://httpbin.org/post"
}<jupyter_text>Using Requests to GET Currency Exchange DataHere's a handy endpoint where we can GET foreign currency exchange rates in JSON format, https://api.exchangeratesapi.io/latest<jupyter_code>url = 'https://api.exchangeratesapi.io/latest'
r = requests.get(url)
print(r.text)<jupyter_output>{"rates":{"MXN":21.7145,"AUD":1.5897,"HKD":8.9178,"RON":4.7626,"HRK":7.4275,"CHF":1.1371,"IDR":15917.9,"CAD":1.5024,"USD":1.1361,"ZAR":15.752,"JPY":125.93,"BRL":4.2574,"HUF":317.06,"CZK":25.663,"NOK":9.7725,"INR":80.853,"PLN":4.3282,"ISK":136.1,"PHP":59.144,"SEK":10.5858,"ILS":4.1148,"GBP":0.86055,"SGD":1.5332,"CNY":7.6077,"TRY":6.0254,"MYR":4.6157,"RUB":74.6158,"NZD":1.652,"KRW":1270.0,"THB":35.583,"BGN":1.9558,"DKK":7.4616},"base":"EUR","date":"2019-02-26"}<jupyter_text>**It looks like the default is base:EUR, but we want exchange rates for USD, so we can pass in a parameter for base.  We can also put in any date we want.**<jupyter_code>url = 'https://api.exchangeratesapi.io/2018-01-15'
r = requests.get(url, params={'base':'USD'})
print(r.text)<jupyter_output>{"rates":{"MXN":18.8315549401,"AUD":1.2571475116,"HKD":7.823572534,"RON":3.7694876599,"HRK":6.0552252179,"CHF":0.9610654069,"IDR":13307.03754989,"CAD":1.2432190274,"USD":1.0,"JPY":110.621487334,"BRL":3.1959762157,"PHP":50.2997474953,"CZK":20.7957970188,"NOK":7.8771686894,"INR":63.5175531482,"PLN":3.3954549157,"MYR":3.9560153132,"ZAR":12.302191089,"ILS":3.399609025,"GBP":0.7252830496,"SGD":1.3214140262,"HUF":251.6086991936,"EUR":0.8145312373,"CNY":6.4380548994,"TRY":3.7828459721,"SEK":8.0096929217,"RUB":56.4333306182,"NZD":1.3706931661,"KRW":1063.5660177568,"THB":31.9247373137,"BGN":1.5930601939,"DKK":6.0679319052},"base":"USD","date":"2018-01-15"}<jupyter_text>Decoding JSON dataNow we have the rates in JSON format. We need to convert that to usable data. The JSON library basically has two functions:  - json.loads( ) converts a text string into Python dict/list objects.  - json.dumps( ) converts dict/list objects into a string.  We need to decode the JSON data into a dictionary, then get the rate for GBP, convert it to a float, and do a conversion.<jupyter_code>rates_json = json.loads(r.text)['rates']
print(rates_json)
print(rates_json['GBP'])
gbp = float(rates_json['GBP'])
print('200USD = ', str(gbp * 200), 'GBP')<jupyter_output>{'MXN': 18.8315549401, 'AUD': 1.2571475116, 'HKD': 7.823572534, 'RON': 3.7694876599, 'HRK': 6.0552252179, 'CHF': 0.9610654069, 'IDR': 13307.03754989, 'CAD': 1.2432190274, 'USD': 1.0, 'JPY': 110.621487334, 'BRL': 3.1959762157, 'PHP': 50.2997474953, 'CZK': 20.7957970188, 'NOK': 7.8771686894, 'INR': 63.5175531482, 'PLN': 3.3954549157, 'MYR': 3.9560153132, 'ZAR': 12.302191089, 'ILS': 3.399609025, 'GBP': 0.7252830496, 'SGD': 1.3214140262, 'HUF': 251.6086991936, 'EUR': 0.8145312373, 'CNY': 6.4380548994, 'TRY': 3.7828459721, 'SEK': 8.0096929217, 'RUB': 56.4333306182, 'NZD': 1.3706931661, 'KRW': 1063.5660177568, 'THB': 31.9247373137, 'BGN': 1.5930601939, 'DKK': 6.0679319052}
0.7252830496
200USD =  145.05660992 GBP<jupyter_text>Using Requests to GET Song DataEvery API has documentation on how to use it.  You can find the docs for this Song Data API [here.](https://documenter.getpostman.com/view/3719697/RzfarXB4)<jupyter_code>url = 'https://musicdemons.com/api/v1/artist'
r = requests.get(url)
print(r.text[:700])
url = 'https://musicdemons.com/api/v1/artist/21'
r = requests.get(url)
print(r.text)
url = 'https://musicdemons.com/api/v1/artist/21'
headers = {'with': 'songs,members'}
r = requests.get(url, headers=headers)
print(r.text[:700])
import json
text_json = json.loads(r.text)
print(text_json['name'])
for song in text_json['songs']:
    print(song['title'])<jupyter_output>Coldplay
Something Just Like This
Hymn For The Weekend
Sky Full Of Stars
Fix You
Brothers & Sisters
Shiver
The Scientist
Yellow
Trouble
Every Teardrop Is a Waterfall
Life in Technicolor ii
Adventure Of A Lifetime
Magic
The Hardest Part
Viva la Vida
1.36
42
A Head Full of Dreams
A Hopeful Transmission
A Message
A Rush of Blood to the Head
Princess of China
{'fraction_non_alphanumeric': 0.1860032262067254, 'fraction_numerical': 0.13326715473383793, 'mean_word_length': 5.272373540856031, 'pattern_counts': {'":': 104, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 23, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '36005', 'license': 'MIT', 'n_tokens_mistral': 3779, 'n_tokens_neox': 3086, 'n_words': 677, 'path': 'Web Data Mining/Python Requests.ipynb', 'repo_name': 'H2oPtic/Jupyter-Education', 'size': '8054'}