<jupyter_start><jupyter_text>Python and APIsAngela, Moritz & Thomas  Outline of Session 5In these slides* Introduction * JSON* Hardcoding access with `requests`* Popular pre-written modules for geocoding and mappingExternal scripts* Trade data with Comtrade API (kudos Federico)* The Twitter API (kudos Charles)  What is an API? API: Application programming interface_"a set of clearly defined methods of communication between various software components."_- Pre-internet days: Extension of software beyond its usual capabilities- Nowadays: Interface by web service providers for you to connect/retrieve with your own application (i.e. without going through GUI)- You send data  You get data back- Most APIs have a similar structure: the REST architecture (REpresentational State Transfer)- Inititatives like the Linux Foundation's [OpenAPI](https://www.openapis.org/) develop these types of standards  Illustration: General Set-upSource: [MuleSoft Videos](https://www.youtube.com/embed/s7wmiS2mSXY)  Illustration: Making a request  Illustration: Returning Content  Elements of an API* A protocol (eg: https)* A server (eg: httpbin.org)* A method name / location (eg: /get)* A set of arguments (eg: hello=world and foo=bar)➥ https://httpbin.org/get?hello=world&foo=bar  The Request* You need to read the documentation (e.g. [Eurostat API documentation](https://ec.europa.eu/eurostat/web/json-and-unicode-web-services/getting-started/rest-request))* You need to specify the URL (if it is a remote API)* You need to import the `requests` module in Python  The returning object* Usually it is encoded in JSON * The other popular formats for data structure are XML (mainly old stuff) and CSV (spreadsheet compatible)  What is a JSON?* JSON = [(JavaScript Object Notation)](https://www.json.org/)* It is one of the most popular format for data in the world* It looks like a Python dictionary, except for the fact that:    - JSON is a string (it is inside a text file)    - JSON must use double quotation mark    - in JavaScript it is defined as Object* In Python there is a built-in module called `json` module * It follows this strcture {"key" : "value"}; see example below:<jupyter_code>{
  "friends": [
    {
      "name": "Jose",
      "degree": "Applied Computing"
    },
    {
      "name": "Rolf",
      "degree": "Computer Science"
    },
    {
      "name": "Anna",
      "degree": "Physics"
    }
  ]
}<jupyter_output><empty_output><jupyter_text>What can I do with APIs? - Retrieve [World Bank Dev Indicators](https://datahelpdesk.worldbank.org/knowledgebase/articles/898581-api-basic-call-structures)- Track stock prices with [OpenFIGI](https://www.openfigi.com/api)- Geocode an address with [Here Maps](https://developer.here.com/)- Convert fiat currency with [opencurrency](https://openexchangerates.org/) or crypto with [alternative.me](https://alternative.me/crypto/api/)- Send tweets with [Twitter](https://developer.twitter.com/)- Download images from Mars or the Moon from [NASA](https://api.nasa.gov/)- ...   The Starwars Character API- Let's get data on Starwars characters! - We find [the documentation](https://swapi.co/documentation) - We import the `requests` module - We set the base URL of the API: `https://swapi.co/api/`<jupyter_code>import requests

url = "https://swapi.co/api/"

# Launch request
base_req = requests.get(url)
print(base_req)
# Explore json content of object
print(base_req.json())<jupyter_output>{'people': 'https://swapi.co/api/people/', 'planets': 'https://swapi.co/api/planets/', 'films': 'https://swapi.co/api/films/', 'species': 'https://swapi.co/api/species/', 'vehicles': 'https://swapi.co/api/vehicles/', 'starships': 'https://swapi.co/api/starships/'}<jupyter_text>Making sense of the answer- The call to the basic URL returns various subfields`{    'people': 'https://swapi.co/api/people/',     'planets': 'https://swapi.co/api/planets/',     'films': 'https://swapi.co/api/films/',     'species': 'https://swapi.co/api/species/',     'vehicles': 'https://swapi.co/api/vehicles/',     'starships': 'https://swapi.co/api/starships/'    }`- The documentation adds: `/people/:id/ -- get a specific people resource`- Let's request info on the first id of the subfield `people`- We can request further info on a sub-subfield (e.g. `starships`)<jupyter_code># Let's try
url = "https://swapi.co/api/people/1"

# Launch request
req_1 = requests.get(url)

# Extract json
req_1_js = req_1.json()
print(req_1_js)
# Let's extract Luke's starships
ships = req_1_js.get("starships")
print(ships)

# And get entry of first starship
ship = requests.get(ships[0]).json()
print(ship)<jupyter_output>['https://swapi.co/api/starships/12/', 'https://swapi.co/api/starships/22/']
{'name': 'X-wing', 'model': 'T-65 X-wing', 'manufacturer': 'Incom Corporation', 'cost_in_credits': '149999', 'length': '12.5', 'max_atmosphering_speed': '1050', 'crew': '1', 'passengers': '0', 'cargo_capacity': '110', 'consumables': '1 week', 'hyperdrive_rating': '1.0', 'MGLT': '100', 'starship_class': 'Starfighter', 'pilots': ['https://swapi.co/api/people/1/', 'https://swapi.co/api/people/9/', 'https://swapi.co/api/people/18/', 'https://swapi.co/api/people/19/'], 'films': ['https://swapi.co/api/films/2/', 'https://swapi.co/api/films/3/', 'https://swapi.co/api/films/1/'], 'created': '2014-12-12T11:19:05.340000Z', 'edited': '2014-12-22T17:35:44.491233Z', 'url': 'https://swapi.co/api/starships/12/'}<jupyter_text>Geocoding- Most map services like Google Maps, Open Street Maps, Bing, etc. have APIs- Let's check out Open Street Maps' (OSM) API called [Nominatim](https://developer.mapquest.com/)- We will use the API through the Python module `geopy`- To install new modules check out this [tutorial](https://jakevdp.github.io/blog/2017/12/05/installing-python-packages-from-jupyter/)- But we still need to sign up with our email to obtain a key for Nominatim  Retriving a location* We install `geopy`* We sign up and obtain the key (in this example we use Moritz's key)* We search the location of two popular places among ULB students: the ULB and the bar Tavernier* We store and print out their coordinates<jupyter_code>#import sys 
#!{sys.executable} -m pip install geopy

from geopy.geocoders import Nominatim

# Replace the user agent key with your own key
geolocator = Nominatim(user_agent="jjG4qnPniTAGpG7O0q8XcMhARm0Pxcln")
location_ULB = geolocator.geocode("Universite libre de Bruxelles")

location_tav = geolocator.geocode("Bar Tavernier Ixelles")


# Did it work?
print(location_ULB)
print(location_tav)
# Like a charm...

# Store coordinates in tuple
ulb_coords = [location_ULB.latitude, location_ULB.longitude]
print(ulb_coords)

tav_coords = [location_tav.latitude, location_tav.longitude]
print(tav_coords)<jupyter_output>Université Libre de Bruxelles (Campus du Solbosch), Square <NAME>ze - Albert Devèzesquare, Ixelles - Elsene, Région de Bruxelles-Capitale - Brussels Hoofdstedelijk Gewest, 1050, België - Belgique - Belgien
Tavernier, 445, Chaussée de Boondael - Boondaalse Steenweg, Ixelles - Elsene, Région de Bruxelles-Capitale - Brussels Hoofdstedelijk Gewest, 1050, België - Belgique - Belgien
[50.8134537, 4.381308819663028]
[50.816496, 4.388727177209292]<jupyter_text>Mapping- Sadly, not everybody can associate a place to its coordinates in their mind :-P - With the free [Leaflet API](https://leafletjs.com/reference-1.6.0.html) we can pin point them on a map- Unfortunately written in Java Script- But no need to learn, use "wrapper" Python Module `Folium`- Example: Map of environmental violations of shale gas drilling in Pennsyilvania ([website](http://stateimpact.npr.org/pennsylvania/drilling/violations/))   Producing a map* We install `Folium`* We create a map for the ULB, specifying the location and a zoom level* We display the map<jupyter_code># Install Python module Folium, a wrapper of the Leaflet API
#import sys
#!{sys.executable} -m pip install folium

import folium
from folium.plugins import MarkerCluster
import pandas as pd

#Create the map
my_map = folium.Map(location = ulb_coords, zoom_start = 14)



# Display map
my_map<jupyter_output><empty_output><jupyter_text>Adding markers to the map* It is still not clear where the ULB and Tavernier are located: we need some markers!* We pin point them on the map by adding some representative icons ;-)<jupyter_code># Let's add markers


folium.Marker(ulb_coords, popup = 'ULB', icon = folium.Icon(icon='book')).add_to(my_map)
folium.Marker(tav_coords, popup = 'STATA User Meeting', icon = folium.Icon(icon='glass', color='red')).add_to(my_map)

my_map<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13004536466209143, 'fraction_numerical': 0.02035593811794812, 'mean_word_length': 4.813387423935091, 'pattern_counts': {'":': 7, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 41, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '506970', 'license': 'Apache-2.0', 'n_tokens_mistral': 2971, 'n_tokens_neox': 2775, 'n_words': 888, 'path': 'Session_5/Session 5 - APIs Intro.ipynb', 'repo_name': 'CharlesHoffreumon/Classes', 'size': '8602'}