<jupyter_start><jupyter_text>APIsOverview of today's topic:  - What are APIs and how do you work with them?  - Geocoding place names and addresses  - Reverse-geocoding coordinates  - Looking up places near some location  - Using data portals programmaticallyTo follow along with this lecture, you need a working Google API key to use the Google Maps Geocoding API and the Google Places API Web Service. These APIs require you to set up billing info, but we won't use them in class beyond the free threshold.<jupyter_code>import re
import time

import folium
import geopandas as gpd
import osmnx as ox
import pandas as pd
import requests
from geopy.geocoders import GoogleV3
from keys import google_api_key

# define a pause duration between API requests
pause = 0.1<jupyter_output><empty_output><jupyter_text>An API is an application programming interface. It provides a structured way to send commands or requests to a piece of software. "API" often refers to a web service API. This is like web site (but designed for applications, rather than humans, to use) that you can send requests to to execute commands or query for data. Today, REST APIs are the most common. To use them, you simply send them a request, and they reply with a response, similar to how a web browser works. The request is sent to an endpoint (a URL) typically with a set of parameters to provide the details of your query or command.In the example below, we make a request to the [ipify](https://www.ipify.org/) API and request a JSON formatted response. Then we look up the location of the IP address it returned, using the [ip-api](https://ip-api.com/) API.<jupyter_code># what is your current public IP address?
url = "https://api.ipify.org?format=json"
data = requests.get(url).json()
data
# and what is the location of that IP address?
url = "http://ip-api.com/json/" + data["ip"]
requests.get(url).json()<jupyter_output><empty_output><jupyter_text>What's the current weather? Use the [National Weather Service](https://www.weather.gov/documentation/services-web-api) API.<jupyter_code># query for the forecast url for a pair of lat-lng coords
location = "34.019268,-118.283554"
url = "https://api.weather.gov/points/" + location
data = requests.get(url).json()

# extract the forecast url and retrieve it
forecast_url = data["properties"]["forecast"]
forecast = requests.get(forecast_url).json()

# convert the forecast to a dataframe
pd.DataFrame(forecast["properties"]["periods"]).head()<jupyter_output><empty_output><jupyter_text>You can use any web service's API in this same basic way: request the URL with some parameters. Read the API's documentation to know how to use it and what to send. You can also use many web service's through a Python package to make complex services easier to work with. For example, there's a fantastic package called [cenpy](https://cenpy-devs.github.io/cenpy/) that makes downloading and working with US census data super easy. 1. Geocoding"Geocoding" means converting a text description of some place (such as the place's name or its address) into geographic coordinates identifying the place's location on Earth. These geographic coordinates may take the form of a single latitude-longitude coordinate pair, or a bounding box, or a boundary polygon, etc. 1a. Geocoding place names with OpenStreetMap via OSMnx[OpenStreetMap](https://www.openstreetmap.org/) is a worldwide mapping platform that anyone can contribute to. [OSMnx](https://github.com/gboeing/osmnx) is a Python package to work with OpenStreetMap for geocoding, downloading geospatial data, and modeling/analyzing networks. OpenStreetMap and OSMnx are free to use and do not require an API key.<jupyter_code># geocode a place name to lat-lng
place = "University of Southern California"
latlng = ox.geocode(place)
latlng
# geocode a series of place names to lat-lng
places = pd.Series(["San Diego, California",
                    "Los Angeles, California",
                    "San Francisco, California",
                    "Seattle, Washington",
                    "Vancouver, British Columbia"])
coords = places.map(ox.geocode)
# parse out lats and lngs to individual columns in a dataframe
pd.DataFrame({"place": places,
              "lat": coords.map(lambda x: x[0]),
              "lng": coords.map(lambda x: x[1])})<jupyter_output><empty_output><jupyter_text>Instead of lat-lng coordinates, we can also geocode place names to their place's *boundaries* with OSMnx. This essentially looks-up the place in OpenStreetMap's database (note: that means the place has to exist in its database!) then returns its details, including geometry and bounding box, as a GeoPandas GeoDataFrame.<jupyter_code># geocode a list of place names to a GeoDataFrame
# by default, OSMnx retrieves the first [multi]polygon object
# specify which_result=1 to retrieve the top match, regardless of geometry type
gdf_places = ox.geocode_to_gdf(places.to_list(), which_result=1)
gdf_places
# geocode a single place name to a GeoDataFrame
gdf = ox.geocode_to_gdf(place)
gdf
# extract the value from row 0's geometry column
polygon = gdf["geometry"].iloc[0]
polygon<jupyter_output><empty_output><jupyter_text>Use OSMnx to query for geospatial entities within USC's boundary polygon. You can specify what kind of entities to retrieve by using a `tags` dictionary.<jupyter_code># get all the buildings within that polygon
tags = {"building": True}
gdf_bldg = ox.geometries_from_polygon(polygon, tags)
gdf_bldg.shape
# plot the building footprints
fig, ax = ox.plot_footprints(gdf_bldg)
# now it's your turn
# get all the building footprints within santa monica
# model the street network for some city
G = ox.graph_from_place("Culver City, California", network_type="drive")
fig, ax = ox.plot_graph(G)
# calculate basic street network statistics
ox.basic_stats(G)
# solve a shortest path (by travel time) between two nodes
orig = list(G)[10]
dest = list(G)[100]
G = ox.add_edge_travel_times(ox.add_edge_speeds(G))
route = ox.shortest_path(G, orig=orig, dest=dest, weight="travel_time")
fig, ax = ox.plot_graph_route(G, route)<jupyter_output><empty_output><jupyter_text>OSMnx is pretty powerful! If you want to play around with it, check out its [documentation](https://osmnx.readthedocs.io/en/stable/) and interactive [usage examples](https://github.com/gboeing/osmnx-examples). 1b: Geocoding addresses to lat-lngYou can geocode addresses as well with OpenStreetMap, but it can be a little hit-or-miss compared to the data coverage of commercial closed-source services.<jupyter_code># geocode an address to lat-lng
address = "704 S Alvarado St, Los Angeles, California"
latlng = ox.geocode(address)
latlng<jupyter_output><empty_output><jupyter_text>We will use the Google Maps geocoding API. Their geocoder is very powerful, but you do have to pay for it beyond a certain threshold of free usage.Documentation: https://developers.google.com/maps/documentation/geocoding/start<jupyter_code>locations = pd.DataFrame(["704 S Alvarado St, Los Angeles, CA",
                          "100 Larkin St, San Francisco, CA",
                          "350 5th Ave, New York, NY"], columns=["address"])
locations
# function accepts an address string, sends it to Google API, returns lat-lng result
def geocode(address, print_url=False):

    # pause for some duration before each request, to not hammer their server
    time.sleep(pause)

    # api url with placeholders to fill in with variables' values
    url_template = "https://maps.googleapis.com/maps/api/geocode/json?address={}&key={}"
    url = url_template.format(address, google_api_key)
    if print_url:
        print(url)

    # send request to server, get response, and convert json string to dict
    data = requests.get(url).json()

    # if results were returned, extract lat-lng from top result
    if len(data["results"]) > 0:
        lat = data["results"][0]["geometry"]["location"]["lat"]
        lng = data["results"][0]["geometry"]["location"]["lng"]

        # return lat-lng as a string
        return "{},{}".format(lat, lng)
# test the function
geocode("350 5th Ave, New York, NY")
# for each value in the address column, geocode it, save results as new column
locations["latlng"] = locations["address"].map(geocode)
locations
# parse the result into separate lat and lng columns, if desired
locations[["lat", "lng"]] = pd.DataFrame(data=locations["latlng"].str.split(",").to_list())
locations
# now it's your turn
# create a new pandas series of 3 addresses and use our function to geocode them
# then create a new pandas series of 3 famous site names and use our function to geocode them
# create new variables to contain your work so as to not overwrite the locations df<jupyter_output><empty_output><jupyter_text>2. Google Places APIWe will use Google's Places API to look up places in the vicinity of some location.Documentation: https://developers.google.com/places/web-service/intro<jupyter_code># google places API URL, with placeholders
url_template = "https://maps.googleapis.com/maps/api/place/search/json?keyword={}&location={}&radius={}&key={}"

# what keyword to search for
keyword = "restaurant"

# define the radius (in meters) for the search
radius = 500

# define the location coordinates
location = "34.019268,-118.283554"
# add our variables into the url, submit the request to the api, and load the response
url = url_template.format(keyword, location, radius, google_api_key)
response = requests.get(url)
data = response.json()
# how many results did we get?
len(data["results"])
# inspect a result
data["results"][0]
# turn the results into a dataframe of places
places = pd.DataFrame(data=data["results"], columns=["name", "geometry", "rating", "vicinity"])
places.head()
# parse out lat-long and return it as a series
# this creates a dataframe of all the results when you .apply()
def parse_coords(geometry):
    if isinstance(geometry, dict):
        lng = geometry["location"]["lng"]
        lat = geometry["location"]["lat"]
        return pd.Series({"lat": lat, "lng": lng})


# test our function
places["geometry"].head().apply(parse_coords)
# now run our function on the whole dataframe and save the output to 2 new dataframe columns
places[["lat", "lng"]] = places["geometry"].apply(parse_coords)
places_clean = places.drop("geometry", axis="columns")
# sort the places by rating
places_clean = places_clean.sort_values(by="rating", ascending=False)
places_clean.head(10)
# now it's your turn
# find the five highest-rated bars within 1/2 mile of pershing square
# create new variables to contain your work so as to not overwrite places and places_clean<jupyter_output><empty_output><jupyter_text>3. Reverse geocodingReverse geocoding, as you might expect from its name, does the opposite of regular geocoding: it takes a pair of coordinates on the Earth's surface and looks up what address or place corresponds to that location.We'll use Google's reverse geocoding API. Documentation: https://developers.google.com/maps/documentation/geocoding/introReverseGeocodingAs we saw with OSMnx, you often don't have to query the API yourself manually: many popular APIs have dedicated Python packages to work with them. You *can* do this manually, just like in the previous Google examples, but it's a little more complicated to parse Google's address component results. If we just want addresses, we can use [geopy](https://geopy.readthedocs.io/) to simply interact with Google's API automatically for us.<jupyter_code># we'll use the points from the Places API, but you could use any point data here
points = places_clean[["lat", "lng"]].head()
points
# create a column to put lat-lng into the format google likes
points["latlng"] = points.apply(lambda row: "{},{}".format(row["lat"], row["lng"]), axis="columns")
points.head()
# tell geopy to reverse geocode using Google's API and return address
def reverse_geopy(latlng):
    time.sleep(pause)
    geocoder = GoogleV3(api_key=google_api_key)
    address, _ = geocoder.reverse(latlng, exactly_one=True)
    return address
# now reverse-geocode the points to addresses
points["address"] = points["latlng"].map(reverse_geopy)
points.head()<jupyter_output><empty_output><jupyter_text>4. Data PortalsMany governments and agencies now open up their data to the public through a data portal. These often offer APIs to query them for real-time data. This example uses the LA Open Data Portal... browse the portal for public datasets: https://data.lacity.org/browseLet's look at parking meter data for those that have sensors telling us if they're currently occupied or vacant: https://data.lacity.org/A-Livable-and-Sustainable-City/LADOT-Parking-Meter-Occupancy/e7h6-4a3e<jupyter_code># define API endpoint
url = "https://data.lacity.org/resource/e7h6-4a3e.json"

# request the URL and download its response
response = requests.get(url)

# parse the json string into a Python dict
data = response.json()
len(data)
# turn the json data into a dataframe
df = pd.DataFrame(data)
df.shape
df.columns
df.head()<jupyter_output><empty_output><jupyter_text>We have parking space ID, occupancy status, and reporting time. But we don't know where these spaces are! Fortunately the LA GeoHub has sensor location data: http://geohub.lacity.org/datasets/parking-meter-sensors/data<jupyter_code># define API endpoint
url = "https://opendata.arcgis.com/datasets/723c00530ea441deaa35f25e53d098a8_16.geojson"

# request the URL and download its response
response = requests.get(url)

# parse the json string into a Python dict
data = response.json()
len(data["features"])
# turn the geojson data into a geodataframe
gdf = gpd.GeoDataFrame.from_features(data)
gdf.shape
# what columns are in our data?
gdf.columns
gdf.head()
# now merge sensor locations with current occupancy status
parking = pd.merge(left=gdf, right=df, left_on="SENSOR_UNIQUE_ID", right_on="spaceid", how="inner")
parking.shape
parking = parking[["occupancystate", "geometry", "ADDRESS_SPACE"]]

# extract lat and lon from geometry column
parking["lon"] = parking["geometry"].x
parking["lat"] = parking["geometry"].y

parking
# how many vacant vs occupied spots are there right now?
parking["occupancystate"].value_counts()
# map it
vacant = parking[parking["occupancystate"] == "VACANT"]
ax = vacant.plot(c="b", markersize=1, alpha=0.5)

occupied = parking[parking["occupancystate"] == "OCCUPIED"]
ax = vacant.plot(ax=ax, c="r", markersize=1, alpha=0.5)<jupyter_output><empty_output><jupyter_text>That's impossible to see! At this scale, all the vacant spots are obscured by occupied spots next to them. It would be much better if we had an interactive map...<jupyter_code># create leaflet web map centered/zoomed to downtown LA
m = folium.Map(location=(34.05, -118.25), zoom_start=15, tiles="cartodbpositron")

# add blue markers for each vacant spot
cols = ["lat", "lon", "ADDRESS_SPACE"]
for lat, lng, address in vacant[cols].values:
    folium.CircleMarker(location=(lat, lng),
                        radius=5,
                        color="#3186cc",
                        fill=True,
                        fill_color="#3186cc",
                        tooltip=address).add_to(m)

# add red markers for each occupied spot
for lat, lng, address in occupied[cols].values:
    folium.CircleMarker(location=(lat, lng),
                        radius=5,
                        color="#dc143c",
                        fill=True,
                        fill_color="#dc143c",
                        tooltip=address).add_to(m)
# now view the web map we created
m<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09660406483148958, 'fraction_numerical': 0.009647543092359146, 'mean_word_length': 4.579117330462863, 'pattern_counts': {'":': 6, '<': 53, '<?xml version=': 0, '>': 54, 'https://': 20, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '274667', 'license': 'MIT', 'n_tokens_mistral': 4733, 'n_tokens_neox': 4493, 'n_words': 1746, 'path': 'modules/12-apis/lecture.ipynb', 'repo_name': 'gboeing/ppd430', 'size': '15548'}