<jupyter_start><jupyter_text>Obtain Hex Addresses

We will use reverse geocoding to find the nearest address to each hex centre.<jupyter_code>import geopandas as gpd
import numpy as np
import plotly.express as px
from diskcache import Cache
from geopy import Nominatim
from geopy.distance import distance
from ratelimiter import RateLimiter
from tqdm.notebook import tqdm
df_hex = gpd.read_file('../data/BBMP_hex.geojson')
print(df_hex.shape)
df_hex.head()<jupyter_output>(942, 9)<jupyter_text>Reverse Geocoding

Data is loaded from the GeoJSON file created earlier and displayed. We obtain the closest addresses of each hex centre using Nominatim reverse geocoding. We need to cache the responses received, since querying Nominatim is quite slow, and we want to avoid repeating requests.<jupyter_code># Set cache location for Nominatim requests
nom_cache = '../data/cache/nominatim'

# Create Nominatim geocoder object with custom user agent as required by their terms of service
geocoder = Nominatim(user_agent = 'coursera_capstone')

# Create RateLimiter object to ensure we don't exceed Nominatim's 1 reqest per second rule
limiter = RateLimiter(max_calls = 1, period = 1)
# Empty lists to store responses
address = []
addr_coord = []

points = df_hex[['centre_lat', 'centre_lon']].to_records()

queries = 0 # Added variable to track number of records retrieved from cache

with Cache(nom_cache) as cache:
    for p in tqdm(points):
        query = (p.centre_lat, p.centre_lon)
        key = str(query) #! key must be a unique string
        
        if key in cache:
            response = cache[key] # Read cached value
        else:
            with limiter:
                response = geocoder.reverse(query, timeout = 30, addressdetails=True)
                cache[key] = response # Set cache value
                queries += 1
            
        address.append(response.address)
        addr_coord.append((response.latitude, response.longitude))
print('{} new queries made.'.format(queries))<jupyter_output><empty_output><jupyter_text>Calculating Distances

We use geopy to calculate the geodesic distance between each hexagon centre and the address obtained. We want to check if the addresses provided are close to the hex centres.<jupyter_code>centre_coord = df_hex[['centre_lat', 'centre_lon']].to_records(index=False)

address_error = [] # Empty list to store data

for centre, addr in zip(centre_coord, addr_coord):
    dist = distance(centre, addr).meters
    address_error.append(dist) # Use Geopy function

# Plot histogram
address_err_fig = px.histogram(
    x = address_error,
    histnorm = 'percent',
    #cumulative = True,
    template = 'plotly',
)

address_err_fig.update_layout(
    title = "Error in address locations",
    title_x = 0.5,
    xaxis_title = 'Distance between Hex centre & Address (meters)',
    yaxis_title = 'Percentage of hexes',
    bargap = 0.01,
)<jupyter_output><empty_output><jupyter_text>The reverse geocoding is accurate enough. Almost all of the addresses returned are within a few hundred meters of the hexagon centre. In any case, we will mainly be using the hex centres for future functions - the address is mainly for reference.<jupyter_code># Assign to new columns
df_hex['address'] = address
df_hex.to_feather('../data/bangalore_hex_addresses.feather') # Save file
df_hex.head() # Display final table<jupyter_output><ipython-input-6-61f42f6f3292>:3: UserWarning:

this is an initial implementation of Parquet/Feather file support and associated metadata.  This is tracking version 0.1.0 of the metadata specification at https://github.com/geopandas/geo-arrow-spec

This metadata specification does not yet make stability promises.  We do not yet recommend using this in a production setting unless you are able to rewrite your Parquet/Feather files.

To further ignore this warning, you can do: 
import warnings; warnings.filterwarnings('ignore', message='.*initial implementation of Parquet.*')
{'fraction_non_alphanumeric': 0.07634916687391197, 'fraction_numerical': 0.0074608306391444916, 'mean_word_length': 3.928921568627451, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '617505', 'license': 'MIT', 'n_tokens_mistral': 1228, 'n_tokens_neox': 1177, 'n_words': 423, 'path': 'notebooks/02_bangalore_hex_addresses.ipynb', 'repo_name': 'ngjoseph/Coursera_Capstone', 'size': '4021'}