<jupyter_start><jupyter_text>Calgary Data for Good Micro Hackathon 1Author: <NAME>Date: April 23, 2020 IntroductionI do GIS professionally and thought that the best thing I could do to help would be to do what's called [geocoding](https://en.wikipedia.org/wiki/Geocoding) to the Kijiji datasets that have been provided. Geocoding is the process where you convert the address of a physical location to a latitude/longitude coordinate pair. It was mentioned as being a stretch goal and I thought this was no better time to help the Calgary community. SummaryI intend to use OpenStreetMap's geocoder, Nominatim, to do the heavy lifting for me. It has a handy set of [APIs](https://nominatim.org/release-docs/develop/api/Search/) that make this kind of work straightforward. My plan to do this is to read the input `.csv` files, put them into dataframes, then iterate through the addresses with Python's `requests` library. I will create two additional columns in the dataframes to store the latitude and longitude values where each record's values will be written after they're geocoded.I'd like to have a quick look at some of the values that Nominatim returns to show that it works and to practice using `folium`, which is a neat library I've seen in a number of projects but never had the chance to use myself. ChallengesLooks like the main challenge in this project will be parsing each address into something that Nominatim can use. Hopefully the `location` values are properly separated by commas because that will make my life a lot easier.So, without further ado, let's get started!<jupyter_code>import requests
import pandas as pd
import folium
import re
# Read the csv
df = pd.read_csv('kijiji-page_calgary_5KfromCityCentre_longTerm_24032020.csv')
df.head(4)<jupyter_output><empty_output><jupyter_text>Things are already looking a little fishy with the `location` column. In the previous 4 records of the original data we have the following address patterns:* City, Province, Country, Postal Code (Without the space)* Appartment Number and Street Address, Province and Postal Code (With the space)* Appartment Number and Street Address, Postal Code (without the space), Appartment Number and Street Address again, Province* Appartment Number and Street Address, City, Province and Postal Code (With the space), CountryLet's take a longer look at the top 15 `location` values.<jupyter_code>df.location.values[:15]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05349122090649244, 'fraction_numerical': 0.00898325847284606, 'mean_word_length': 5.380208333333333, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '523419', 'license': 'MIT', 'n_tokens_mistral': 658, 'n_tokens_neox': 609, 'n_words': 347, 'path': 'kijiji-geocoding.ipynb', 'repo_name': 'roaur/d4g-micro-datathon-1', 'size': '2455'}