<jupyter_start><jupyter_text>Segmenting and Clustering Neighborhoods in TorontoIBM Applied Data Science Capstone: Case Study ProjectDate: Feb 2021 Introduction:For this data project, I segmented and clustered the neighborhoods in Toronto, Canada, based on postal code and district information found from an open data source. Then, I generated a folium map to visualize the communities and how they cluster together. Data : The data I used is a Wikipedia page with the postal code and district information for [Toronto](https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M). Considering the latitude and longitude values were not included in the data, I used a separate [CSV file](https://cocl.us/Geospatial_data) that contains the geographical coordinates of each postal code.  Methodology: To prepare for this analysis, I used the beautiful soup package to scrape the Wikipedia page, wrangle, clean, then transform the data into a pandas data frame. After converting the data for each neighborhood, I added the geospatial CSV file into my data frame. Finally, I utilized the Foursquare location data to get the geographical coordinates of Toronto, Canada and displayed my data in a folium map. I decided to work with boroughs that only contain the word "Toronto." Then I added a marker for each neighborhood within the map.<jupyter_code>#Importing Libraries
import requests 
from bs4 import BeautifulSoup
import pandas as pd 
import numpy as np 
import random 

!pip install geopy
from geopy.geocoders import Nominatim 
from pandas.io.json import json_normalize

! pip install folium==0.5.0
import folium 

print('Folium installed')
print('Libraries imported.')<jupyter_output>Requirement already satisfied: geopy in /opt/conda/envs/Python-3.7-main/lib/python3.7/site-packages (2.1.0)
Requirement already satisfied: geographiclib<2,>=1.49 in /opt/conda/envs/Python-3.7-main/lib/python3.7/site-packages (from geopy) (1.50)
Requirement already satisfied: folium==0.5.0 in /opt/conda/envs/Python-3.7-main/lib/python3.7/site-packages (0.5.0)
Requirement already satisfied: jinja2 in /opt/conda/envs/Python-3.7-main/lib/python3.7/site-packages (from folium==0.5.0) (2.11.3)
Requirement already satisfied: requests in /opt/conda/envs/Python-3.7-main/lib/python3.7/site-packages (from folium==0.5.0) (2.25.1)
Requirement already satisfied: branca in /opt/conda/envs/Python-3.7-main/lib/python3.7/site-packages (from folium==0.5.0) (0.4.2)
Requirement already satisfied: six in /opt/conda/envs/Python-3.7-main/lib/python3.7/site-packages (from folium==0.5.0) (1.15.0)
Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/envs/Python-3.7-main/lib/python3.7/site-packages (from [...]<jupyter_text>Loading and Transforming Data<jupyter_code>req = requests.get("https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M")
soup = BeautifulSoup(req.content, 'lxml')
table = soup.find_all('table')[0]
df = pd.read_html(str(table))
neighbourhood = pd.DataFrame(df[0])
neighbourhood.head(12)<jupyter_output><empty_output><jupyter_text>Cleaning Data<jupyter_code>neighbourhood = neighbourhood[neighbourhood != 'Not assigned']
neighbourhood = neighbourhood.dropna()
neighbourhood.reset_index(drop=True)
neighbourhood = neighbourhood.reset_index(drop=True)
neighbourhood
neighbourhood.shape<jupyter_output><empty_output><jupyter_text>Adding Latitude and Longitude to dataframe<jupyter_code>geo_data = pd.read_csv('https://cocl.us/Geospatial_data')
geo_data.head()
geo_nd = neighbourhood.merge(geo_data, on = 'Postal Code', how = 'left')
geo_nd.head(12)
toronto_df = geo_nd[geo_nd['Borough'].astype(str).str.contains('Toronto')]<jupyter_output><empty_output><jupyter_text>Generating folim map to visualize neighborhoods<jupyter_code>address = 'Toronto, Canada'

geolocator = Nominatim(user_agent="ca_explorer")
location = geolocator.geocode(address)
latitude = location.latitude
longitude = location.longitude
print('The geograpical coordinate of Canada are {}, {}.'.format(latitude, longitude))<jupyter_output>The geograpical coordinate of Canada are 43.6534817, -79.3839347.<jupyter_text>Clustered Neighborhood Map<jupyter_code># create map of Toronto using latitude and longitude values
map_toronto = folium.Map(location=[latitude,longitude], zoom_start=14)

# adding markers to the map
for lat, lng, borough, neighborhood in zip(toronto_df['Latitude'], toronto_df['Longitude'],toronto_df['Neighbourhood'], toronto_df['Borough']):
    label = '{}, {}'.format(neighborhood, borough)
    label = folium.Popup(label, parse_html=True)
    folium.CircleMarker(
        [lat, lng],
        radius=5,
        popup=label,
        color='blue',
        fill=True,
        fill_color='#3186cc',
        fill_opacity=0.7,
        parse_html=False).add_to(map_toronto)  
    
map_toronto<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10799584631360332, 'fraction_numerical': 0.024922118380062305, 'mean_word_length': 5.642758620689655, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 25, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '661760', 'license': 'MIT', 'n_tokens_mistral': 1639, 'n_tokens_neox': 1549, 'n_words': 412, 'path': 'Segmenting and Clustering Neighborhoods in Toronto (1).ipynb', 'repo_name': 'brimoe/Portfolio', 'size': '4815'}