<jupyter_start><jupyter_text>Segmenting and Clustering Neighborhoods in Toronto  Objective In this assignment, we're required to explore, segment, and cluster the neighborhoods in the city of Toronto. We'll be scraping the Wikipedia page and wrangle the data, clean it, and then read it into a pandas dataframe so that it is in a structured format like the New York dataset.Once the data is in a structured format, we'll replicate the analysis that we did to the New York City dataset to explore and cluster the neighborhoods in the city of Toronto.  Setting up the pre-requisites Before we start work, let's install and import the required libraries and setup options that will help in later stages.<jupyter_code># Uncomment to install required packages

#!pip install beautifulsoup4
#!pip install lxml
#!pip install geopy
#!pip install geocoder
#!pip install folium

print('Installations done.')
# Importing necessary libraries

import numpy as np # library to handle data in a vectorized manner
import pandas as pd # library for data analsysis

import requests # library to handle requests
import json # library to handle JSON files
from pandas.io.json import json_normalize # tranform JSON file into a pandas dataframe

# Matplotlib and associated plotting modules
import matplotlib.cm as cm
import matplotlib.colors as colors

import folium # map rendering library

# K-means for clustering 
from sklearn.cluster import KMeans

# For error handling, progress tracking and API throtling
from random import randint
from time import sleep
import sys
import logging

# For web scrapping and location retrieval
from bs4 import BeautifulSoup
import geocoder as gc # import geocoder
from geopy.geocoders import Nominatim # we could've used geocoder (above) as well

# JSON and HTML formatting
import pprint

print('Libraries imported.')
# Adjust options
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

# Incase we need to reset
#pd.reset_option('display.max_colwidth')
#pd.reset_option('display.max_rows')

# So that more data can be printed on a single line
pp = pprint.PrettyPrinter(width=120)

print('Options adjusted.')<jupyter_output>Options adjusted.<jupyter_text>Part 1 **Requirement:** _Use pandas, or the BeautifulSoup package, or any other way you are comfortable with to transform the data in the table on the Wikipedia page into the pandas dataframe._ Let's download the postal codes of Toronto city through the Wikipdia page, scrape using **BeautifulSoup** and create the DataFrame. _Ensure that all requirements from point 2 & 3 of the assignment are met here._<jupyter_code># Create request object, download page and read the desired table
url = requests.get('https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M').text
page = BeautifulSoup(url,'lxml')
table = page.find ('table', class_='wikitable sortable')

# Load the headers from the table
header_list=[]
for header in table.findAll ('th'):
    header_list.append(header.text.replace('\n', '').title().replace(' ',''))

# Load the table content row-by-row
output_rows = []
for table_row in table.findAll('tr'):
    columns = table_row.findAll('td')
    output_row = []
    for column in columns:

        # Replace any unwanted characters
        output_row.append(column.text.replace('\n', '').replace(' / ',', '))

    # Ignore rows with unassigned boroughs
    if output_row and not output_row[1]=='Not assigned': 
        output_rows.append(output_row)

# Insert table into the DataFrame and naming columns
df = pd.DataFrame(output_rows)
df.columns = header_list

# Ensuring neighboorhoods are combined against the PostalCode in a single row. The requirement was already met in the source page.
print ('\'PostalCode\' column has unique values:', df['PostalCode'].nunique()==df['PostalCode'].size)

# Checking if every borough has a neighborhood, incase we need to assing a borough to "Not assigned" neighborhood.  The requirement was already met in the source page.
print ('\'Neighborhood\' column doen\'t have a blank value:', df[df['Neighborhood']==''].size==0)

# Print number of rows of the dataframe
df.shape<jupyter_output>'PostalCode' column has unique values: True
'Neighborhood' column doen't have a blank value: True<jupyter_text>Part 2 **Requirement:** _Use the Geocoder package or the CSV file to retrieve the location information._ Let's try to use the **Geocoder** package with **Arcgis** provider for the location coordinates. Finally, load the data in the DataFrame.<jupyter_code># initialize the variable to None
lat_lng_coords = None

# Preparing lists for holding the latitude and longitude information
lat_list=[]
lng_list=[]

# This will show progress
sys.stdout.write("Location retrieval progress: %d%%   \r" % (0))
sys.stdout.flush()

# Setting up logging, incase things go wrong
logging.basicConfig(level=logging.ERROR,
                    format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s', 
                    filename='errors.log', 
                    filemode='w')

for ind in df.index:
    # loop until you get the coordinates
    attempt = 1
    while(lat_lng_coords is None):
        try:
            # Attempting to retrieve location against Postal Code
            g = gc.arcgis('{}, Toronto, Ontario'.format(df['PostalCode'][ind]))
            lat_lng_coords = g.latlng
        except:
            if attempt <=5: # Limiting to 5 attempts
                logging.error('Failed attempt #{} for {}, Toronto, Ontario'.format(attempt, df['PostalCode'][ind]))
                attempt = attempt + 1
                
                # Trying to avoid blocking by the server due to quick calls
                sleep(0.01/randint(1,10000))
                pass
            else:
                raise

    # Loadting the coordinates in respective lists
    lat_list.append(round(lat_lng_coords[0], 6))
    lng_list.append(round(lat_lng_coords[1], 6))
    lat_lng_coords = None
    
    # Updating progress
    sys.stdout.write("Location retrieval progress: %d%%   \r" % (ind/df['PostalCode'].size*100))
    sys.stdout.flush()
    
    # Trying to avoid blocking by the server due to quick calls
    sleep(0.01/randint(1,10000))

sys.stdout.write("Location retrieval progress: %d%%   \r" % (100))

# Assigning data to DataFrame
df['Latitude']=lat_list
df['Longitude']=lng_list

df<jupyter_output>Location retrieval progress: 100%<jupyter_text>Backup plan Saving the DataFrame as CSV file for later retreival, incase we get blocked by the provider due to exceeding the limit etc.<jupyter_code>#df.to_csv('Toronto_Neighborhoods.csv', header=True, index=False)
#df = pd.read_csv('Toronto_Neighborhoods.csv')<jupyter_output><empty_output><jupyter_text>Part 3 **Requirement:** _Explore and cluster the neighborhoods in Toronto_  Explore data and visualize neighboorhoods Retreive geographical coordinates of Toronto city for the map using _trn_explorer_ as the user_agent.<jupyter_code># Setting up the address
address = 'Toronto, Ontario'

# Initializing and retreivng the coordinates
geolocator = Nominatim(user_agent="trn_explorer")
location = geolocator.geocode(address)
latitude = location.latitude
longitude = location.longitude

print('The geograpical coordinate of Toronto city are {}, {}.'.format(latitude, longitude))<jupyter_output>The geograpical coordinate of Toronto city are 43.6534817, -79.3839347.<jupyter_text>Let's select only those boroughs that contain the word Toronto e.g., Downtown Toronto etc. and slice the DataFrame.<jupyter_code># Selecting boroughs that contain the word Toronto
df_toronto = df[df['Borough'].str.contains('Toronto', regex=False)].reset_index(drop=True)
df_toronto.head()<jupyter_output><empty_output><jupyter_text>Create a map of Toronto city with selected neighborhoods superimposed on top using **Folium** visualization library.<jupyter_code># create map of Toronto using latitude and longitude values
map_toronto = folium.Map(location=[latitude, longitude], zoom_start=12)

# add markers to map
for lat, lng, label in zip(df_toronto['Latitude'], df_toronto['Longitude'], df_toronto['Neighborhood']):
    label = folium.Popup(label, parse_html=True)
    folium.CircleMarker(
        [lat, lng],
        radius=5,
        popup=label,
        color='blue',
        fill=True,
        fill_color='#3186cc',
        fill_opacity=0.7,
        parse_html=False).add_to(map_toronto)  
    
map_toronto<jupyter_output><empty_output><jupyter_text>Explore selected boroughs Let's use the Foursquare API to explore the neighborhoods and segment them.<jupyter_code># Define Foursquare Credentials, Version and Limits.
CLIENT_ID = '3BCVAXKBYMT2WBM10IJUSYVZUVSU15GUZLHLNS4NKPFSBLEM' # your Foursquare ID
CLIENT_SECRET = '<KEY>' # your Foursquare Secret
VERSION = '20180605' # Foursquare API version

LIMIT = 100 # limit of number of venues returned by Foursquare API
radius = 500 # define radius

print('Your credentails:')
print('CLIENT_ID: ' + CLIENT_ID)
print('CLIENT_SECRET:' + CLIENT_SECRET)<jupyter_output>Your credentails:
CLIENT_ID: 3BCVAXKBYMT2WBM10IJUSYVZUVSU15GUZLHLNS4NKPFSBLEM
CLIENT_SECRET:MDG201J4RR3FWOQ0RGFPBUE1TDQFAO0DH0KXL5MHJ3DA0J0T<jupyter_text>Let's borrow the function from New York city lab to explore all the neighborhoods in selected boroughs.<jupyter_code>def getNearbyVenues(names, latitudes, longitudes, radius=500):
    
    venues_list=[]
    for name, lat, lng in zip(names, latitudes, longitudes):
        print(name)
            
        # create the API request URL
        url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}'.format(
            CLIENT_ID, 
            CLIENT_SECRET, 
            VERSION, 
            lat, 
            lng, 
            radius, 
            LIMIT)
            
        # make the GET request
        results = requests.get(url).json()["response"]['groups'][0]['items']
        
        # return only relevant information for each nearby venue
        venues_list.append([(
            name, 
            lat, 
            lng, 
            v['venue']['name'], 
            v['venue']['location']['lat'], 
            v['venue']['location']['lng'],  
            v['venue']['categories'][0]['name']) for v in results])

    nearby_venues = pd.DataFrame([item for venue_list in venues_list for item in venue_list])
    nearby_venues.columns = ['Neighborhood', 
                  'Neighborhood Latitude', 
                  'Neighborhood Longitude', 
                  'Venue', 
                  'Venue Latitude', 
                  'Venue Longitude', 
                  'Venue Category']
    
    return(nearby_venues)<jupyter_output><empty_output><jupyter_text>Let's explore all near-by venues and save them in new DataFrame.<jupyter_code>toronto_venues = getNearbyVenues(names=df_toronto['Neighborhood'],
                                   latitudes=df_toronto['Latitude'],
                                   longitudes=df_toronto['Longitude']
                                  )<jupyter_output>Regent Park, Harbourfront
Queen's Park, Ontario Provincial Government
Garden District, Ryerson
St. James Town
The Beaches
Berczy Park
Central Bay Street
Christie
Richmond, Adelaide, King
Dufferin, Dovercourt Village
Harbourfront East, Union Station, Toronto Islands
Little Portugal, Trinity
The Danforth West, Riverdale
Toronto Dominion Centre, Design Exchange
Brockton, Parkdale Village, Exhibition Place
India Bazaar, The Beaches West
Commerce Court, Victoria Hotel
Studio District
Lawrence Park
Roselawn
Davisville North
Forest Hill North & West
High Park, The Junction South
North Toronto West
The Annex, North Midtown, Yorkville
Parkdale, Roncesvalles
Davisville
University of Toronto, Harbord
Runnymede, Swansea
Moore Park, Summerhill East
Kensington Market, Chinatown, Grange Park
Summerhill West, Rathnelly, South Hill, Forest Hill SE, Deer Park
CN Tower, King and Spadina, Railway Lands, Harbourfront West, Bathurst Quay, South Niagara, Island airport
Rosedale
Stn A PO Boxes
St. James Town,[...]<jupyter_text>Let's check the size of the resulting dataframe and peek in to the contents.<jupyter_code>print(toronto_venues.shape)
toronto_venues.head()<jupyter_output>(1595, 7)<jupyter_text>And number of venues  returned for each neighborhood...<jupyter_code>toronto_venues.groupby('Neighborhood').count()<jupyter_output><empty_output><jupyter_text>How many unique categories can be curated from all the returned venues?<jupyter_code>print('There are {} uniques categories.'.format(len(toronto_venues['Venue Category'].unique())))<jupyter_output>There are 219 uniques categories.<jupyter_text>Analyze each neighborhood Let's prepare data for the clustering.<jupyter_code># One hot encoding
toronto_onehot = pd.get_dummies(toronto_venues[['Venue Category']], prefix="", prefix_sep="")

# Add neighborhood column back to dataframe
toronto_onehot['Neighborhood'] = toronto_venues['Neighborhood'] 

# Move neighborhood column as the first column
col='Neighborhood'
temp = toronto_onehot[col]
toronto_onehot.drop(labels=[col], axis=1,inplace = True)
toronto_onehot.insert(0, col, temp)

toronto_onehot.head()<jupyter_output><empty_output><jupyter_text>Let's explore the size of the DataFrame.<jupyter_code>toronto_onehot.shape<jupyter_output><empty_output><jupyter_text>Let's group rows by neighborhood and by taking the mean of the frequency of occurrence of each category and create a new DataFrame from it.<jupyter_code>toronto_grouped = toronto_onehot.groupby('Neighborhood').mean().reset_index()
toronto_grouped<jupyter_output><empty_output><jupyter_text>Let's confirm the size of the resulting DataFrame.<jupyter_code>toronto_grouped.shape<jupyter_output><empty_output><jupyter_text>Let's print each neighborhood along with the top 5 most common venues<jupyter_code>num_top_venues = 5

for hood in toronto_grouped['Neighborhood']:
    print("----"+hood+"----")
    temp = toronto_grouped[toronto_grouped['Neighborhood'] == hood].T.reset_index()
    temp.columns = ['venue','freq']
    temp = temp.iloc[1:]
    temp['freq'] = temp['freq'].astype(float)
    temp = temp.round({'freq': 2})
    print(temp.sort_values('freq', ascending=False).reset_index(drop=True).head(num_top_venues))
    print('\n')<jupyter_output>----Berczy Park----
                venue  freq
0         Coffee Shop  0.09
1        Cocktail Bar  0.04
2  Seafood Restaurant  0.04
3              Bakery  0.03
4               Hotel  0.03


----Brockton, Parkdale Village, Exhibition Place----
         venue  freq
0  Coffee Shop  0.09
1         Café  0.07
2    Gift Shop  0.04
3  Pizza Place  0.04
4        Diner  0.04


----Business reply mail Processing CentrE----
                 venue  freq
0          Coffee Shop  0.07
1                Hotel  0.05
2  Japanese Restaurant  0.04
3                 Café  0.04
4           Restaurant  0.03


----CN Tower, King and Spadina, Railway Lands, Harbourfront West, Bathurst Quay, South Niagara, Island airport----
               venue  freq
0        Coffee Shop  0.08
1         Restaurant  0.06
2               Café  0.06
3  French Restaurant  0.05
4               Park  0.05


----Central Bay Street----
             venue  freq
0      Coffee Shop  0.19
1            Plaza  0.04
2   Clothing Store  0.04
3[...]<jupyter_text>Let's borrow the function from the course lab to sort the venues in descending order.<jupyter_code>def return_most_common_venues(row, num_top_venues):
    row_categories = row.iloc[1:]
    row_categories_sorted = row_categories.sort_values(ascending=False)
    
    return row_categories_sorted.index.values[0:num_top_venues]<jupyter_output><empty_output><jupyter_text>Now let's create the new dataframe and display the top 10 venues for each neighborhood.<jupyter_code>num_top_venues = 10

indicators = ['st', 'nd', 'rd']

# create columns according to number of top venues
columns = ['Neighborhood']
for ind in np.arange(num_top_venues):
    try:
        columns.append('{}{} Most Common Venue'.format(ind+1, indicators[ind]))
    except:
        columns.append('{}th Most Common Venue'.format(ind+1))

# create a new dataframe
neighborhoods_venues_sorted = pd.DataFrame(columns=columns)
neighborhoods_venues_sorted['Neighborhood'] = toronto_grouped['Neighborhood']

for ind in np.arange(toronto_grouped.shape[0]):
    neighborhoods_venues_sorted.iloc[ind, 1:] = return_most_common_venues(toronto_grouped.iloc[ind, :], num_top_venues)

neighborhoods_venues_sorted.head()<jupyter_output><empty_output><jupyter_text>Cluster the neighborhoods Run k-means to cluster the neighborhood into **7 clusters**. _This number is chosen for the best result._<jupyter_code># set number of clusters
kclusters = 7

toronto_grouped_clustering = toronto_grouped.drop('Neighborhood', 1)

# run k-means clustering
kmeans = KMeans(init = "k-means++", n_clusters=kclusters, random_state=0, n_init = 3).fit(toronto_grouped_clustering)

# check cluster labels generated for each row in the dataframe
kmeans.labels_[0:10]

pd.DataFrame(kmeans.labels_).groupby(0)[0].count()<jupyter_output><empty_output><jupyter_text>Let's create a new dataframe that includes the cluster as well as the top 10 venues for each neighborhood. Using **'right'** option, as we might not get venue information for all Postal Codes. This will result in _NaN_ error in later stages.<jupyter_code># add clustering labels
neighborhoods_venues_sorted.insert(0, 'Cluster Labels', kmeans.labels_)

toronto_merged = df_toronto

# merge toronto_grouped with toronto_data to add latitude/longitude for each neighborhood
toronto_merged = toronto_merged.join(neighborhoods_venues_sorted.set_index('Neighborhood'), on='Neighborhood', how='right')

toronto_merged.head() # check the last columns!<jupyter_output><empty_output><jupyter_text>Let's visualize the resulting clusters.<jupyter_code># create map
map_clusters = folium.Map(location=[latitude, longitude], zoom_start=12)

# set color scheme for the clusters
x = np.arange(kclusters)
ys = [i + x + (i*x)**2 for i in range(kclusters)]
colors_array = cm.rainbow(np.linspace(0, 1, len(ys)))
rainbow = [colors.rgb2hex(i) for i in colors_array]

# add markers to the map
markers_colors = []
for lat, lon, poi, cluster in zip(toronto_merged['Latitude'], toronto_merged['Longitude'], toronto_merged['Neighborhood'], toronto_merged['Cluster Labels']):
    label = folium.Popup(str(poi) + ' Cluster ' + str(cluster), parse_html=True)
    folium.CircleMarker(
        [lat, lon],
        radius=5,
        popup=label,
        color=rainbow[int(cluster)-1],
        fill=True,
        fill_color=rainbow[int(cluster)-1],
        fill_opacity=0.7).add_to(map_clusters)
       
map_clusters<jupyter_output><empty_output><jupyter_text>Examine Clusters Looks like the people in cluster 0 and 3 love to hang-out but 0 are on the a little reserved side. People in cluster 2 seem to be health concious and athletics oriented. Cluster 5 seem to inhabited by large sized families with kids.The cluster 1, 4 and 6 seem to be outliers.<jupyter_code>toronto_merged.loc[toronto_merged['Cluster Labels'] == 0, toronto_merged.columns[[1,2] + list(range(6, toronto_merged.shape[1]))]]
toronto_merged.loc[toronto_merged['Cluster Labels'] == 2, toronto_merged.columns[[1,2] + list(range(6, toronto_merged.shape[1]))]]
toronto_merged.loc[toronto_merged['Cluster Labels'] == 3, toronto_merged.columns[[1,2] + list(range(6, toronto_merged.shape[1]))]]
toronto_merged.loc[toronto_merged['Cluster Labels'] == 5, toronto_merged.columns[[1,2] + list(range(6, toronto_merged.shape[1]))]]
toronto_merged.loc[toronto_merged['Cluster Labels'] == 1, toronto_merged.columns[[1,2] + list(range(6, toronto_merged.shape[1]))]]
toronto_merged.loc[toronto_merged['Cluster Labels'] == 4, toronto_merged.columns[[1,2] + list(range(6, toronto_merged.shape[1]))]]
toronto_merged.loc[toronto_merged['Cluster Labels'] == 6, toronto_merged.columns[[1,2] + list(range(6, toronto_merged.shape[1]))]]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09949325171842859, 'fraction_numerical': 0.015503487030254377, 'mean_word_length': 4.267441860465116, 'pattern_counts': {'":': 0, '<': 90, '<?xml version=': 0, '>': 89, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '418315', 'license': 'MIT', 'n_tokens_mistral': 6501, 'n_tokens_neox': 6072, 'n_words': 1822, 'path': 'Segmenting_and_Clustering_Neighborhoods_in_Toronto.ipynb', 'repo_name': 'githubfezan/Coursera_Capstone', 'size': '19974'}