<jupyter_start><jupyter_text>Capstone Project: Segmenting & Clustering Neighborhood in Toronto Rohan Chakraborty<jupyter_code>from bs4 import BeautifulSoup
import requests
import csv
import pandas as pd
import json
from pandas.io.json import json_normalize
import numpy as np<jupyter_output><empty_output><jupyter_text>Part 1  Data Scraping<jupyter_code># send the GET request
data = requests.get('https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M').text

# parse data from the html into a beautifulsoup object
soup = BeautifulSoup(data, 'lxml')

# create three lists to store table data
postalCodeList = []
boroughList = []
neighborhoodList = []
# append the data into the respective lists
for row in soup.find('table').find_all('tr'):
    cells = row.find_all('td')
    if(len(cells) > 0):
        postalCodeList.append(cells[0].text.rstrip('\n'))
        boroughList.append(cells[1].text.rstrip('\n'))
        neighborhoodList.append(cells[2].text.rstrip('\n'))
# create a new DataFrame from the three lists
toronto_df = pd.DataFrame({"Postal Code": postalCodeList,
                           "Borough": boroughList,
                           "Neighborhood": neighborhoodList})

toronto_df.head()
#drop cells with Borough not assigned
toronto_dropped = toronto_df[toronto_df.Borough != 'Not assigned'].reset_index(drop=True)
toronto_dropped.head()
# group multiple neighborhoods having same postal code
toronto_group = toronto_dropped.groupby(['Postal Code', 'Borough'], as_index=False).agg(lambda x: ", ".join(x))
toronto_group['Neighborhood'] = toronto_group['Neighborhood'].str.replace('/', ',')
toronto_group.head()
# rows of dataframe
toronto_group.shape<jupyter_output><empty_output><jupyter_text>Part 2  Adding Lattitude and Longitude to the dataframe<jupyter_code>import types
from botocore.client import Config
import ibm_boto3

def __iter__(self): return 0

# @hidden_cell
# The following code accesses a file in your IBM Cloud Object Storage. It includes your credentials.
# You might want to remove those credentials before you share the notebook.
client_id = ibm_boto3.client(service_name='s3',
    ibm_api_key_id='enter ibm_api_key_id',
    ibm_auth_endpoint="enter ibm_auth_endpoint",
    config=Config(signature_version='oauth'),
    endpoint_url='https://s3.eu-geo.objectstorage.service.networklayer.com')

body = client_id.get_object(Bucket='bucket',Key='Geospatial_Coordinates.csv')['Body']
# add missing __iter__ method, so pandas accepts body as file-like object
if not hasattr(body, "__iter__"): body.__iter__ = types.MethodType( __iter__, body )

coordinates = pd.read_csv(body)
coordinates.head(5)<jupyter_output><empty_output><jupyter_text>Merge the two dataframes<jupyter_code>dataframe = pd.merge(toronto_group, coordinates, on='Postal Code')
dataframe.head()<jupyter_output><empty_output><jupyter_text>save these into csv files<jupyter_code>toronto_group.to_csv('toronto_postal_codes.csv', index = False)
dataframe.to_csv('final_toronto_geospatial.csv', index = False)<jupyter_output><empty_output><jupyter_text>Part 3<jupyter_code>print('The dataframe has {} boroughs'.format(
        len(dataframe['Borough'].unique())
    )
)<jupyter_output>The dataframe has 10 boroughs<jupyter_text>Use geopy library to get the latitude and longitude values of Toronto City<jupyter_code>from geopy.geocoders import Nominatim
import requests
import matplotlib.cm as cm
import matplotlib.colors as colors
from sklearn.cluster import KMeans
import folium

address = 'Toronto , Ontario'

geolocator = Nominatim(user_agent="toronto_explorer")
location = geolocator.geocode(address)
latitude = location.latitude
longitude = location.longitude
print('The geograpical coordinate of Toronto City are {}, {}.'.format(latitude, longitude))<jupyter_output>The geograpical coordinate of Toronto City are 43.6534817, -79.3839347.<jupyter_text>Create a map of Toronto with neighborhoods superimposed on top<jupyter_code># create map of Toronto using latitude and longitude values
map_toronto = folium.Map(location=[latitude, longitude], zoom_start=10)

# add markers to map
for lat, lng, borough, neighborhood in zip(dataframe['Latitude'], dataframe['Longitude'], dataframe['Borough'], dataframe['Neighborhood']):
    label = '{}, {}'.format(neighborhood, borough)
    label = folium.Popup(label, parse_html=True)
    folium.CircleMarker(
        [lat, lng],
        radius=5,
        popup=label,
        color='blue',
        fill=True,
        fill_color='#3186cc',
        fill_opacity=0.7,
        parse_html=False).add_to(map_toronto)  
    
map_toronto<jupyter_output><empty_output><jupyter_text>We segment and cluster only the neighborhoods in Scarborough<jupyter_code>scarborough_data = dataframe[dataframe['Borough'] == 'Scarborough'].reset_index(drop=True)
scarborough_data.head()
address = 'Scarborough, Toronto'

geolocator = Nominatim(user_agent="toronto_explorer")
location = geolocator.geocode(address)
latitude = location.latitude
longitude = location.longitude
print('The geograpical coordinate of Scarborough are {}, {}.'.format(latitude, longitude))
# create map of Scarborough using latitude and longitude values
map_scarborough = folium.Map(location=[latitude, longitude], zoom_start=10)

# add markers to map
for lat, lng, borough, neighborhood in zip(scarborough_data['Latitude'], scarborough_data['Longitude'], scarborough_data['Borough'], scarborough_data['Neighborhood']):
    label = '{}, {}'.format(neighborhood, borough)
    label = folium.Popup(label, parse_html=True)
    folium.CircleMarker(
        [lat, lng],
        radius=5,
        popup=label,
        color='blue',
        fill=True,
        fill_color='#3186cc',
        fill_opacity=0.7,
        parse_html=False).add_to(map_scarborough)  
    
map_scarborough<jupyter_output><empty_output><jupyter_text>Define Foursquare Credentials and Version<jupyter_code># @hidden_cell

CLIENT_ID = 'enter foursquare client id' # your Foursquare ID
CLIENT_SECRET = 'enter foursquare client secret' # your Foursquare Secret
VERSION = 'enter foursquare api version' # Foursquare API version

print('Your credentails:')
print('CLIENT_ID: ' + CLIENT_ID)
print('CLIENT_SECRET:' + CLIENT_SECRET)<jupyter_output><empty_output><jupyter_text>Let's explore the first neighborhood in our dataframe<jupyter_code>scarborough_data.loc[0, 'Neighborhood']<jupyter_output><empty_output><jupyter_text>Get the neighborhood's latitude and longitude values<jupyter_code>neighborhood_latitude = scarborough_data.loc[0, 'Latitude'] # neighborhood latitude value
neighborhood_longitude = scarborough_data.loc[0, 'Longitude'] # neighborhood longitude value

neighborhood_name = scarborough_data.loc[0, 'Neighborhood'] # neighborhood name

print('Latitude and longitude values of {} are {}, {}.'.format(neighborhood_name, 
                                                               neighborhood_latitude, 
                                                               neighborhood_longitude))<jupyter_output>Latitude and longitude values of Malvern , Rouge are 43.806686299999996, -79.19435340000001.<jupyter_text>Now, let's get the top 100 venues that are in Malvern , Rouge within a radius of 500 meters<jupyter_code>LIMIT = 100 
radius = 500 

url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}'.format(
    CLIENT_ID, 
    CLIENT_SECRET, 
    VERSION, 
    neighborhood_latitude, 
    neighborhood_longitude, 
    radius, 
    LIMIT)

results = requests.get(url).json()
results
# function that extracts the category of the venue
def get_category_type(row):
    try:
        categories_list = row['categories']
    except:
        categories_list = row['venue.categories']
        
    if len(categories_list) == 0:
        return None
    else:
        return categories_list[0]['name']<jupyter_output><empty_output><jupyter_text>Clean the json and structure it into a pandas dataframe<jupyter_code>venues = results['response']['groups'][0]['items']
    
nearby_venues = json_normalize(venues) # flatten JSON

# filter columns
filtered_columns = ['venue.name', 'venue.categories', 'venue.location.lat', 'venue.location.lng']
nearby_venues =nearby_venues.loc[:, filtered_columns]

# filter the category for each row
nearby_venues['venue.categories'] = nearby_venues.apply(get_category_type, axis=1)

# clean columns
nearby_venues.columns = [col.split(".")[-1] for col in nearby_venues.columns]

nearby_venues.head()
print('{} venues were returned by Foursquare.'.format(nearby_venues.shape[0]))<jupyter_output>1 venues were returned by Foursquare.<jupyter_text>Function to repeat the same process to all the neighborhoods in Scarborough<jupyter_code>def getNearbyVenues(names, latitudes, longitudes, radius=500):
    
    venues_list=[]
    for name, lat, lng in zip(names, latitudes, longitudes):
        print(name)
            
        # create the API request URL
        url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}'.format(
            CLIENT_ID, 
            CLIENT_SECRET, 
            VERSION, 
            lat, 
            lng, 
            radius, 
            LIMIT)
            
        # make the GET request
        results = requests.get(url).json()["response"]['groups'][0]['items']
        
        # return only relevant information for each nearby venue
        venues_list.append([(
            name, 
            lat, 
            lng, 
            v['venue']['name'], 
            v['venue']['location']['lat'], 
            v['venue']['location']['lng'],  
            v['venue']['categories'][0]['name']) for v in results])

    nearby_venues = pd.DataFrame([item for venue_list in venues_list for item in venue_list])
    nearby_venues.columns = ['Neighborhood', 
                  'Neighborhood Latitude', 
                  'Neighborhood Longitude', 
                  'Venue', 
                  'Venue Latitude', 
                  'Venue Longitude', 
                  'Venue Category']
    
    return(nearby_venues)
scarborough_venues = getNearbyVenues(names=scarborough_data['Neighborhood'],
                                   latitudes=scarborough_data['Latitude'],
                                   longitudes=scarborough_data['Longitude']
                                  )
print(scarborough_venues.shape)
scarborough_venues.head()<jupyter_output>(96, 7)<jupyter_text>No. of venues were returned for each neighborhood<jupyter_code>scarborough_venues.groupby('Neighborhood').count()<jupyter_output><empty_output><jupyter_text>Unique categories curated from all the returned venues<jupyter_code>print('There are {} uniques categories.'.format(len(scarborough_venues['Venue Category'].unique())))<jupyter_output>There are 58 uniques categories.<jupyter_text>Analyze Each Neighborhood<jupyter_code># one hot encoding
scarborough_onehot = pd.get_dummies(scarborough_venues[['Venue Category']], prefix="", prefix_sep="")

# add neighborhood column back to dataframe
scarborough_onehot['Neighborhood'] = scarborough_venues['Neighborhood'] 

# move neighborhood column to the first column
fixed_columns = [scarborough_onehot.columns[-1]] + list(scarborough_onehot.columns[:-1])
scarborough_onehot = scarborough_onehot[fixed_columns]

scarborough_onehot.head()
scarborough_onehot.shape<jupyter_output><empty_output><jupyter_text>Group rows by neighborhood and by taking the mean of the frequency of occurrence of each category<jupyter_code>scarborough_grouped = scarborough_onehot.groupby('Neighborhood').mean().reset_index()
scarborough_grouped
scarborough_grouped.shape<jupyter_output><empty_output><jupyter_text>Print each neighborhood along with the top 5 most common venues<jupyter_code>num_top_venues = 5

for hood in scarborough_grouped['Neighborhood']:
    print("----"+hood+"----")
    temp = scarborough_grouped[scarborough_grouped['Neighborhood'] == hood].T.reset_index()
    temp.columns = ['venue','freq']
    temp = temp.iloc[1:]
    temp['freq'] = temp['freq'].astype(float)
    temp = temp.round({'freq': 2})
    print(temp.sort_values('freq', ascending=False).reset_index(drop=True).head(num_top_venues))
    print('\n')<jupyter_output>----Agincourt----
                       venue  freq
0  Latin American Restaurant  0.25
1                     Lounge  0.25
2             Breakfast Spot  0.25
3               Skating Rink  0.25
4        American Restaurant  0.00


----Birch Cliff , Cliffside West----
                   venue  freq
0  General Entertainment  0.25
1           Skating Rink  0.25
2                   Café  0.25
3        College Stadium  0.25
4    American Restaurant  0.00


----Cedarbrae----
                  venue  freq
0      Hakka Restaurant  0.12
1  Caribbean Restaurant  0.12
2                Bakery  0.12
3                  Bank  0.12
4    Athletics & Sports  0.12


----Clarks Corners , <NAME>ter , Sullivan----
                 venue  freq
0          Pizza Place  0.14
1    Convenience Store  0.07
2   Chinese Restaurant  0.07
3             Pharmacy  0.07
4  Fried Chicken Joint  0.07


----Cliffside , Cliffcrest , Scarborough Village West----
                 venue  freq
0  American Restaurant   0.5
1  [...]<jupyter_text>Put this into a *pandas* dataframe<jupyter_code>#function to sort the venues in descending order

def return_most_common_venues(row, num_top_venues):
    row_categories = row.iloc[1:]
    row_categories_sorted = row_categories.sort_values(ascending=False)
    
    return row_categories_sorted.index.values[0:num_top_venues]
#create the new dataframe and display the top 10 venues for each neighborhood

num_top_venues = 10

indicators = ['st', 'nd', 'rd']

# create columns according to number of top venues
columns = ['Neighborhood']
for ind in np.arange(num_top_venues):
    try:
        columns.append('{}{} Most Common Venue'.format(ind+1, indicators[ind]))
    except:
        columns.append('{}th Most Common Venue'.format(ind+1))

# create a new dataframe
neighborhoods_venues_sorted = pd.DataFrame(columns=columns)
neighborhoods_venues_sorted['Neighborhood'] = scarborough_grouped['Neighborhood']

for ind in np.arange(scarborough_grouped.shape[0]):
    neighborhoods_venues_sorted.iloc[ind, 1:] = return_most_common_venues(scarborough_grouped.iloc[ind, :], num_top_venues)

neighborhoods_venues_sorted.head()<jupyter_output><empty_output><jupyter_text>Cluster Neighborhoods<jupyter_code>#Run k-means to cluster the neighborhood into 5 clusters

# set number of clusters
kclusters = 5

scarborough_grouped_clustering = scarborough_grouped.drop('Neighborhood', 1)

# run k-means clustering
kmeans = KMeans(n_clusters=kclusters, random_state=0).fit(scarborough_grouped_clustering)

# check cluster labels generated for each row in the dataframe
kmeans.labels_[0:10] 
#create a new dataframe that includes the cluster as well as the top 10 venues for each neighborhood

# add clustering labels
neighborhoods_venues_sorted.insert(0, 'Cluster Labels', kmeans.labels_)

scarborough_merged = scarborough_data

# merge toronto_grouped with toronto_data to add latitude/longitude for each neighborhood
scarborough_merged = scarborough_merged.join(neighborhoods_venues_sorted.set_index('Neighborhood'), on='Neighborhood')

scarborough_merged.head() # check the last columns!
#visualize the resulting clusters

# create map
map_clusters = folium.Map(location=[latitude, longitude], zoom_start=11)

# set color scheme for the clusters
x = np.arange(kclusters)
ys = [i + x + (i*x)**2 for i in range(kclusters)]
colors_array = cm.rainbow(np.linspace(0, 1, len(ys)))
rainbow = [colors.rgb2hex(i) for i in colors_array]

# add markers to the map
markers_colors = []
for lat, lon, poi, cluster in zip(scarborough_merged['Latitude'], scarborough_merged['Longitude'], scarborough_merged['Neighborhood'], scarborough_merged['Cluster Labels']):
    label = folium.Popup(str(poi) + ' Cluster ' + str(cluster), parse_html=True)
    folium.CircleMarker(
        [lat, lon],
        radius=5,
        popup=label,
        color=rainbow[int(cluster)],
        fill=True,
        fill_color=rainbow[int(cluster)],
        fill_opacity=0.7).add_to(map_clusters)
       
map_clusters<jupyter_output><empty_output><jupyter_text>Examine Clusters Examine each cluster and determine the discriminating venue categories that distinguish each cluster. Based on the defining categories, you can then assign a name to each cluster. I will leave this exercise to you.  Cluster 1<jupyter_code>scarborough_merged.loc[scarborough_merged['Cluster Labels'] == 0, scarborough_merged.columns[[1] + list(range(5, scarborough_merged.shape[1]))]]<jupyter_output><empty_output><jupyter_text>Cluster 2<jupyter_code>scarborough_merged.loc[scarborough_merged['Cluster Labels'] == 1, scarborough_merged.columns[[1] + list(range(5, scarborough_merged.shape[1]))]]<jupyter_output><empty_output><jupyter_text>Cluster 3<jupyter_code>scarborough_merged.loc[scarborough_merged['Cluster Labels'] == 2, scarborough_merged.columns[[1] + list(range(5, scarborough_merged.shape[1]))]]<jupyter_output><empty_output><jupyter_text>Cluster 4<jupyter_code>scarborough_merged.loc[scarborough_merged['Cluster Labels'] == 3, scarborough_merged.columns[[1] + list(range(5, scarborough_merged.shape[1]))]]<jupyter_output><empty_output><jupyter_text>Cluster 5<jupyter_code>scarborough_merged.loc[scarborough_merged['Cluster Labels'] == 4, scarborough_merged.columns[[1] + list(range(5, scarborough_merged.shape[1]))]]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11001257286547034, 'fraction_numerical': 0.015087438564407361, 'mean_word_length': 4.436160298229264, 'pattern_counts': {'":': 3, '<': 103, '<?xml version=': 0, '>': 104, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '126073', 'license': 'MIT', 'n_tokens_mistral': 5750, 'n_tokens_neox': 5404, 'n_words': 1216, 'path': 'Segmenting and Clustering.ipynb', 'repo_name': 'rohanchakraborty/Capstone-Project', 'size': '17502'}