<jupyter_start><jupyter_text>Peer-graded Assignment: Segmenting and Clustering Neighborhoods in Toronto PART 1 1. Start by creating a new Notebook for this assignment. 2. Scrape data from wikipedia. 3. To create the above dataframe:- The dataframe will consist of three columns: PostalCode, Borough, and Neighborhood- Only process the cells that have an assigned borough. Ignore cells with a borough that is **Not assigned.- More than one neighborhood can exist in one postal code area. For example, in the table on the Wikipedia page, you will notice that **M5A** is listed twice and has two neighborhoods: **Harbourfront** and **Regent Park**. These two rows will be combined into one row with the neighborhoods separated with a comma as shown in **row 11**  in the above table.- If a cell has a borough but a **Not assigned** neighborhood, then the neighborhood will be the same as the borough.- Clean your Notebook and add Markdown cells to explain your work and any assumptions you are making.- In the last cell of your notebook, use the **.shape** method to print the number of rows of your dataframe. 4. Submit a link to your Notebook on your Github repository.<jupyter_code>#Load all the necessary libraries
import requests
import pandas as pd
import numpy as np
from pandas.io.json import json_normalize
from bs4 import BeautifulSoup<jupyter_output><empty_output><jupyter_text>Scrape data from the wikipedia pagehttps://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M<jupyter_code>url = "https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M"
html = requests.get(url).text
soup = BeautifulSoup(html, 'html5lib')<jupyter_output><empty_output><jupyter_text>Clean the data We will ignore all the "Not assigned" Borough If two boroughs share the same postal code, they will be combined into the same row, separated by a comma<jupyter_code>table_contents=[]
table=soup.find('table')
for row in table.findAll('td'):
    cell = {}
    if row.span.text=='Not assigned': # Ignore 'not assigned'
        pass
    else:
        cell['PostalCode'] = row.p.text[:3]
        cell['Borough'] = (row.span.text).split('(')[0]
        cell['Neighborhood'] = (((((row.span.text).split('(')[1]).strip(')')).replace(' /',',')).replace(')',' ')).strip(' ')
        table_contents.append(cell)

# print(table_contents)
df=pd.DataFrame(table_contents)
df['Borough']=df['Borough'].replace({'Downtown TorontoStn A PO Boxes25 The Esplanade':'Downtown Toronto Stn A',
                                             'East TorontoBusiness reply mail Processing Centre969 Eastern':'East Toronto Business',
                                             'EtobicokeNorthwest':'Etobicoke Northwest','East YorkEast Toronto':'East York/East Toronto',
                                             'MississaugaCanada Post Gateway Processing Centre':'Mississauga'})
df.head()<jupyter_output><empty_output><jupyter_text>Check for unassigned Borough<jupyter_code>#check for unassigned borough
print(str(df.loc[df['Borough'] == 'Not assigned'].shape[0]) + " unassigned borough.")<jupyter_output>0 unassigned borough.<jupyter_text>Check for shape of the dataframe<jupyter_code>print(f"Shape of the Dataframe: {df.shape}")<jupyter_output>Shape of the Dataframe: (103, 3)<jupyter_text>Export toronto data into a csv file, to be used in Part 2<jupyter_code>df.to_csv(r'toronto_df.csv', index=False)<jupyter_output><empty_output><jupyter_text>PART 2 We need to find latitude and longitude coordinates of each neighborhood. Merge the toronto data in part 1 with their corresponding lat, long in part 2We are able to solve this problem by using Geocoder package, or with the provided GeoSpatial Dataset  Geocoder method was provided with the assignment<jupyter_code>#import geocoder # import geocoder
#
# initialize your variable to None
#lat_lng_coords = None
#
# loop until you get the coordinates
#while(lat_lng_coords is None):
#  g = geocoder.google('{}, Toronto, Ontario'.format(postal_code))
#  lat_lng_coords = g.latlng
#
#latitude = lat_lng_coords[0]
#longitude = lat_lng_coords[1]<jupyter_output><empty_output><jupyter_text>We will load the data from the provided csv file, GeoSpatial Dataset<jupyter_code>import os, types
from botocore.client import Config
import ibm_boto3

def __iter__(self): return 0

# @hidden_cell
# The following code accesses a file in your IBM Cloud Object Storage. It includes your credentials.
# You might want to remove those credentials before you share the notebook.

if os.environ.get('RUNTIME_ENV_LOCATION_TYPE') == 'external':
    endpoint_94a137416d5148a598130aa554926487 = 'https://s3.us.cloud-object-storage.appdomain.cloud'
else:
    endpoint_94a137416d5148a598130aa554926487 = 'https://s3.private.us.cloud-object-storage.appdomain.cloud'

client_94a137416d5148a598130aa554926487 = ibm_boto3.client(service_name='s3',
    ibm_api_key_id='<KEY>',
    ibm_auth_endpoint="https://iam.cloud.ibm.com/oidc/token",
    config=Config(signature_version='oauth'),
    endpoint_url=endpoint_94a137416d5148a598130aa554926487)

body = client_94a137416d5148a598130aa554926487.get_object(Bucket='ibmcapstoneprojectweek3-donotdelete-pr-3y8kzl8iq7vpgc',Key='Geospatial_Coordinates.csv')['Body']
# add missing __iter__ method, so pandas accepts body as file-like object
if not hasattr(body, "__iter__"): body.__iter__ = types.MethodType( __iter__, body )

geo_data = pd.read_csv(body)
geo_data.head()

df<jupyter_output><empty_output><jupyter_text>Combine the toront data from part 1 with their respective longitude and latitude<jupyter_code>#rename Postal Code to PostalCode
geo_data.rename(columns={"Postal Code" : "PostalCode"}, inplace=True)
#join the two tables, merging them on the PostalCode column
df = df.join(geo_data.set_index("PostalCode"), on="PostalCode")<jupyter_output><empty_output><jupyter_text>The merged dataframe<jupyter_code>df.shape<jupyter_output><empty_output><jupyter_text>PART 3 Explore and cluster the neighborhoods in Toronto. You can decide to work with only boroughs that contain the word Toronto and then replicate the same analysis we did to the New York City data. It is up to you.  Just make sure:- to add enough Markdown cells to explain what you decided to do and to report any observations you make.- to generate maps to visualize your neighborhoods and how they cluster together.<jupyter_code>#import the necessary libraries
!conda install -c conda-forge geopy --yes
from geopy.geocoders import Nominatim
import matplotlib.cm as cm
import matplotlib.colors as colors
from sklearn.cluster import KMeans
import json
from pandas.io.json import json_normalize
!conda install -c conda-forge folium=0.5.0 --yes
import folium
print('Libraries imported.')<jupyter_output>Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

Libraries imported.<jupyter_text>Let's see how many boroughs and neighborhoods are in here<jupyter_code>print('The dataframe has {} boroughs and {} neighborhoods.'.format(
        len(df['Borough'].unique()),
        df.shape[0]
    )
)<jupyter_output>The dataframe has 15 boroughs and 103 neighborhoods.<jupyter_text>Use geopy library to get the latitude and longtitude of Toronto<jupyter_code>address = "Toronto, Canada"

geolocator = Nominatim(user_agent="trt-explorer")
location = geolocator.geocode(address)
latitude = location.latitude
longitude = location.longitude
print('The geographical coordinate of Toronto are {}, {}.'.format(latitude, longitude))<jupyter_output>The geographical coordinate of Toronto are 43.6534817, -79.3839347.<jupyter_text>Create a map of Toronto with neighborhoods superimposed on top<jupyter_code>#create map of TOronto using latitude and longitude values
map_toronto = folium.Map(location=[latitude, longitude], zoom_start=10)

#add markers to map
for lat, lng, borough, neighborhood in zip(df['Latitude'], df['Longitude'], df['Borough'], df['Neighborhood']):
    label = "{}, {}".format(neighborhood, borough)
    label = folium.Popup(label, parse_html=True)
    folium.CircleMarker(
        [lat, lng],
        radius=5,
        popup=label,
        color="orange",
        fill=True,
        fill_color="#3186cc",
        fill_opacity=0.7,
        parse_html=False).add_to(map_toronto)
    
map_toronto<jupyter_output><empty_output><jupyter_text>Utilize the Foursquare API to explore the neighborhoods and segment them Define Foursquare Credentials and Version<jupyter_code>VERSION = '20180605' # Foursquare API version
LIMIT = 100 # A default Foursquare API limit value


#CLIENT_ID = 'GP5BBVIGOHTEBZGGZVP1E50A0V1WALVRS05W5IAKONWOIQJT' # your Foursquare ID
#CLIENT_SECRET = '<KEY>' # your Foursquare Secret
CLIENT_ID = 'XVY0YGK3DX5QGHMN2TGSK2EWA55P3JNPIVC5QVW5SGIGUI2L' 
CLIENT_SECRET = '<KEY>'

print('Your credentails:')
print('CLIENT_ID: ' + CLIENT_ID)
print('CLIENT_SECRET:' + CLIENT_SECRET)<jupyter_output>Your credentails:
CLIENT_ID: XVY0YGK3DX5QGHMN2TGSK2EWA55P3JNPIVC5QVW5SGIGUI2L
CLIENT_SECRET:T53Z3HT4W5DVALRIPBK2DPD4NFOCISMUTMNBLNW13KEJTAIJ<jupyter_text>Let's explore the first neighborhood in our dataframeGet the name first<jupyter_code>print(f'The first neighborhood in our dataframe is {df.loc[0, "Neighborhood"]}')<jupyter_output>The first neighborhood in our dataframe is Parkwoods<jupyter_text>Get the neighborhood's latitude and longitude values<jupyter_code>neighborhood_lat = df.loc[0, "Latitude"]
neighborhood_lng = df.loc[0, "Longitude"]

neighborhood_name = df.loc[0, "Neighborhood"]

print("Latitude and longitude values of {} are {}, {}".format(neighborhood_name,
                                                             neighborhood_lat,
                                                             neighborhood_lng))<jupyter_output>Latitude and longitude values of Parkwoods are 43.7532586, -79.3296565<jupyter_text>Now let's get the top 100 venues that are in Regent Park within a radius of 500 metersFirst, let's create the GET request URL<jupyter_code>LIMIT = 100
radius = 500

url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}'.format(
    CLIENT_ID, 
    CLIENT_SECRET, 
    VERSION, 
    lat, 
    lng, 
    radius, 
    LIMIT)<jupyter_output><empty_output><jupyter_text>Send the GET request and examine the results<jupyter_code>results = requests.get(url).json()
results<jupyter_output><empty_output><jupyter_text>We'll use get_category_type function from the Foursquare lab<jupyter_code>#function that extracts the category of the venue
def get_category_type(row):
    try:
        categories_list = row['categories']
    except:
        categories_list = row['venue.categories']
        
    if len(categories_list) == 0:
        return None
    else:
        return categories_list[0]['name']<jupyter_output><empty_output><jupyter_text>Clean the json and structure it into a pandas dataframe<jupyter_code>venues = results['response']['groups'][0]['items']

nearby_venues = json_normalize(venues) # flatten JSON

#filter columns
filtered_columns = ['venue.name', 'venue.categories', 'venue.location.lat', 'venue.location.lng']
nearby_venues = nearby_venues.loc[:, filtered_columns]

#filter the category for each row
nearby_venues['venue.categories'] = nearby_venues.apply(get_category_type, axis=1)

#clean columns
nearby_venues.columns = [col.split(".")[-1] for col in nearby_venues.columns]

nearby_venues.head()
print('{} venues were returned by Foursquare.'.format(nearby_venues.shape[0]))<jupyter_output>3 venues were returned by Foursquare.<jupyter_text>Let's create a function to repeat the same process to all the neighborhoods in Toronto<jupyter_code>def getNearbyVenues(names, latitudes, longitudes, radius=500):
    
    venues_list=[]
    for name, lat, lng in zip(names, latitudes, longitudes):
        print(name)
            
        # create the API request URL
        url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}'.format(
            CLIENT_ID, 
            CLIENT_SECRET, 
            VERSION, 
            lat, 
            lng, 
            radius, 
            100)
            
        # make the GET request
        results = requests.get(url).json()["response"]['groups'][0]['items']
        
        # return only relevant information for each nearby venue
        venues_list.append([(
            name, 
            lat, 
            lng, 
            v['venue']['name'], 
            v['venue']['location']['lat'], 
            v['venue']['location']['lng'],  
            v['venue']['categories'][0]['name']) for v in results])

    nearby_venues = pd.DataFrame([item for venue_list in venues_list for item in venue_list])
    nearby_venues.columns = ['Neighborhood', 
                  'Neighborhood Latitude', 
                  'Neighborhood Longitude', 
                  'Venue', 
                  'Venue Latitude', 
                  'Venue Longitude', 
                  'Venue Category']
    
    return(nearby_venues)<jupyter_output><empty_output><jupyter_text>We'll run the above function on each neighborhood and create a new dataframe called toronto_venues<jupyter_code>toronto_venues = getNearbyVenues(df['Neighborhood'],
                                df['Latitude'],
                                df['Longitude']
                                )<jupyter_output>Parkwoods
Victoria Village
Regent Park, Harbourfront
Lawrence Manor, Lawrence Heights
Ontario Provincial Government
Islington Avenue
Malvern, Rouge
Don Mills North
Parkview Hill, Woodbine Gardens
Garden District, Ryerson
Glencairn
West Deane Park, Princess Gardens, Martin Grove, Islington, Cloverdale
Rouge Hill, Port Union, Highland Creek
Don Mills South
Woodbine Heights
St. James Town
Humewood-Cedarvale
Eringate, Bloordale Gardens, Old Burnhamthorpe, Markland Wood
Guildwood, Morningside, West Hill
The Beaches
Berczy Park
Caledonia-Fairbanks
Woburn
Leaside
Central Bay Street
Christie
Cedarbrae
Hillcrest Village
Bathurst Manor, Wilson Heights, Downsview North
Thorncliffe Park
Richmond, Adelaide, King
Dufferin, Dovercourt Village
Scarborough Village
Fairview, Henry Farm, Oriole
Northwood Park, York University
The Danforth  East
Harbourfront East, Union Station, Toronto Islands
Little Portugal, Trinity
Kennedy Park, Ionview, East Birchmount Park
Bayview Village
Downsview East
The Danforth[...]<jupyter_text>Check the size of the reslting dataframe<jupyter_code>print(toronto_venues.shape)
toronto_venues<jupyter_output>(309, 7)<jupyter_text>Let's check how many venues were returned for each neighborhood<jupyter_code>toronto_venues[['Venue Category']]<jupyter_output><empty_output><jupyter_text>Fidn out how many unique categories can be curated from all the returned venues<jupyter_code>print('There are {} uniques categories.'.format(len(toronto_venues['Venue Category'].unique())))
toronto_venues.groupby(['Neighborhood']).count()<jupyter_output>There are 3 uniques categories.<jupyter_text>Analyze each neighborhood<jupyter_code># one hot encoding
toronto_onehot = pd.get_dummies(toronto_venues[['Venue Category']], prefix="", prefix_sep="")

# add neighborhood column back to dataframe
toronto_onehot['Neighborhood'] = toronto_venues['Neighborhood'] 

# move neighborhood column to the first column
fixed_columns = [toronto_onehot.columns[-1]] + list(toronto_onehot.columns[:-1])
toronto_onehot = toronto_onehot[fixed_columns]

toronto_onehot.head()
toronto_onehot.shape<jupyter_output><empty_output><jupyter_text>Group rows by neighborhood and by taking the mean of the frequency of occurence of each category<jupyter_code>toronto_grouped = toronto_onehot.groupby('Neighborhood').mean().reset_index()
toronto_grouped
toronto_grouped.shape<jupyter_output><empty_output><jupyter_text>Let's print each neighborhood along with the top 5 most common venues<jupyter_code>num_top_venues = 5

for hood in toronto_grouped['Neighborhood']:
    print("====" + hood + "====")
    temp = toronto_grouped[toronto_grouped['Neighborhood'] == hood].T.reset_index()
    temp.columns = ['venue', 'freq']
    temp = temp.iloc[1:]
    temp['freq'] = temp['freq'].astype(float)
    temp = temp.round({'freq': 2})
    print(temp.sort_values('freq', ascending=False).reset_index(drop=True).head(num_top_venues))
    print('\n')<jupyter_output>====Agincourt====
                       venue  freq
0  Latin American Restaurant  0.25
1             Breakfast Spot  0.25
2                     Lounge  0.25
3               Skating Rink  0.25
4          Mobile Phone Shop  0.00


====Alderwood, Long Branch====
            venue  freq
0     Pizza Place  0.25
1     Coffee Shop  0.12
2  Sandwich Place  0.12
3             Pub  0.12
4      Playground  0.12


====<NAME>, <NAME>, Downsview North====
                 venue  freq
0                 Bank  0.09
1          Coffee Shop  0.09
2    Mobile Phone Shop  0.05
3     Sushi Restaurant  0.05
4  Fried Chicken Joint  0.05


====Bayview Village====
                 venue  freq
0                 Café  0.25
1  Japanese Restaurant  0.25
2                 Bank  0.25
3   Chinese Restaurant  0.25
4  Monument / Landmark  0.00


====Bedford Park, Lawrence Manor East====
                     venue  freq
0           Sandwich Place  0.08
1       Italian Restaurant  0.08
2              Coffe[...]<jupyter_text>Let's put that into a pandas dataframe let's write a function to sort the venues in descending order<jupyter_code>def return_most_common_venues(row, num_top_venues):
    row_categories = row.iloc[1:]
    row_categories_sorted = row_categories.sort_values(ascending=False)
    
    return row_categories_sorted.index.values[0:num_top_venues]<jupyter_output><empty_output><jupyter_text>Let's create the new dataframe and display the top 10 venues for each neighorhood<jupyter_code>num_top_venues = 10

indicators = ['st', 'nd', 'rd']

# create columns according to number of top venues
columns = ['Neighborhood']
for ind in np.arange(num_top_venues):
    try:
        columns.append('{}{} Most Common Venue'.format(ind+1, indicators[ind]))
    except:
        columns.append('{}th Most Common Venue'.format(ind+1))

# create a new dataframe
neighborhoods_venues_sorted = pd.DataFrame(columns=columns)
neighborhoods_venues_sorted['Neighborhood'] = toronto_grouped['Neighborhood']

for ind in np.arange(toronto_grouped.shape[0]):
    neighborhoods_venues_sorted.iloc[ind, 1:] = return_most_common_venues(toronto_grouped.iloc[ind, :], num_top_venues)

neighborhoods_venues_sorted.head()<jupyter_output><empty_output><jupyter_text>Run k-means to cluster the neighborhood into 5 clusters<jupyter_code>kclusters = 5

toronto_grouped_clustering = toronto_grouped.drop('Neighborhood', 1)

#run k-means clustering
kmeans = KMeans(n_clusters=kclusters, random_state=0).fit(toronto_grouped_clustering)

#check cluster Labels generated for each row in the dataframe
kmeans.labels_[0:10]<jupyter_output><empty_output><jupyter_text>Create a new dataframe that includes the cluster as well as the top 10 venues for each neighborhood<jupyter_code>#add clustering labels
neighborhoods_venues_sorted['Cluster Labels'] = pd.Series([], dtype=int)
neighborhoods_venues_sorted['Cluster Labels'] = list(kmeans.labels_)
#neighborhoods_venues_sorted.drop('Cluster Labels', axis=1)

toronto_merged = df

toronto_merged = toronto_merged.join(neighborhoods_venues_sorted.set_index('Neighborhood'), on='Neighborhood')

toronto_merged.head()
toronto_merged.dropna(inplace=True)
toronto_merged.shape<jupyter_output><empty_output><jupyter_text>Visuzlie the clusters<jupyter_code># create map
map_clusters = folium.Map(location=[latitude, longitude], zoom_start=11)

# set color scheme for the clusters
x = np.arange(kclusters)
ys = [i + x + (i*x)**2 for i in range(kclusters)]
colors_array = cm.rainbow(np.linspace(0, 1, len(ys)))
rainbow = [colors.rgb2hex(i) for i in colors_array]

# add markers to the map
markers_colors = []
for lat, lon, poi, cluster in zip(toronto_merged['Latitude'], toronto_merged['Longitude'], toronto_merged['Neighborhood'], toronto_merged['Cluster Labels']):
    label = folium.Popup(str(poi) + ' Cluster ' + str(cluster), parse_html=True)
    folium.CircleMarker(
        [lat, lon],
        radius=5,
        popup=label,
        color=rainbow[int(cluster)-1],
        fill=True,
        fill_color=rainbow[int(cluster)-1],
        fill_opacity=0.7).add_to(map_clusters)
       
map_clusters<jupyter_output><empty_output><jupyter_text>Let's examine the 5 clusters Cluster 1<jupyter_code>toronto_merged.loc[toronto_merged['Cluster Labels'] == 0, toronto_merged.columns[[1] + list(range(5, toronto_merged.shape[1]))]]<jupyter_output><empty_output><jupyter_text>Cluster 2<jupyter_code>toronto_merged.loc[toronto_merged['Cluster Labels'] == 1, toronto_merged.columns[[1] + list(range(5, toronto_merged.shape[1]))]]<jupyter_output><empty_output><jupyter_text>Cluster 3<jupyter_code>toronto_merged.loc[toronto_merged['Cluster Labels'] == 2, toronto_merged.columns[[1] + list(range(5, toronto_merged.shape[1]))]]<jupyter_output><empty_output><jupyter_text>Cluster 4<jupyter_code>toronto_merged.loc[toronto_merged['Cluster Labels'] == 3, toronto_merged.columns[[1] + list(range(5, toronto_merged.shape[1]))]]<jupyter_output><empty_output><jupyter_text>Cluster 5<jupyter_code>toronto_merged.loc[toronto_merged['Cluster Labels'] == 4, toronto_merged.columns[[1] + list(range(5, toronto_merged.shape[1]))]]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10269493019156732, 'fraction_numerical': 0.021939793125840715, 'mean_word_length': 4.676671932596103, 'pattern_counts': {'":': 0, '<': 149, '<?xml version=': 0, '>': 149, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '39', 'dir': 'jupyter-structured-clean-dedup', 'id': '78295', 'license': 'MIT', 'n_tokens_mistral': 7342, 'n_tokens_neox': 6800, 'n_words': 1764, 'path': 'Week3/Capstone_week3_project_complete.ipynb', 'repo_name': 'jjaw/Coursera_Capstone', 'size': '21700'}