<jupyter_start><jupyter_text>Import libraries<jupyter_code>import urllib.request
from bs4 import BeautifulSoup
import requests
from geopy.geocoders import Nominatim # convert an address into latitude and longitude values

import numpy as np
import pandas as pd
from sklearn.cluster import KMeans

import folium
import matplotlib.cm as cm
import matplotlib.colors as colors<jupyter_output><empty_output><jupyter_text>1. Prepare and preprocess dataset<jupyter_code># use Wikipedia URL from 29th March 2019
url = 'https://en.wikipedia.org/w/index.php?title=List_of_postal_codes_of_Canada:_M&oldid=890001695'
page = urllib.request.urlopen(url)

soup = BeautifulSoup(page, "lxml")

# Capture the table 
table = soup.find('table', class_='wikitable')
# Populate lists and preprocess data
PostalCodes = []
Boroughs = []
Neighborhoods = []

for row in table.find_all('tr'):
    cells=row.findAll('td')
    
    if(cells != []):
      postal_code  = cells[0].get_text()
      borough      = cells[1].get_text()
      neighborhood = cells[2].get_text()
      # Only process the cells that have an assigned borough. Ignore cells with a borough that is Not assigned.
      if(borough != 'Not assigned'): 
        PostalCodes.append(postal_code)
        Boroughs.append(borough)
        neighborhood = neighborhood.replace('\n','')
        # If a cell has a borough but a Not assigned neighborhood, then the neighborhood will be the same as the borough.
        if(neighborhood != 'Not assigned'):          
          Neighborhoods.append(neighborhood)
        else:  
          Neighborhoods.append(borough)
        
df=pd.DataFrame()
df['Postal Code']=PostalCodes
df['Borough']=Boroughs
df['Neighborhood']=Neighborhoods
df['Neighborhood'] = df.groupby(['Postal Code'])['Neighborhood'].transform(lambda x: ','.join(x))

df = df[['Postal Code','Borough','Neighborhood']].drop_duplicates()
df<jupyter_output><empty_output><jupyter_text>``` Tried to use geocoder, but it is unreliable:!pip install geocoderimport geocoder  import geocoderLatitudes = []Longitudes = [] initialize your variable to Nonelat_lng_coords = Nonefor postal_code in df['Postal Code']:  loop until you get the coordinates   while(lat_lng_coords is None):   g = geocoder.google('{}, Toronto, Ontario'.format(postal_code))   lat_lng_coords = g.latlng  latitude = lat_lng_coords[0]  longitude = lat_lng_coords[1]  Latitudes.append(latitude)  Longitudes.append(longitude)```<jupyter_code># Tried to use geocoder library but did not work as expected.
# Get data from http://cocl.us/Geospatial_data. curl must be used with these flags because of the redirections of the link shortener:  
!curl -O -J -L https://cocl.us/Geospatial_data 
latlong_df = pd.read_csv("Geospatial_Coordinates.csv") 
latlong_df.head()
df = pd.merge(df, latlong_df, on='Postal Code')
df<jupyter_output><empty_output><jupyter_text>2. Analyze dataFirst, a map of Toronto with neighborhoods marks will be created. Then the neighborhoods (more specifically, pairs of latitude and longitude) will be clustered according to their most common venues.   Map of Toronto with borough and neighborhoods markers<jupyter_code>address = 'Toronto, ON'

geolocator = Nominatim(user_agent="coursera_explorer")
location = geolocator.geocode(address)
latitude = location.latitude
longitude = location.longitude
print('The geograpical coordinate of Toronto are {}, {}.'.format(latitude, longitude))

# create map of New York using latitude and longitude values
map_toronto = folium.Map(location=[latitude, longitude], zoom_start=10)

# add markers to map
for lat, lng, borough, neighborhood in zip(df['Latitude'], df['Longitude'], df['Borough'], df['Neighborhood']):
    label = '{}  ({})'.format(borough, neighborhood)
    label = folium.Popup(label, parse_html=True)
    folium.CircleMarker(
        [lat, lng],
        radius=5,
        popup=label,
        color='blue',
        fill=True,
        fill_color='#3186cc',
        fill_opacity=0.7,
        parse_html=False).add_to(map_toronto)  
    
map_toronto<jupyter_output>The geograpical coordinate of Toronto are 43.653963, -79.387207.<jupyter_text>Find popular venues in each neighborhood<jupyter_code>CLIENT_ID = '' # your Foursquare ID 
CLIENT_SECRET = '' # your Foursquare Secret
VERSION = '20180605' # Foursquare API version
LIMIT = 100

def getNearbyVenues(names, latitudes, longitudes, radius=500):
    
    venues_list=[]
    for name, lat, lng in zip(names, latitudes, longitudes):
        print(name)
            
        # create the API request URL
        url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}'.format(
            CLIENT_ID, 
            CLIENT_SECRET, 
            VERSION, 
            lat, 
            lng, 
            radius, 
            LIMIT)
            
        # make the GET request
        results = requests.get(url).json()["response"]['groups'][0]['items']
        
        # return only relevant information for each nearby venue
        venues_list.append([(
            name, 
            lat, 
            lng, 
            v['venue']['name'], 
            v['venue']['location']['lat'], 
            v['venue']['location']['lng'],  
            v['venue']['categories'][0]['name']) for v in results])

    nearby_venues = pd.DataFrame([item for venue_list in venues_list for item in venue_list])
    nearby_venues.columns = ['Neighborhood', 
                  'Neighborhood Latitude', 
                  'Neighborhood Longitude', 
                  'Venue', 
                  'Venue Latitude', 
                  'Venue Longitude', 
                  'Venue Category']
    
    return(nearby_venues)
# sometimes the foursquare API fails. Recommended to try this method twice:
toronto_venues = getNearbyVenues(names=df['Neighborhood'],
                                 latitudes=df['Latitude'],
                                 longitudes=df['Longitude']
                                 )
print('There are {} uniques categories.'.format(len(toronto_venues['Venue Category'].unique())))
# one hot encoding
toronto_onehot = pd.get_dummies(toronto_venues[['Venue Category']], prefix="", prefix_sep="")

# add neighborhood column back to dataframe
toronto_onehot['Neighborhood'] = toronto_venues['Neighborhood'] 

# move neighborhood column to the first column
fixed_columns = [toronto_onehot.columns[-1]] + list(toronto_onehot.columns[:-1])
toronto_onehot = toronto_onehot[fixed_columns]

toronto_onehot.head()
toronto_grouped = toronto_onehot.groupby('Neighborhood').mean().reset_index()
toronto_grouped.head()<jupyter_output><empty_output><jupyter_text>Find most common venues in each neighborhood<jupyter_code>def return_most_common_venues(row, num_top_venues):
    row_categories = row.iloc[1:]
    row_categories_sorted = row_categories.sort_values(ascending=False)
    
    return row_categories_sorted.index.values[0:num_top_venues]
num_top_venues = 10

indicators = ['st', 'nd', 'rd']

# create columns according to number of top venues
columns = ['Neighborhood']
for ind in np.arange(num_top_venues):
    try:
        columns.append('{}{} Most Common Venue'.format(ind+1, indicators[ind]))
    except:
        columns.append('{}th Most Common Venue'.format(ind+1))

# create a new dataframe
neighborhoods_venues_sorted = pd.DataFrame(columns=columns)
neighborhoods_venues_sorted['Neighborhood'] = toronto_grouped['Neighborhood']

for ind in np.arange(toronto_grouped.shape[0]):
    neighborhoods_venues_sorted.iloc[ind, 1:] = return_most_common_venues(toronto_grouped.iloc[ind, :], num_top_venues)

neighborhoods_venues_sorted.head()<jupyter_output><empty_output><jupyter_text>Cluster neighborhoods ny common venues<jupyter_code># set number of clusters
kclusters = 5

toronto_grouped_clustering = toronto_grouped.drop('Neighborhood', 1)

# run k-means clustering
kmeans = KMeans(n_clusters=kclusters, init='k-means++', random_state=0).fit(toronto_grouped_clustering)

# add clustering labels
neighborhoods_venues_sorted.insert(0, 'Cluster Labels', kmeans.labels_)

toronto_merged = df

# merge toronto_grouped with toronto_data to add latitude/longitude for each neighborhood
toronto_merged = toronto_merged.join(neighborhoods_venues_sorted.set_index('Neighborhood'), on='Neighborhood')

# drop NaN in the data 
toronto_merged = toronto_merged.dropna()

toronto_merged.head()<jupyter_output><empty_output><jupyter_text>Map with neighborhoods labeled by cluster<jupyter_code># create map
map_clusters = folium.Map(location=[latitude, longitude], zoom_start=11)

# set color scheme for the clusters
x = np.arange(kclusters)
ys = [i + x + (i*x)**2 for i in range(kclusters)]
colors_array = cm.rainbow(np.linspace(0, 1, len(ys)))
rainbow = [colors.rgb2hex(i) for i in colors_array]

# add markers to the map
markers_colors = []
for lat, lon, poi, cluster in zip(toronto_merged['Latitude'], toronto_merged['Longitude'], toronto_merged['Neighborhood'], toronto_merged['Cluster Labels']):
    label = folium.Popup(str(poi) + ' Cluster ' + str(cluster), parse_html=True)
    folium.CircleMarker(
        [lat, lon],
        radius=5,
        popup=label,
        color=rainbow[int(cluster)-1],
        fill=True,
        fill_color=rainbow[int(cluster)-1],
        fill_opacity=0.7).add_to(map_clusters)
       
map_clusters<jupyter_output><empty_output><jupyter_text>Clusters analysis Cluster 1 is made by the neighborhoods which most common venues are restaurants (Fast Food, Pizza Place, etc.)<jupyter_code>toronto_merged.loc[toronto_merged['Cluster Labels'] == 0, toronto_merged.columns[[2] + list(range(5, toronto_merged.shape[1]))]]<jupyter_output><empty_output><jupyter_text>The most common venues in neighborhoods of Cluster 2 are parks.<jupyter_code>toronto_merged.loc[toronto_merged['Cluster Labels'] == 1, toronto_merged.columns[[2] + list(range(5, toronto_merged.shape[1]))]]<jupyter_output><empty_output><jupyter_text>All neighborhoods of Cluster 3 has a baseball field.<jupyter_code>toronto_merged.loc[toronto_merged['Cluster Labels'] == 2, toronto_merged.columns[[2] + list(range(5, toronto_merged.shape[1]))]]<jupyter_output><empty_output><jupyter_text>Cluster 4 is made by the most of neighboorhoods and is the most heterogeneous.<jupyter_code>toronto_merged.loc[toronto_merged['Cluster Labels'] == 3, toronto_merged.columns[[2] + list(range(5, toronto_merged.shape[1]))]]<jupyter_output><empty_output><jupyter_text>There are only 2 neighborhoods in Cluster 5. Their distribution of most common venues are exactly the same.<jupyter_code>toronto_merged.loc[toronto_merged['Cluster Labels'] == 4, toronto_merged.columns[[2] + list(range(5, toronto_merged.shape[1]))]]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11608980355472404, 'fraction_numerical': 0.010851262862488307, 'mean_word_length': 4.446255731023943, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 52, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '654519', 'license': 'MIT', 'n_tokens_mistral': 3597, 'n_tokens_neox': 3378, 'n_words': 774, 'path': 'Segmenting_and_Clustering_Neighborhoods_in_Toronto.ipynb', 'repo_name': 'htadashi/Coursera_Capstone', 'size': '10690'}