<jupyter_start><jupyter_text>Applied Data Science Capstone - Final Project  This notebook holds code for the Applied Data Science Capstone course final project. ( Part of the IBM Data Science Professional Certification )  Where to Open your Pizza Place? - With the Power of Clustering If you are someone in Stockholm County looking for an excellent location to open a pizza place of your own, this could help you.How about finding a group of similar neighbourhoods in Stockholm with pizza places as the most common venue type for most neighbourhoods?Neighbourhoods with well-established pizza place businesses should be good for your new business, too, only if there are not too many similar businesses already in the neighbourhood. That is why this solution gives you not one neighbourhood but also find similar neighbourhoods. So if you have too many well-established and flourishing pizza place businesses in a neighbourhood, you can go for a similar neighbourhood (based on your other specific requirements), which does not have too many of such joint but does have a potential for profits for a new joint if you open one.Confusing, was it? Let me explain this to you through steps. With the result in hand, the idea clarifies further.  Step 1: Import all the required libraries. Before you start creating models to solve any problem, you need to gather, cleanse and most importantly, explore data. You use many utility functions, classes, and induction models with which you train your data. Let us import all the libraries required for our analysis.<jupyter_code># library for data manipulations and analyses
import pandas as pd

# customise row/column display aspects
pd.set_option('display.max_rows', None)
pd.set_option('display.max_columns', None)

# library for similar manipulation making use of vectors
import numpy as np

# library for HTTP requests
import requests

# library for converting address into latitude and longitude values
from geopy.geocoders import Nominatim

# library for map rendering
!pip install folium
import folium

# scipy's cdist for determining optimal K in K-Means clustering
from scipy.spatial.distance import cdist

# sklearn's KMeans for clustering
from sklearn.cluster import KMeans

# library for colors on maps
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import matplotlib.colors as colors<jupyter_output>Requirement already satisfied: folium in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (0.12.1)
Requirement already satisfied: jinja2>=2.9 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from folium) (3.0.0)
Requirement already satisfied: requests in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from folium) (2.25.1)
Requirement already satisfied: branca>=0.3.0 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from folium) (0.4.2)
Requirement already satisfied: numpy in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from folium) (1.19.2)
Requirement already satisfied: MarkupSafe>=2.0.0rc2 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from jinja2>=2.9->folium) (2.0.1)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from requests->folium) (2021.5.30)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/envs/Python-3.8-m[...]<jupyter_text>Note: We are using ```Folium``` for map visualisations in this code, but for some reason, ```Folium``` maps are not visible on the GitHub .ipynb files. Nevertheless, we have placed map image files in this repository separately.  Step 2: Gather data for analysis. To fetch the venue information from the Foursquare API, we need the geocoordinate information of the Stockholm neighbourhoods. Once we have this information, we can use the latitudes and longitudes to find out the venues in each neighbourhood.For this data, we use the data present in the ```SE.zip``` file present at ```http://download.geonames.org/export/zip/```. This file consists of the Stockholm County postal code level latitude and longitude information and other information such as district name, municipal name and municipal code.If you download and unzip the ```SE.zip``` file, you will find that the information we need is present in the text file named ```SE.txt```. This file has been uploaded to the same repository as this notebook (```SE_geo_data.txt```) and is being used directly for convenience.<jupyter_code># data file URL from GitHub
data_URL = 'https://raw.githubusercontent.com/guptasaumya/coursera-capstone-final/master/SE_geo_data.txt'

# load data into pandas data frame
postal_geocoordinate_data = pd.read_csv(data_URL,
                                        # data file is tab seperated
                                        delimiter="\t",
                                        # file as no column headers
                                        header=None,
                                        # read only the required information
                                        usecols=[2, 3, 9, 10],
                                        names=["Neighbourhood", "County", "Latitude", "Longitude"])

postal_geocoordinate_data.head()<jupyter_output><empty_output><jupyter_text>We now have a couple of pre-processing to be done on this data. Since we are dealing in Stockholm County, we remove all the postal codes belonging to counties other than Stockholm. Then we drop the ```County``` column since we have no use left of it.<jupyter_code># remove data of other counties
stockholm_data = postal_geocoordinate_data[postal_geocoordinate_data.County == "Stockholm"].drop("County", axis=1)

# remove column not needed
stockholm_data = stockholm_data[["Neighbourhood", "Latitude", "Longitude"]]

stockholm_data.head()<jupyter_output><empty_output><jupyter_text>We see more than one row for each district because of multiple postal codes within the district. Remember, this is postal codes data. However, we find that the geocoordinate information for all postal codes within a district is the same, indicating a lack of exact data. It is the case with most of the districts but not all. Hence to keep consistency, we remove the duplicates using the ```Neighbourhood``` subset. It results in one postal code per district of Stockholm County with its geocoordinate information.<jupyter_code># remove duplicate rows
stockholm_data.drop_duplicates(subset = ['Neighbourhood'], inplace=True)<jupyter_output><empty_output><jupyter_text>Then we make consistent the spelling of the municipalities - Järfälla and Norrtälje in the data.<jupyter_code># correct naming inconsistency
stockholm_data = stockholm_data[~stockholm_data["Neighbourhood"].isin(["Norrtalje", "Järfalla"])]<jupyter_output><empty_output><jupyter_text>We end up with geocoordinate data for 121 districts of Stockholm County.<jupyter_code># check shape (row represent number of districts in Stockholm county)
print(stockholm_data.shape)

stockholm_data.head()<jupyter_output>(121, 3)<jupyter_text>Step 3: Visualise the neighbourhoods/postal codes coordinates on the Stockholm County map. We will cluster these 121 districts based on the frequency of various types of venues within them. We start by first placing each of the districts on the map of Stockholm County using ```Folium```.<jupyter_code># use geopy library to get latitude and longitude values of city of Stockholm
address = "Stockholm, Sweden"

geolocator = Nominatim(user_agent="stockholm_explorer")
location = geolocator.geocode(address)
latitude = location.latitude
longitude = location.longitude

# create map of Stockholm using latitude and longitude values
map_stockholm = folium.Map(location=[latitude, longitude], zoom_start=8)

# add markers to map
for lat, lng, Neighbourhood in zip(stockholm_data["Latitude"],
                                  stockholm_data["Longitude"],
                                  stockholm_data["Neighbourhood"]):
    label = '{}'.format(Neighbourhood)
    label = folium.Popup(label, parse_html=True)
    folium.CircleMarker(
        [lat, lng],
        radius=5,
        popup=label,
        color='blue',
        fill=True,
        fill_color='#3186cc',
        fill_opacity=0.7,
        parse_html=False).add_to(map_stockholm)

map_stockholm<jupyter_output><empty_output><jupyter_text>Note: For this map result, check out https://github.com/guptasaumya/coursera-capstone-final/blob/master/map_Stockholm.png in this repository. Now that we have the information of the geocoordinates of the Stockholm County districts, we can use it to get the venues data using the Foursquare API. We are trying to get the information on the most common venue categories for the districts and use this information for clustering.  Step 4: Define Foursquare API usage credentials and version. In order to use the Foursquare API, we define some API query parameters. The following cell consists of client credentials and hence have been marked hidden by the owner.<jupyter_code># The code was removed by Watson Studio for sharing.<jupyter_output><empty_output><jupyter_text>We also define the ```VERSION``` and ```LIMIT``` parameters as follows. The former defines the API version we want to access (by date), and the latter defines the number of results to return.<jupyter_code># define Foursquare API version
VERSION = '20210816'

# define Foursquare API limit value
LIMIT = 100<jupyter_output><empty_output><jupyter_text>Step 5: Retrieve all venues for the districts within 500 meters radius. We now define a function to retrieve nearby venues for each of these districts. The function will return multiple rows corresponding to each district.<jupyter_code># define function gets nearby venues for each district
def getNearbyVenues(names, latitudes, longitudes, radius=500):

    # 'venues_list' stores venues data for all postal codes in list of lists format
    venues_list = []
    
    for name, lat, lng in zip(names, latitudes, longitudes):
        
        print(name)

        # create API request URL
        url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}'.format(
            CLIENT_ID,
            CLIENT_SECRET,
            VERSION,
            lat,
            lng,
            radius,
            LIMIT)

        # send GET requests and examine results
        response = requests.get(url).json()["response"]['groups'][0]['items']

        # append only relevant information for each nearby venue
        venues_list.append([(
            name,
            lat,
            lng,
            v['venue']['name'],
            v['venue']['location']['lat'],
            v['venue']['location']['lng'],
            v['venue']['categories'][0]['name']) for v in response])

    # create data frame from list of lists containing PostalCode data
    nearby_venues = pd.DataFrame([item for venue_list in venues_list
                                  for item in venue_list])
    
    # rename columns for uniform column name format
    nearby_venues.columns = ['Neighbourhood',
                             'NeighbourhoodLatitude',
                             'NeighbourhoodLongitude',
                             'Venue',
                             'VenueLatitude',
                             'VenueLongitude',
                             'VenueCategory']

    # return resulting data frame
    return(nearby_venues)<jupyter_output><empty_output><jupyter_text>We also create a utility function to sort the venues retrieved using the Foursquare API.<jupyter_code># define function to sort venues in descending order
def return_most_common_venues(row, num_top_venues):
    row_categories = row.iloc[1:]
    row_categories_sorted = row_categories.sort_values(ascending=False)

    return row_categories_sorted.index.values[0:num_top_venues]<jupyter_output><empty_output><jupyter_text>Now we call the function we created, passing our data to it.<jupyter_code># call function to get nearby venues for each neighbourhood
stockholm_venues = getNearbyVenues(stockholm_data['Neighbourhood'],
                                   stockholm_data['Latitude'],
                                   stockholm_data['Longitude'],
                                   # finding neraby venues within 500 meters radius
                                   500)

stockholm_venues.head()<jupyter_output>Vallentuna
Brottby
Ingmarsö
Åkersberga
Ljusterö
Österskär
Gällnöby
Skälvik
Nämdö
Stavsnäs
Runmarö
Sandhamn
Djurhamn
Stavsudda
Möja
Gustavsberg
Ingarö
Värmdö
Järfälla
Ekerö
Drottningholm
Munsö
Adelsö
Stenhamra
Färentuna
Svartsjö
Skå
Skärholmen
Huddinge
Kungens Kurva
Segeltorp
Skogås
Trångsund
Vårby
Norsborg
Tullinge
Tumba
Grödinge
Uttran
Rönninge
Dalarö
Ornö
Utö
Hårsfjärden
Haninge
Handen
Vega
Västerhaninge
Tungelsta
Muskö
Tyresö
Nykvarn
Täby
Danderyd
Enebyberg
Djursholm
Stocksund
Stockholm
Årsta
Enskede Gård
Johanneshov
Enskededalen
Stockholm-Globen
Enskede
Farsta
Bandhagen
Älvsjö
Hägersten
Skarpnäck
Bagarmossen
Sköndal
Bromma
Vällingby
Spånga
Kista
Hässelby
Mölnbo
Enhörna
Södertälje
Järna
Hölö
Nacka
Nacka Strand
Saltsjö-Duvnäs
Saltsjö-Boo
Saltsjöbaden
Älta
Sundbyberg
Solna
Tomteboda
Lidingö
Vaxholm
Bergshamra
Gräddö
Blidö
Yxlan
Furusund
Vätö
Väddö
Arholma
Björkö
Barnens Ö
Grisslehamn
Herräng
Norrtälje
Rimbo
Hallstavik
Skebobruk
Ösmo
Sorunda
Stora Vika
Nynäshamn
Sollentuna
Sigtuna
Uppl[...]<jupyter_text>As seen in the result of the ```.head()``` function above, we get the venue coordinates and the category using the API.  Step 6: Explore the most common venues in the county. Our client wants to open a pizza place. Let us find out what are some of the most common venues in Stockholm county.<jupyter_code>most_common_venues = stockholm_venues['VenueCategory'].value_counts().loc[lambda x: x>5].reset_index()['index']

stockholm_venues = stockholm_venues[stockholm_venues['VenueCategory'].isin(most_common_venues)]

stockholm_venues.VenueCategory.value_counts().plot(kind="bar", figsize=(15, 6), title="Total Number of Venues in Stockholm County\nin Each Category")<jupyter_output><empty_output><jupyter_text>Pizza Places are the most common venues in Stockholm county, followed by Hotels as of 2021.  Step 7: Pre-process the data before clustering analysis. For clustering, we process the data to get the venue category frequency related information for neighbourhoods. First, we do a one-hot encoding of the ```VenueCategory``` column and add the ```Neighbourhood``` column in the front. Second, we group the resulting encoded data by the ```Neighbourhood``` column using mean aggregate. A higher mean value of a venue category for a neighbourhood represents the high frequency of that one.<jupyter_code># one-hot encoding
stockholm_onehot = pd.get_dummies(stockholm_venues[['VenueCategory']],
                                prefix="",
                                prefix_sep="")

# add 'Neighbourhood' column to this data frame
stockholm_onehot['Neighbourhood'] = stockholm_venues['Neighbourhood'] 

# move 'Neighbourhood' column to first position
fixed_columns = [stockholm_onehot.columns[-1]] + list(stockholm_onehot.columns[:-1])
stockholm_onehot = stockholm_onehot[fixed_columns]

# group rows by neighbourhood taking mean of frequency of occurrence of each category
stockholm_grouped = stockholm_onehot.groupby('Neighbourhood').mean().reset_index()

stockholm_grouped.head()<jupyter_output><empty_output><jupyter_text>The result here represents our clustering input. In other words, clustering will be performed on these numbers.  Step 8: Get the top 10 most common venue categories for each neighbourhood. After clustering, we need to examine the clusters formed to select the cluster the client would like to examine further in detail using other methods. Giving this cluster of districts to the client for their consideration is the goal of this project. To examine the clusters, we find the top 10 most common venues for each neighbourhood. After the clustering is done, we are going to select our cluster based on this.<jupyter_code># we will look into top 10 most common venue categories
num_top_venues = 10

# define positions indicator suffixes
indicators = ['st', 'nd', 'rd']

# create columns according to number of top venues
columns = ['Neighbourhood']
for ind in np.arange(num_top_venues):
    try:
        columns.append('{}{} Most Common Venue'.format(ind+1, indicators[ind]))
    except:
        columns.append('{}th Most Common Venue'.format(ind+1))

# create new data frame structure
stockholm_venues_sorted = pd.DataFrame(columns=columns)

# add neighbourhood data
stockholm_venues_sorted['Neighbourhood'] = stockholm_grouped['Neighbourhood']

# add top 10 most common venue categories for these neighbourhoods
for ind in np.arange(stockholm_grouped.shape[0]):
    stockholm_venues_sorted.iloc[ind, 1:] = return_most_common_venues(stockholm_grouped.iloc[ind, :],
                                                                          num_top_venues)

stockholm_venues_sorted.head()<jupyter_output><empty_output><jupyter_text>Step 9: Find the optimal K-value for K-Means clustering. Now we start with our clustering. We use the K-Means clustering method for this. For this, we need to find the optimal K-value, the value for which we will receive optimal clusters.We use the elbow method for this. Here, we repeatedly cluster with different K values, measure distortions or inertias (costs), and select the one where the cost plot forms an elbow (considered the optimal point).<jupyter_code># find optimal k checking both distortions and inertias

# define clustering data (data on basis of which clustering is performed)
stockholm_grouped_clustering = stockholm_grouped.drop('Neighbourhood', 1)

# we will check elbow plots for distortions and inertias
distortions = []
inertias = []

# check for K ranging from 1 to 10
K = range(1, 10)

for k in K:
    # define KMeans object
    kmeans_model = KMeans(n_clusters = k, random_state=0)
    
    # fit clustering data
    kmeans_model.fit(stockholm_grouped_clustering)
 
    # calculate distortion measure
    distortions.append(sum(np.min(cdist(stockholm_grouped_clustering, kmeans_model.cluster_centers_,
                                        'euclidean'), axis = 1)) / stockholm_grouped_clustering.shape[0])
    # calculate inertia measure
    inertias.append(kmeans_model.inertia_)
    
# plot distortions elbow plot (with different Ks)
plt.plot(K, distortions, 'bx-')
plt.xlabel('Values of K')
plt.ylabel('Distortion')
plt.title('The Elbow Method using Distortion')
plt.show()

# plot inertias elbow plot (with different Ks)
plt.plot(K, inertias, 'bx-')
plt.xlabel('Values of K')
plt.ylabel('Inertia')
plt.title('The Elbow Method using Inertia')
plt.show()<jupyter_output><empty_output><jupyter_text>5 number of clusters seems to be the elbow and hence a good K-value.  Step 10: Perform K-Means clustering. We now cluster using the ```KMeans``` function and 5 as the number of clusters.<jupyter_code># fit kmeans model with 5 clusters - optimal value of k
kclusters = 5

# run k-means clustering
kmeans = KMeans(n_clusters=kclusters, random_state=1).fit(stockholm_grouped_clustering)

# check cluster labels generated for each row in data frame
kmeans.labels_<jupyter_output><empty_output><jupyter_text>Step 11: Merge the cluster information with the common venues information. This step is essential as we will use the merge data to select our cluster.<jupyter_code># combine clustering information with other data for cluster examination
stockholm_venues_sorted.insert(0, 'ClusterLabels', kmeans.labels_)

stockholm_merged = stockholm_data

# merge stockholm_grouped with stockholm_data to add latitude/longitude for each neighbourhood
stockholm_merged = stockholm_merged.join(stockholm_venues_sorted.set_index('Neighbourhood'),
                                     on='Neighbourhood', how='right')

stockholm_merged.head()<jupyter_output><empty_output><jupyter_text>We can check the size of our clusters. At this point, we decide whether we are satisfied with our clustering results or want to go back and change other parameters such as the number of clusters or the algorithm itself.We like our cluster sizes and hence move forward to examine individual clusters.<jupyter_code># look at number of neighbourhoods per cluster
stockholm_merged.ClusterLabels.value_counts()<jupyter_output><empty_output><jupyter_text>Step 12: Display clustered Stockholm County neighbourhoods on the map. Cluster exploration has to start with exploring the cluster on the map. Here we use color to distinguish districts belonging to different clusters.<jupyter_code># create map of Stockholm using latitude and longitude value
map_clusters = folium.Map(location=[latitude, longitude], zoom_start=11)

# set color scheme for clusters
colors_array = cm.rainbow(np.linspace(0, 1, kclusters))
rainbow = [colors.rgb2hex(i) for i in colors_array]

# add markers to map
markers_colors = []
for lat, lon, poi, cluster in zip(stockholm_merged['Latitude'],
                                  stockholm_merged['Longitude'],
                                  stockholm_merged['Neighbourhood'],
                                  stockholm_merged['ClusterLabels']):
    label = folium.Popup(str(poi) + ' Cluster ' + str(cluster),
                         parse_html=True)
    folium.CircleMarker(
        [lat, lon],
        radius=5,
        popup=label,
        color=rainbow[cluster-1],
        fill=True,
        fill_color=rainbow[cluster-1],
        fill_opacity=0.7).add_to(map_clusters)

    # view clusters at end
map_clusters<jupyter_output><empty_output><jupyter_text>Note: For this map result, check out https://github.com/guptasaumya/coursera-capstone-final/blob/master/map_clustered_neighbourhoods_Stockholm.png in this repository.  Step 13:  Examine clusters. It is time to examine each cluster and select the one we want to forward to the client. How we select that cluster, you ask. We look at the ```1st Most Common Venue``` column value counts. It will list how many districts have a particular venue category as their most common venue.Since our client wants to open a pizza place, we try to see if any cluster has a pizza place as the most common venue category for most of the districts present in that cluster. Yes, read that again!<jupyter_code># Examine cluster 1
cluster_1 = stockholm_merged.loc[stockholm_merged['ClusterLabels'] == 1,
                               stockholm_merged.columns[[0] + list(range(4,
                                                                       stockholm_merged.shape[1]))]]
cluster_1['1st Most Common Venue'].value_counts().plot(kind="bar", y = "1st Most Common Venue", title="Number of Districts for 1st Most Common Venue\nin Cluster 1")<jupyter_output><empty_output><jupyter_text>For cluster 1, only two districts have pizza place as their most common venue. Pizza Places do not seem to be the driving factor for all these districts belonging to cluster 1, but Hotels seem to be. Suppose these two districts already have sufficiently many pizza places. In that case, there is not much certainty of a beneficial outcome for the client's pizza place in the other districts since pizza place is not a very common venue in other districts of the cluster.<jupyter_code># Examine cluster 2
cluster_2 = stockholm_merged.loc[stockholm_merged['ClusterLabels'] == 2,
                               stockholm_merged.columns[[0] + list(range(4,
                                                                       stockholm_merged.shape[1]))]]
cluster_2['1st Most Common Venue'].value_counts().plot(kind="bar", y = "1st Most Common Venue", title="Number of Districts for 1st Most Common Venue\nin Cluster 2")
# Examine cluster 3
cluster_3 = stockholm_merged.loc[stockholm_merged['ClusterLabels'] == 3,
                               stockholm_merged.columns[[0] + list(range(4,
                                                                       stockholm_merged.shape[1]))]]
cluster_3['1st Most Common Venue'].value_counts().plot(kind="bar", y = "1st Most Common Venue", title="Number of Districts for 1st Most Common Venue\nin Cluster 3")
# Examine cluster 4
cluster_4 = stockholm_merged.loc[stockholm_merged['ClusterLabels'] == 4,
                               stockholm_merged.columns[[0] + list(range(4,
                                                                       stockholm_merged.shape[1]))]]
cluster_4['1st Most Common Venue'].value_counts().plot(kind="bar", y = "1st Most Common Venue", title="Number of Districts for 1st Most Common Venue\nin Cluster 4")
# Examine cluster 0
cluster_0 = stockholm_merged.loc[stockholm_merged['ClusterLabels'] == 0,
                               stockholm_merged.columns[[0] + list(range(4,
                                                                       stockholm_merged.shape[1]))]]
cluster_0['1st Most Common Venue'].value_counts().plot(kind="bar", y = "1st Most Common Venue", title="Number of Districts for 1st Most Common Venue\nin Cluster 0")<jupyter_output><empty_output><jupyter_text>In this cluster, pizza place is the most common venue for many districts. Hence, it is evident that the districts are together in this cluster because of their score for the pizza place category. Let us examine the districts in the cluster 0.<jupyter_code>stockholm_merged[stockholm_merged['ClusterLabels'] == 0]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07626018179671822, 'fraction_numerical': 0.010427733837012553, 'mean_word_length': 3.7950943396226413, 'pattern_counts': {'":': 0, '<': 91, '<?xml version=': 0, '>': 99, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 43, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '193055', 'license': 'MIT', 'n_tokens_mistral': 7485, 'n_tokens_neox': 7068, 'n_words': 2604, 'path': 'the_battle_of_neighbourhoods.ipynb', 'repo_name': 'guptasaumya/coursera-capstone-final', 'size': '25413'}