<jupyter_start><jupyter_text>What kind of restaurant in Riga? Business problemGiven growth of the number of visitors to Riga, there is opportunity for opening new restaurants in Riga. The first thing to be clear about when opening a restaurant is to know what kind of food to serve.The present investigation seeks to answer the following question of prospective entrepreneurs wanting to open a restaurant in Riga: > *What type of restaurant, serving what type of food, is relatively likely to be successful in Riga?* Methodology1. Get a list of museums spread over Europe as points of focus. Reason for taking museums is that these are spots visited by tourists and hence we can expect touristic facilities like restaurants in the vicinity.2. Get an overview of restaurants near each of these museums from FourSquare.3. Use K-means to cluster the different areas and analyse the characteristics of the clusters.4. Compare Riga with other areas within the cluster it falls into.5. Where a restaurant type is underrepresented in Riga compared to the other places in its cluster, we conclude that there is a relatively good probability that a restaurant of this type can be successful.  Generic importsWe import the following python libraries.<jupyter_code>import pandas as pd
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

import numpy as np

import json # library to handle JSON files

#!conda install -c conda-forge geopy --yes # uncomment this line if you haven't completed the Foursquare API lab
#from geopy.geocoders import Nominatim # convert an address into latitude and longitude values
# !pip3 install geocoder==0.6.0
import geocoder
from geopy import Nominatim

import requests # library to handle requests
from pandas.io.json import json_normalize # tranform JSON file into a pandas dataframe

# Matplotlib and associated plotting modules
import matplotlib.cm as cm
import matplotlib.colors as colors
import matplotlib.pyplot as plt
%matplotlib inline

# import k-means from clustering stage
from sklearn.cluster import KMeans

#!conda install -c conda-forge folium=0.5.0 --yes # uncomment this line if you haven't completed the Foursquare API lab
import folium # map rendering library

import pickle
print('Libraries imported.')<jupyter_output>Libraries imported.<jupyter_text>1. Get a list of museums in EuropeTake a start here: https://culturetourist.com/museums/15-best-museums-in-europe-you-have-to-visit-this-year/Unfortunately, it is hard to find a list of addresses of museums, so we manually go for them. Several museums were added manually later, in order to have a better spread of locations over Europe.<jupyter_code>lnma_riga = 'Latvian National Museum of Art, Riga'

museum_addresses_input = {
    'Rijksmuseum, Amsterdam': 'Museumstraat 1, 1071 XX, Amsterdam',
    'Prado Museum, Madrid': 'Paseo del Prado, s/n, 28014, Madrid',
    'Uffizi Gallery, Florence': 'Piazzale degli Uffizi 6, 50122, Florence',
    'Museum of Warsaw': 'Rynek Starego Miasta 28-42, 00-272 Warszawa',
    'National Museum of the Azulejo, Lisbon': (38.7246327,-9.1159956), #The following doesn't work: <NAME> 4, 1900-312 Lisboa, Portugal',
    'Musee Louvre, Paris': 'Rue de Rivoli, 75001, Paris',
    'Acropolis Museum, Athens': 'Dionysiou Areopagitou 15, 117 42, Athens',
    'Hermitage Museum, St. Petersburg': 'Palace Square, 2, St Petersburg, Russia',
    'Vasa Museum, Stockholm': 'Galärvarvsvägen 14, 115 21 Stockholm',
    'Picasso Museum, Barcelona': '<NAME>, 15-23, 08003 Barcelona, Spain',
    'National Gallery, London': 'Trafalgar Square, United Kingdom',
    'DDR Museum, Berlin': 'Karl-Liebknecht-Str. 1, 10178 Berlin, Germany',
    'Vatican Museums, Rome': 'Viale Vaticano, 00165 Roma RM, Italy',
    'Musée d\'Orsay, Paris': '1 Rue de la Légion d\'Honneur, 75007 Paris, France',
    'Viking Ship Museum, Roskilde': 'Vindeboder 12, 4000 Roskilde, Denmark',
    'Tate Modern, London': 'Bankside, London SE1 9TG, United Kingdom',
    'Peggy Guggenheim Museum, Venice': 'Dorsoduro, 701-704, 30123 Venezia VE, Italy',
    'Museum Ludwig, Cologne': 'Heinrich-Böll-Platz, 50667 Köln, Germany',
    'Musée d\'Art et de Histoire, Genève': 'Rue Charles-Galland 2, 1206 Genève, Switzerland',
    'The Rosengard Collection, Lucerne': 'Pilatusstrasse 10, 6003 Luzern, Switzerland',
    'Musee d\'Art Comtemporani de Barcelona': 'Plaça dels Angels, 1, Barcelona',
    'Museu Coleção Berardo, Lisbon': 'Praça do Império, 1449-003 Lisboa, Portugal',
    'National Gallery Prague, Prague': 'Staroměstské nám., 110 15 Staré Město, Czechia',
    'Pushkin Museum of Fine Art, Moscow': 'Ulitsa Volkhonka 12, Moscow, Russia',
    'Thorvaldsens Museum, Copenhagen': 'Thorvaldsens Museum, Copenhagen, Denmark',
    'Statens Museum for Kunst, Copenhagen': (55.6887927,12.5081192), # Failed to get address
    'The Bohdan and Varvara Khanenko National Museum of Arts, Kiev': (50.4411509,30.5123974),
    lnma_riga: 'Jaņa Rozentāla laukums 1, Rīga'
}<jupyter_output><empty_output><jupyter_text>Read locations of museumsIf we have a saved version of the overview of museums with their locations we read it from file.Otherwise, we use the above list of museums and get their locations from Foursquare.<jupyter_code>def get_museum_locations(address_dict):
    geolocator = Nominatim(user_agent='museum-explorer')
    d = {'museum': [], 'input_address': [], 'address': [], 'latitude': [], 'longitude': []}
    for museum, raw_address in address_dict.items():
        print(museum, '#', raw_address)
        if isinstance(raw_address, tuple):
            location = {'latitude': raw_address[0], 'longitude': raw_address[1]}
            d['museum'].append(museum)
            d['input_address'].append(np.NaN)
            d['address'].append(np.NaN)
            d['latitude'].append(raw_address[0])
            d['longitude'].append(raw_address[1])
        else:
            location = geolocator.geocode(raw_address)
            if location:
                d['museum'].append(museum)
                d['input_address'].append(raw_address)
                d['address'].append(location.address)
                d['latitude'].append(location.latitude)
                d['longitude'].append(location.longitude)
            else:
                print("*** Failed to get address for", museum, "from", raw_address, "***")
    df = pd.DataFrame(d)
    df.columns = ['Museum', 'Input Address', 'Address', 'Latitude', 'Longitude']
    
    # Save locally
    with open('museums.pickle','wb') as f:
        pickle.dump(df, f)
    return df

museum_locations = None
try:
    with open('museums.pickle', 'rb') as f:
        museum_locations = pickle.load(f)
except:
    print('no pickle file found for museums, reading locations from geolocator')
    museum_locations = get_museum_locations(museum_addresses_input)
    
    
print("Areas: based on {} museums around Europe.".format(museum_locations.shape[0]))
museum_locations<jupyter_output>Areas: based on 28 museums around Europe.<jupyter_text>2. Get venues near the different museums  Define Foursquare credentials As this notebook is shared publicly, credentials are not included in the notebook itself, but rather in a text file residing in the same directory. This text file has the format:- CLIENT_ID: your client id- CLIENT_SECRET: your client secret<jupyter_code>### Set Foursquare properties and credentials
foursquare_secret = {'CLIENT_ID': 'NA', 'CLIENT_SECRET': 'NA', 'VERSION': '20180605'}
with open('foursquare.secret', 'r') as file:
    lines = file.readlines()
    for l in lines:
        ar = l.split(':')
        foursquare_secret[ar[0]] = ar[1].strip()
        
CLIENT_ID = foursquare_secret['CLIENT_ID']
CLIENT_SECRET = foursquare_secret['CLIENT_SECRET']
VERSION = '20180605'<jupyter_output><empty_output><jupyter_text>Restrict venue categoryWe are interested only in restaurants, and in FourSquare these are a subset of the top category 'Food'<jupyter_code># See: https://developer.foursquare.com/docs/resources/categories
wanted_categories = {
    'Food': '4d4b7105d754a06374d81259' # This is the top category under which restaurants fall
}
list(wanted_categories.keys())<jupyter_output><empty_output><jupyter_text>Extend 'radius' to increase query resultsFourSquare limits the number of venues resulting from a query to 50. We want more than that, partly because the search result includes bars, coffee shops, and other non-restaurant venues that we will filter out from the query result, so we will be left with fewer restaurants than the limit on venues that FourSquare sets.To remedy this limitation, we will query a number of locations surrounding the center location ("museum") that we are interested in. The following methods identify such a surrounding.As a result, we are not really querying venues within a *radius* of the central location, but within the radius of a number of locations surrounding it. A diagram below the methods illustrates the shape of the resulting vicinity of the central location in which venues will be queried.<jupyter_code>def lat_lon_move(latitude, longitude, north, east):
    '''Return move in meters north and east of (latitude,longitude)'''
    # Earth’s radius, sphere
    R=6378137

    # Coordinate offsets in radians
    dLat = north/R
    dLon = east/(R*np.cos(np.pi*latitude/180))

    # OffsetPosition, decimal degrees
    lat_new = latitude + dLat * 180/np.pi
    lon_new = longitude + dLon * 180/np.pi
    return (lat_new, lon_new)
    
def surround(latitude, longitude, radius):
    '''
    FourSquare search has an upper limit of 50 results. Part of those are not restaurants
    and have to be filtered out of the query result. To get more results, we surround the center
    location with other locations so we can query each of those.
    '''
    lt = lat_lon_move(latitude, longitude, radius, -1*radius)
    rt = lat_lon_move(latitude, longitude, radius, radius)
    lb = lat_lon_move(latitude, longitude, -1*radius, -1*radius)
    rt = lat_lon_move(latitude, longitude, -1*radius, radius)
    ww = lat_lon_move(latitude, longitude, 0, -2*radius)
    ee = lat_lon_move(latitude, longitude, 0, 2*radius)
    nn = lat_lon_move(latitude, longitude, 2*radius, 0)
    ss = lat_lon_move(latitude, longitude, -2*radius, 0)
    return [(latitude,longitude), lt, rt, lb, rt, ww, ee, nn, ss]
def plot_surroundings(radius=350, title='Vicinity in meters'):
    '''Display the shape of the vicinity (rather than radius) in which we actually query for nearby venues'''
    circle_center = plt.Circle((0, 0), radius, color='g')
    circle_museum = plt.Circle((0, 0), 20, color='y')
    circle_lt = plt.Circle((radius, -1*radius), radius, color='r')
    circle_rt = plt.Circle((radius, radius), radius, color='r')
    circle_lb = plt.Circle((-1*radius, -1*radius), radius, color='r')
    circle_rb = plt.Circle((-1*radius, radius), radius, color='r')
    circle_ww = plt.Circle((0, 2*radius), radius, color='r')
    circle_ee = plt.Circle((0,-2*radius), radius, color='r')
    circle_nn = plt.Circle((2*radius, 0), radius, color='r')
    circle_ss = plt.Circle((-2*radius, 0), radius, color='r')

    fig, ax = plt.subplots() # note we must use plt.subplots, not plt.subplot

    ax.add_artist(circle_lt)
    ax.add_artist(circle_rt)
    ax.add_artist(circle_lb)
    ax.add_artist(circle_rb)
    ax.add_artist(circle_ww)
    ax.add_artist(circle_ee)
    ax.add_artist(circle_nn)
    ax.add_artist(circle_ss)

    ax.add_artist(circle_center)
    ax.add_artist(circle_museum)
    
    ax.set_xlim(-3.5*radius,3.5*radius)
    ax.set_ylim(-3.5*radius,3.5*radius)
    
    ax.set_title(title)
    plt.show()
plot_surroundings(radius=350, title='Vicinity covered in meters from the center location (yellow) for radius=350m' )
def get_nearby_venues(names, latitudes, longitudes, radius=500, limit=50, intent='browse', categories=None):
    '''
    @param categories list of Foursquare category identifiers
    '''
    venues_list=[]
    venues_ids=[]
    print('Now getting venues for:')
    for name, lat_museum, long_museum in zip(names, latitudes, longitudes):
        print('-', name)
        
        # We need more results than the limit=50 that FourSquare sets, so we search surrounding points
        for lat, lng in surround(lat_museum, long_museum, radius):
            # create the API request URL
            url = 'https://api.foursquare.com/v2/venues/search?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}&intent={}'.format(
                CLIENT_ID, 
                CLIENT_SECRET, 
                VERSION, 
                lat, 
                lng, 
                radius, 
                limit,
                intent
            )
        
            if categories:
                url = url + "&categoryId={}".format(','.join(categories))
                    
            # make the GET request
            results = requests.get(url).json()["response"]['venues']
        
            if results:            
                # print all categories
                for v in results:
                    if len(v['categories']) > 1:
                        print(v['categories'])
            
                # return only relevant information for each nearby venue
                for v in results:
                    if not v['id'] in venues_ids:
                        venues_ids.append(v['id'])
                        venues_list.append([(
                            name, 
                            lat, 
                            lng, 
                            v['name'],
                            v['id'],
                            v['location']['lat'], 
                            v['location']['lng'],
                            v['location']['distance'],
                            v['categories'][0]['name'])])

    nearby_venues = pd.DataFrame([item for venue_list in venues_list for item in venue_list])
    nearby_venues.columns = ['Museum', 
                             'Museum Latitude',
                             'Museum Longitude',
                             'Venue',
                             'Venue id',
                             'Venue Latitude',
                             'Venue Longitude',
                             'Distance',
                             'Venue Category']
    
    with open('nearby_venues_{}.pickle'.format(radius),'wb') as f:
        pickle.dump(nearby_venues, f)
    
    return(nearby_venues)<jupyter_output><empty_output><jupyter_text>Set a mask for museums to be loaded<jupyter_code>test = False # Set to True to read data on one location, so as to limit calls to FourSquare while testing
mask = museum_locations['Museum'].isin(list(museum_addresses_input.keys())) # read all
if test:
    mask = museum_locations['Museum'].isin(['Rijksmuseum, Amsterdam']) 
museum_locations[mask].head()
# --- Configuration here ---
# NOTA BENE: the here defined 'radius' is used to define a vicity that coveres a radius that is actually at least twice as large. See above.
radius = 350
# --- end configuration ---
nearby_venues = None
try:
    with open('nearby_venues_{}.pickle'.format(radius), 'rb') as f:
        print('Reading nearby venues from file')
        nearby_venues = pickle.load(f)
except:
    print('No pickle file found for nearby venues, reading venues from Foursquare')
    nearby_venues = get_nearby_venues(names=museum_locations[mask].Museum,
                                    latitudes=museum_locations[mask].Latitude,
                                    longitudes=museum_locations[mask].Longitude,
                                    radius=radius,
                                    categories=list(wanted_categories.values())
                                    )
        
print("Having {} museum centered locations around Europe.".format(museum_locations.shape[0]))
print('Found {} venues'.format(nearby_venues.shape[0]))
nearby_venues.head()<jupyter_output>Reading nearby venues from file
Having 28 museum centered locations around Europe.
Found 5554 venues<jupyter_text>Check limitWe had set a limit to the number of venues returned from the FourSquare query. Is this limit reasonable?Let's see how many venues are returned for each museum.<jupyter_code>nearby_venues[['Museum','Venue']].groupby('Museum').count()<jupyter_output><empty_output><jupyter_text>Filter out any venue that is not a restaurant<jupyter_code>restaurants = nearby_venues[nearby_venues['Venue Category'].str.contains('Restaurant')]
print("Found {} venues after filter out those without 'Restaurant' in category".format(restaurants.shape[0]))<jupyter_output>Found 2468 venues after filter out those without 'Restaurant' in category<jupyter_text>Check if we actually found a significant number of restaurants for every museum locationDisplay histogram and the number of restaurants in ascending order of each of the museum locations.<jupyter_code>r = restaurants[['Venue Category','Venue']].groupby('Venue Category').count().sort_values(by='Venue', ascending=False)
r.hist('Venue', bins=50)
plt.xlim(xmin=0,xmax=500)  # this line
plt.title('Distribution of the number of restaurants per restaurant category')
plt.xlabel('Number of restaurants')
plt.ylabel('Restaurant Categories')
plt.show()<jupyter_output><empty_output><jupyter_text>How should we interpret this histogram? It tells us that:* In the vast majority of restaurant types we find only up to 10 restaurants all over our dataset* Three restaurant types have more than 100 restaurants and two of them several hundred.Hence, the number of restaurants is distributed very unequally over restaurant types.<jupyter_code>q = restaurants[['Museum','Venue']].groupby('Museum').count().sort_values('Venue',ascending=True)
q<jupyter_output><empty_output><jupyter_text>Let's visualize the distribution of venues per area.<jupyter_code>q.hist(bins=16)
plt.xlim(xmin=0,xmax=160)  # this line
plt.title('Distribution of the number of restaurants per museum vicinity')
plt.xlabel('Number of restaurants')
plt.ylabel('Museum vicinities')
plt.show()<jupyter_output><empty_output><jupyter_text>From the above histogram we learn that the number of venues per area is relatively centered at around 90. This seems like a reasonable basis for continuing the investigation according to the plan set out in the methodology. Let's have a look at the data we have now.<jupyter_code>restaurants.head()
restaurants.shape<jupyter_output><empty_output><jupyter_text>Venue categoriesWhat venue categories - types of restaurants - do we now have in our dataset?Below we first display this in a table, then in a histogram<jupyter_code>category_overview = restaurants[['Venue Category','Venue']].groupby('Venue Category',as_index=False).count().\
    sort_values(by='Venue', ascending=False)
category_overview.head(10)
def display_categories(df, top=20, title='in our dataset'):
    plt.rcdefaults()
    ndf = df.copy().reindex(index=df.index[::-1])
    fig, ax = plt.subplots()
    ax.set_title('Top {} restaurant categories {}'.format(top, title))
    
    ndf[-1*top:].plot.barh(x='Venue Category', y='Venue', ax=ax, legend=False)
    plt.xlabel('Number of restaurants')
    plt.ylabel('')
    plt.show()
display_categories(category_overview, top=20, title='in our dataset')<jupyter_output><empty_output><jupyter_text>Venue categories per museum<jupyter_code># one hot encoding
restaurants_onehot = pd.get_dummies(restaurants[['Venue Category']], prefix="", prefix_sep="")

# add museum column back to dataframe
restaurants_onehot['Museum'] = restaurants['Museum'] 

# move neighborhood column to the first column
fixed_columns = [restaurants_onehot.columns[-1]] + list(restaurants_onehot.columns[:-1])
restaurants_onehot = restaurants_onehot[fixed_columns]

restaurants_onehot.head()
restaurants_grouped = restaurants_onehot.groupby('Museum').mean().reset_index()
restaurants_grouped.head()<jupyter_output><empty_output><jupyter_text>Let's see what the current size is<jupyter_code>restaurants_grouped.shape<jupyter_output><empty_output><jupyter_text>Putting the most popular restaurants per area in a dataframeWe put the most popular restaurants as columns in a dataframe.<jupyter_code>def return_most_common_venues(row, num_top_venues):
    row_categories = row.iloc[1:]
    row_categories_sorted = row_categories.sort_values(ascending=False)
    
    return row_categories_sorted.index.values[0:num_top_venues]
num_top_venues = 15

indicators = ['st', 'nd', 'rd']

# create columns according to number of top venues
columns = ['Museum']
for ind in np.arange(num_top_venues):
    try:
        columns.append('{}{} Most Common Venue'.format(ind+1, indicators[ind]))
    except:
        columns.append('{}th Most Common Venue'.format(ind+1))

# create a new dataframe
museums_venues_sorted = pd.DataFrame(columns=columns)
museums_venues_sorted['Museum'] = restaurants_grouped['Museum']

for ind in np.arange(restaurants_grouped.shape[0]):
    museums_venues_sorted.iloc[ind, 1:] = return_most_common_venues(restaurants_grouped.iloc[ind, :], num_top_venues)

museums_venues_sorted.head()<jupyter_output><empty_output><jupyter_text>3. Cluster museums  Preliminaries: cluster and presentBefore actually doing any clustering of neighbourhoods, let's define methods for:1. Clustering2. Presenting the result of clustering on a map3. Evaluate the clusters<jupyter_code>def cluster_museums(kclusters):
    '''Cluster neighbourhoods in kcluster groups'''
    restaurants_grouped_clustering = restaurants_grouped.drop('Museum', 1)

    # run k-means clustering
    kmeans = KMeans(n_clusters=kclusters, random_state=0).fit(restaurants_grouped_clustering)

    # check cluster labels generated for each row in the dataframe
    kmeans.labels_
    
    # add clustering labels
    nvs = museums_venues_sorted.copy()
    nvs.insert(0, 'Cluster Labels', kmeans.labels_)
    
    # merge toronto_grouped with toronto_data to add latitude/longitude for each neighborhood
    museums_merged = pd.merge(museum_locations, nvs.set_index('Museum'), on='Museum')

    cluster_counts = museums_merged[['Museum', 'Cluster Labels']].groupby("Cluster Labels").count().sort_values(by='Museum', ascending=False)
    cluster_counts.reset_index()
    return museums_merged, cluster_counts
def evaluate_clusters(cluster_counts, restaurants_onehot, museums_merged):
    num_top_venues = 5
        
    clustered_onehot = pd.merge(restaurants_onehot, museums_merged[['Museum', 'Cluster Labels']], 
                            how='left', on='Museum')
    restaurants_grouped_clusters = clustered_onehot.groupby('Cluster Labels').mean().reset_index()

    rows = int(cluster_counts.shape[0]/2)+cluster_counts.shape[0]%2
    cols = 2
    fig_dims = (7*cols, 5*rows)
    fig, axes = plt.subplots(rows, cols, figsize=fig_dims)
    fig.tight_layout(w_pad=11.0, h_pad=4.0)

    i=0
    for cluster_id, freq in cluster_counts.itertuples():
        mask = restaurants_grouped_clusters['Cluster Labels'] == cluster_id
        temp = restaurants_grouped_clusters[mask].T.reset_index()
        temp.columns = ['venue','freq']
        temp = temp.iloc[1:]
        temp['freq'] = temp['freq'].astype(float)
        temp = temp.round({'freq': 2})
        showframe = temp.sort_values('freq', ascending=False).reset_index(drop=True).head(num_top_venues)
        mask2 = showframe['freq'] > 0.0
        # Remove all items with frequency zero
        showframe = showframe[mask2]
        # Reindex to reverse the order as we want top frequency shown at the top of the barchart
        showframe = showframe.reindex(index=showframe.index[::-1])
        # Prevent autoscale from making fewer bars in any chart become extra width
        width = (0.8 * showframe.shape[0] / 5) - (5-showframe.shape[0])*0.01
        title = "Cluster {}: {} museums".format(cluster_id, freq)
        c = 'blue'
        m = museums_merged['Cluster Labels'] == cluster_id
        if freq > 0 and freq <= 3:
            name = ''
            try:
                name = ' | '.join(list(map(lambda x: x.split(',')[0], museums_merged[m]['Museum'].to_list())))
            except:
                name = ' | '.join(list(map(lambda x: x.split(',')[0], museums_merged[m]['Museum'].values)))
            c = 'green'
            designation = ''
            if freq == 1:
                designation = 'outlier'
            elif freq == 2:
                designation = 'pair'
            elif freq == 3:
                designation = 'triplet'
            title = "Cluster {}: {}: {}".format(cluster_id, designation, name)
        print("Cluster {}: ".format(cluster_id), " | ".join(museums_merged[m]['Museum'].tolist()))
        cur_ax = axes[int(i/2), i%2]
        showframe.plot(ax=cur_ax, kind='barh',y='freq', x='venue', width=width, color=c, legend=False)
        cur_ax.set_title(title)
        cur_ax.set_xlim(right=1.0)
        cur_ax.yaxis.set_label_text("")
        i=i+1
    # If the number of plot is odd, we need to remove the last axes slot
    if cluster_counts.shape[0] % 2 != 0:
        fig.delaxes(axes[rows-1, 1])<jupyter_output><empty_output><jupyter_text>Evaluate clustersHere we set the number of clusters we want to distinguish, use K-means to cluster areas (museums), and evaluate the characteristics of each cluster.<jupyter_code># Form the clusters using K-means clustering algorithm
kclusters = 7
museums_merged, cluster_counts = cluster_museums(kclusters)<jupyter_output><empty_output><jupyter_text>Cluster characteristicsBelow we evaluate clusters. The clusters are labelled by a number. For reference, first we list what locations ("museums") are part of each cluster. Second we display a barchart of the five most prevalent restaurant categories in each cluster.For quick scanning, the bars are colored:* **blue** in clusters with a significant number of elements* **green** for clusters with only a few elements.<jupyter_code>evaluate_clusters(cluster_counts, restaurants_onehot, museums_merged)
museums_merged.head()
# Let's create a method to create the map, so we can call it again
# add markers to map
def cluster_map(center_location, latitude_series, longitude_series, name_series, type_series=None, zoom_start=11):
    map = folium.Map(location=center_location, zoom_start=zoom_start)
    if type_series is not None:
        types = type_series.unique().tolist()
        colors_array = cm.rainbow(np.linspace(0, 1, len(types)))
        rainbow = [colors.rgb2hex(i) for i in colors_array]
        for lat, lng, name, typ in zip(latitude_series, longitude_series, name_series, type_series):
            label = name
            label = folium.Popup(label, parse_html=True)
            folium.CircleMarker(
                [lat, lng],
                radius=5,
                popup=label,
                color=rainbow[types.index(typ)],
                fill=True,
                fill_color=rainbow[types.index(typ)],
                fill_opacity=0.7,
                parse_html=False).add_to(map)
    else:
        for lat, lng, name in zip(latitude_series, longitude_series, name_series):
            label = folium.Popup(name, parse_html=True)
            folium.CircleMarker(
                [lat, lng],
                radius=5,
                popup=label,
                color='b',
                fill=True,
                fill_color='b',
                fill_opacity=0.7,
                parse_html=False).add_to(map)
            
    return map<jupyter_output><empty_output><jupyter_text>Centering our mapTo draw our map we must first determine the center of our locations.<jupyter_code>def get_center(df,latitude='Latitude', longitude='Longitude'):
    '''Get the center of all locations to be mapped'''
    max_latitude = df[latitude].max()
    min_latitude = df[latitude].min()
    max_longitude = df[longitude].max()
    min_longitude = df[longitude].min()
    
    return (max_latitude+min_latitude)/2, (max_longitude+min_longitude)/2

center_latitude, center_longitude = get_center(museums_merged, latitude='Latitude', longitude='Longitude')
print('The geograpical coordinates of the center are {}, {}.'.format(center_latitude, center_longitude))<jupyter_output>The geograpical coordinates of the center are 48.953831075, 14.198282613271108.<jupyter_text>Drawing the clustersLet's draw the clusters we found on a map.<jupyter_code>cluster_map(
    (center_latitude, center_longitude),
    museums_merged['Latitude'],
    museums_merged['Longitude'],
    museums_merged['Museum'],
    type_series=museums_merged['Cluster Labels'],
    zoom_start=4
)<jupyter_output><empty_output><jupyter_text>Get all museums in the same cluster as the Riga National Art GalleryFirst get the cluster label of the Riga National Art Gallery, then select the other museumsthat are part of the same cluster.<jupyter_code>riga_cluster_label = museums_merged[museums_merged['Museum'] == lnma_riga]['Cluster Labels'].tolist()[0]
riga_cluster = museums_merged[museums_merged['Cluster Labels'] == riga_cluster_label]
riga_cluster.reset_index(inplace=True)
riga_cluster.head()<jupyter_output><empty_output><jupyter_text>4. Compare Riga with the rest of the cluster it is part ofLet's take the *Riga National Gallery* out of the set, so we can compare the venue categories in Rigawith those in the rest of the cluster it is part of.<jupyter_code>riga_museum = riga_cluster[riga_cluster['Museum'] == lnma_riga]
riga_cluster = riga_cluster[riga_cluster['Museum'] != lnma_riga]
riga_cluster.head()<jupyter_output><empty_output><jupyter_text>Let's get back the original data for venuesGet data for the rest of the cluster.<jupyter_code>baseline = pd.merge(riga_cluster['Museum'], restaurants_grouped, how='left', on='Museum')
baseline.head()<jupyter_output><empty_output><jupyter_text>Get data for Riga<jupyter_code>riga_baseline = pd.merge(riga_museum['Museum'], restaurants_grouped, how='left', on='Museum')
riga_baseline
number_of_top_categories = 10
top_categories = baseline.mean(axis = 0).sort_values(ascending=False)[:number_of_top_categories]
top_categories = top_categories.to_frame().reset_index()
top_categories.columns = ['Venue Category','Average']
top_categories
riga_cluster['Museum'].to_list()
cluster_restaurants_mask = restaurants['Museum'].isin(riga_cluster['Museum'].to_list())
r = restaurants[cluster_restaurants_mask]
category_overview2 = r[['Venue Category','Venue']].groupby('Venue Category',as_index=False).count().\
    sort_values(by='Venue', ascending=False)
display_categories(
    category_overview2, top=20, title=' in the Riga cluster dataset'
)
p = (int(category_overview2[category_overview2['Venue Category'] == 'Restaurant']['Venue']) / category_overview2['Venue'].sum())*100
print("{}% of restaurants belong to the generic category 'Restaurant'".format(int(p)))
riga_categories = baseline<jupyter_output><empty_output><jupyter_text>cluster_restaurants_mask = restaurants['Museum'].isin(riga_cluster['Museum'].to_list())r = restaurants[cluster_restaurants_mask] What restaurant categories do we see in Riga?<jupyter_code>riga_restaurants_mask = restaurants['Museum'] == lnma_riga
r1 = restaurants[riga_restaurants_mask]
category_overview3 = r1[['Venue Category','Venue']].groupby('Venue Category',as_index=False).count().\
    sort_values(by='Venue', ascending=False)
display_categories(
    category_overview3, top=20, title=' in the dataset for the Riga area'
)
print("Total number of restaurants in Riga is: {}".format(r1.shape[0]))<jupyter_output>Total number of restaurants in Riga is: 103<jupyter_text>Drop all columns that have an average lower than the top categories<jupyter_code>top_categories['Venue Category'].tolist()
short_list = baseline[['Museum']+top_categories['Venue Category'].tolist()]
short_list.head()
top_categories_list = top_categories['Venue Category'].tolist()
import seaborn as sns
def overview_plot(general_df, focus_df,title='Spread of venue categories in cluster'):
    '''
    @param general_df Dataframe containing venue category data on all locations
    @param focus_df Dataframe containing venue category data on the focus location
    '''
    fig_dims = (15, 8)
    fig, ax = plt.subplots(figsize=fig_dims)

    # boxplot or violinplot with seaborn
    bplot=sns.violinplot(
        ax=ax,
        data=general_df,
        width=0.5,
        fliersize=5,
        palette="colorblind")

    bplot.set_xticklabels(bplot.get_xticklabels(), rotation=45, horizontalalignment='right')
    bplot.set_title(title)
 

    # add swarmplot for cluster
    bplot=sns.swarmplot(
            ax=ax,
            data=general_df,
            size=4,
            color='white',
            alpha=0.75)

    # add swarmplot for the focus group
    bplot=sns.swarmplot(
            ax=ax,
            data=focus_df,
            color='red',
            size=8,
            alpha=0.75)
overview_plot(
    short_list[top_categories_list], 
    riga_baseline[top_categories_list],
    title='Spread of restaurant categories in cluster'
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10660145213719355, 'fraction_numerical': 0.015554272868122854, 'mean_word_length': 3.780424048794656, 'pattern_counts': {'":': 0, '<': 110, '<?xml version=': 0, '>': 113, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '48905', 'license': 'Apache-2.0', 'n_tokens_mistral': 10553, 'n_tokens_neox': 9877, 'n_words': 2823, 'path': 'riga_restaurants.ipynb', 'repo_name': 'ceesroele/Coursera_Capstone', 'size': '32931'}