<jupyter_start><jupyter_text>Foodie's Guide to Finding a Home in Bahrain> By clustering neighborhoods based on popular venues surrounding them- toc: false- badges: true- comments: true- categories: [ml,sklearn]- show_tags: true- image: images/bahrain-segmentation/bahrain-restaurant.jpg I am a foodie looking for a place stay in Bahrain. I want to study certain areas in Bahrain and the kind of restaurants that surround them.I think that a lot of people, not just the youth could benefit from this, since the issue isn't just finding a decent place to stay in Bahrain, but finding one that best serves their culinary interests perhaps.I mean there are obviously much better factors to look at besides food. However for this problem I want to stick to what I can gain from **Foursquare** with a free license. Thus, by neatly categorizing areas based on their attributes such as frequency of coffee shops, closeness to malls etc; I can can make a better guesstimate of where they might stay.Foursquare allows us to grab information on venues surrounding a given location, and therefore we will look into the most frequent kind of venues surrounding a given area, and cluster areas them on that. So let's get started! <!--  MethodologySince our first and ultimate goal revolves around working with data which are vastly numeric in type such as latitude and longitude values for coordinates, I will leverage the [Pandas](https://pandas.pydata.org/) library to store, load and process this data. We gather the areas in a pandas dataframe for easy processing, and attempt to collect coordinates that can be processed efficiently using such a library.In order to get the most trusted list of places in Bahrain, we will refer to this [Wikipedia page](https://en.wikipedia.org/wiki/Category:Populated_places_in_Bahrain) as a source to begin with. We can extract pieces of text from websites using a popular webscraper called [Beautiful Soup](https://beautiful-soup-4.readthedocs.io/en/latest/). We can use the filter the text from the site for areas in Bahrain. After doing this, we form our first dataframe with that data called `bh_data`.For futher consideration, we will need the ask questions such as, "does it cover all areas mentioning in the Wikipedia Page?" or "Are there ambigious and confusing elements to those locations like repeated points or missing locations?". Upon ansering these, we can then proceed further.After we are confident that we have the names of all major areas in Bahrain, we have to geocode them; a process that converts them from a string to latitude & longitude values marking their location on a map. We can use popular geocoding APIs such as [Open Street Map](https://www.openstreetmap.org/) and [Map Quest](https://www.mapquest.com/). We use both of them, since they're the least difficult to get a free account with, and to ensure we don't have any discrepancies or if the name of a particular location doesn't returns an valid result. After geocoding, we store the cooridinates into our existing dataframe `bh_data` as seen below. Our process of data collection has now ended.| Index |    Area   |  Latitude |  Longitude  ||:-----:|:---------:|:---------:|:-----------:||   0   | A'ali     | 26.154454 | 50.527364   ||   1   | Abu Baham | 26.205737 | 50.541668   ||   2   | Abu Saiba | 30.325299 | 48.266157   ||   3   | <NAME> | 20.639623 | -100.477387 ||   4   | <NAME>  | 26.225405 | 50.59013    |For the sake of visual analysis, we must utilize mapping libraries like [Folium](https://python-visualization.github.io/folium/) to plot our areas as points on a map, so we can better visualize our findings. This will aid us in our analysis as we can better synthesize our results.For the second part, we will look at [Foursquare](https://foursquare.com/) to get us the surrounding food places within 500m of the area's center (an arbitrary radius to avoid collision with venues present in other areas).Foursquare is a service that aids us in obtaining location-based data that details venues, reviews and users' who have reviewed the said venues.For our use-case, we will look only at the list of food places within each area in Bahrain and their food category.For the third part of our project, our business problem relies on segmenting areas based on the most common type of food places within the area. This gives us an idea about the type of area it is from a culinary point-of-view, and allowing us to make judgments on whether the food is ideal to our taste or not. We also want to factor in the total number of food places within an area since some places in Bahrain may not be ideal to live in if they don't even have enough places to eat. --><jupyter_code>#hide
# INSTALLING LIBRARIES
import os
import yaml # Load config file

import numpy as np # library to handle data in a vectorized manner

import pandas as pd # library for data analsysis
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

import json # library to handle JSON files

from bs4 import BeautifulSoup # To scrap data from Wikipedia

import requests # library to handle requests

# Matplotlib and associated plotting modules
import matplotlib.cm as cm
import matplotlib.colors as colors

# import k-means from clustering stage
from sklearn.cluster import KMeans

import folium # map rendering library

from IPython.display import Markdown as md # Convert Python Strings to Markdown text


print('Libraries imported.')
#hide
def MapRender(save_folder):
    count = 1
    def inner(m):
        nonlocal count
        filename = str(count) + '.html'
        count +=1 
        load_path = os.path.join('assets', save_folder)
        
        try:
            os.mkdir(load_path)
        except OSError as error:
            pass
        
        
        if os.path.exists('../_config.yml'):
            with open('../_config.yml', 'r') as file:
                configvars = yaml.load(file, Loader=yaml.FullLoader)
                baseurl = configvars['baseurl']
        
        save_path = '../' + load_path
        m.save(os.path.join(save_path, filename))

        iframe = f'<iframe src="{baseurl}/{os.path.join(load_path, filename)}" width="100%" height="500px"></iframe>'
        
        return md(iframe)
    return inner
render = MapRender(save_folder='bahrain-segmentation')<jupyter_output><empty_output><jupyter_text>Scrap Bahrain Cities/Town Data from WikipediaI need to scrap data from **Wikipedia** to lookup towns and cities in Bahrain. We're going to use the popular webscraper **Beautiful Soup** to do that.<jupyter_code>url = 'https://en.wikipedia.org/wiki/Category:Populated_places_in_Bahrain'
html_doc = requests.get(url).text # Get HTML Doc
soup = BeautifulSoup(html_doc, 'html.parser') # Parse using bs4
blocks = soup.find_all("div", {"class": "mw-category-group"})[1:]

bh_data=[]
for block in blocks:
    places = block.find('ul').find_all('li')
    for place in places:
        bh_data.append(place.a.text.split(',')[0])

bh_data = pd.DataFrame(bh_data, columns=['Area'])
remove_places = ['Rifa and Southern Region', 'Northern City'] # Exclude these places
bh_data = bh_data[bh_data['Area'].apply(lambda item : item not in remove_places)].reset_index(drop=True)
bh_data.head(5)
#hide_input
md(f"So there are about {bh_data.shape[0]} areas in Bahrain to study.")<jupyter_output><empty_output><jupyter_text>Retrieving Coordinates via a Geocoder After that, we need to geocode them; convert them from a simple address to *latitude & longitude* values.Popular geocoders like OpenStreetMap & Map Quest will be used.<jupyter_code>#hide_output
import os
apikey = "<KEY>"
import geocoder

lats = []
lngs = []
for city in bh_data['Area']:
    geocoder_type = 'osm'
    try:
        g = geocoder.osm(f"{city}, Bahrain", key=apikey)
        geodata = g.json
        lats.append(geodata['lat'])
    except:
        geocoder_type = 'MAPQUEST'
        g = geocoder.mapquest(f"{city}, Bahrain", key=apikey)
        geodata = g.json
        lats.append(geodata['lat'])
    lngs.append(geodata['lng'])
    print(city, "|", geocoder_type)
bh_data['Latitude'] = lats
bh_data['Longitude'] = lngs<jupyter_output>A'ali | osm
Abu Baham | osm
Abu Saiba | osm
Al Garrya | MAPQUEST
Al Hajar | osm
Al Kharijiya | MAPQUEST
Al Markh | osm
Al Musalla | osm
Al Qadam | MAPQUEST
Al Qala | osm
Al Qurayyah | MAPQUEST
Amwaj Islands | osm
Arad | osm
Askar | osm
Awali | osm
Budaiya | osm
Jid Ali | osm
Bahrain Bay | osm
Bani Jamra | MAPQUEST
Barbar | osm
Bilad Al Qadeem | osm
Bu Quwah | osm
Buri | osm
Busaiteen | osm
Al Daih | osm
Al Dair | osm
Dar Kulaib | osm
Diplomatic Area | osm
Diraz | osm
Diyar Al Muharraq | osm
Dumistan | MAPQUEST
Durrat Al Bahrain | osm
East Hidd City | osm
Eker | osm
Galali | osm
Al Hidd | osm
Halat Bu Maher | osm
Halat Nuaim | MAPQUEST
Hamad Town | osm
Hamala | osm
Hawar Islands | osm
Hillat Abdul Saleh | MAPQUEST
Isa Town | osm
Janabiyah | osm
Jannusan | osm
Jasra | osm
Jaww | osm
Jid Al-Haj | osm
Jidhafs | osm
Jurdab | MAPQUEST
Karbabad | osm
Karrana | MAPQUEST
Karzakan | osm
Khamis | osm
Ma'ameer | osm
Mahazza | MAPQUEST
Malkiya | osm
Manama | osm
Marquban | MAPQUEST
Muharraq | osm
M[...]<jupyter_text>These are the first few of them that were geocoded!<jupyter_code>#hide_input
bh_data.head()
#hide
# Let's store them as part of the project so we don't have to re-do this step

# Store coordinates as CSV
# bh_data.to_csv("assets/bahrain-locations.csv", index=False)
#hide
# Load Coordinates Data
# bh_data = pd.read_csv("assets/bahrain-locations.csv")<jupyter_output><empty_output><jupyter_text>Visualization on a Map We will now use **Folium** to visualize the map of Bahrain along with each area as points on the map<jupyter_code># create map of Bahrain using latitude and longitude values
latitude, longitude = 26.0766404, 50.334118

map_bahrain = folium.Map(location=[latitude, longitude], zoom_start=10)

# add markers to map
for lat, lng, city in zip(bh_data['Latitude'], bh_data['Longitude'],
                                           bh_data['Area']):
    
    label = folium.Popup(city, parse_html=True)
    folium.CircleMarker(
        [lat, lng],
        radius=5,
        popup=label,
        color='blue',
        fill=True,
        fill_color='#3186cc',
        fill_opacity=0.7,
        parse_html=True).add_to(map_bahrain)  
from IPython.core.display import HTML    
render(map_bahrain)
#hide
map_bahrain<jupyter_output><empty_output><jupyter_text>Foursquare: Exploring Areas for Food Places Futhermore, we'll leverage the Foursquare API to gather the most common types of restaurants associated with an area within 500m of its center. We'll then look at various food places and restaurants and extract their types for further analysis.Note: To filter only restaurants & food places, we will use the specific "Food" category ID : `4d4b7105d754a06374d81259`<jupyter_code>food_categoryId = "4d4b7105d754a06374d81259"
#hide
CLIENT_ID = 'CLIENT_ID_XXXXXXXXXXXXXXXXXXXXXXX' # My Foursquare ID
CLIENT_SECRET = '<KEY>' # My Foursquare Secret
ACCESS_TOKEN = '<KEY>' # My FourSquare Access Token
VERSION = '20180604'
LIMIT = 1000<jupyter_output><empty_output><jupyter_text>Alright, let's look at **all food places** surrouding the first area within a **500m** radius<jupyter_code>#hide_input
md(f"... which happens to be {bh_data.loc[0,'Area']}")
radius = 500
lat, lng = bh_data[['Latitude', 'Longitude']].iloc[0].values

url = 'https://api.foursquare.com/v2/venues/search?&client_id={}&client_secret={}&v={}&ll={},{}&categoryId={}&radius={}&limit={}'.format(CLIENT_ID, CLIENT_SECRET, VERSION, lat, lng, food_categoryId, radius, LIMIT)
results = requests.get(url).json()<jupyter_output><empty_output><jupyter_text>Looking at the first food place in the `results` json, we get this output:<jupyter_code>results['response']['venues'][0]
#hide_input
venue = results['response']['venues'][0]
name = venue['name']
category = venue['categories'][0]['name']
md(f"Thie first venue is {name}, and has a category: `{category}`")<jupyter_output><empty_output><jupyter_text>So now let's build a helpful function to **extract the category** of each food place. We'll use the same area as an example.<jupyter_code># function that extracts the category of the restaurant
def get_category_type(row):
    try:
        categories_list = row['categories']
    except:
        categories_list = row['venue.categories']
        
    if len(categories_list) == 0:
        return None
    else:
        return categories_list[0]['name']
    
venues = results['response']['venues']
    
nearby_food = pd.json_normalize(venues) # flatten JSON

# filter columns
filtered_columns = ['name', 'categories', 'location.lat', 'location.lng']
nearby_food = nearby_food.loc[:, filtered_columns]

# filter the category for each row
nearby_food['categories'] = nearby_food.apply(get_category_type, axis=1)

# clean columns
nearby_food.columns = [col.split(".")[-1] for col in nearby_food.columns]

nearby_food.head()
#hide_input
md(f"These are some of them, in total it returns **{nearby_food.shape[0]}** "
"food places around "
f"**{bh_data.loc[0,'Area']}**.")<jupyter_output><empty_output><jupyter_text>Exploring All Areas<jupyter_code>#hide_input
md(f"We've got still got {bh_data.shape[0]} places to explore, so let's create a function to do this task much faster.")
def getNearbyFoods(names, latitudes, longitudes, radius=500):
    food_categoryId = "4d4b7105d754a06374d81259"
    foods_list=[]
    for name, lat, lng in zip(names, latitudes, longitudes):
        print(name)

            
        # create the API request URL
        url = 'https://api.foursquare.com/v2/venues/search?&client_id={}&client_secret={}&v={}&ll={},{}&categoryId={}&radius={}&limit={}'.format(
            CLIENT_ID, 
            CLIENT_SECRET, 
            VERSION, 
            lat, 
            lng, 
            food_categoryId,
            radius, 
            LIMIT)
            
        # make the GET request
        try:
            results = requests.get(url).json()["response"]['venues']
        except:
            print(results)
            raise KeyError
        
        venue_list = []
        # return only relevant information for each nearby food place
        for v in results:
            vname, vlat, vlng = v['name'], v['location']['lat'], v['location']['lng']
            try:
                vcategory = v['categories'][0]['name']
                venue_list.append((name, 
                                    lat, 
                                    lng,
                                    vname, 
                                    vlat,
                                    vlng,
                                    vcategory))
            except:
                continue
        foods_list.append(venue_list)
    nearby_foods = pd.DataFrame([item for venue_list in foods_list for item in venue_list])
    nearby_foods.columns = ['Area', 
                  'Area Latitude', 
                  'Area Longitude',
                  'Venue', 
                  'Venue Latitude', 
                  'Venue Longitude', 
                  'Venue Category']
    
    return(nearby_foods)<jupyter_output><empty_output><jupyter_text>We run the above function on each area and create a new dataframe called `bh_foods`.<jupyter_code>#hide_output
bh_food = getNearbyFoods(bh_data['Area'], bh_data['Latitude'], 
                                   bh_data['Longitude'], 500)

bh_food.head()
#hide_input
md(f"This gives us a whopping {bh_food.shape[0]} food places")<jupyter_output><empty_output><jupyter_text>We can study the count for each area...<jupyter_code>bh_food.groupby('Area').count().head()<jupyter_output><empty_output><jupyter_text>We've trimmed out the remaining areas for brevity's sake.<jupyter_code>#hide_input
md(f"What's interesting to notice from this data, is that there are "
    f"{len(bh_food['Venue Category'].unique())} unique categories for food.")
#hide_input
md(f"Some of them include: {', '.join(bh_food['Venue Category'].unique()[:5])} "
  "and so on.")<jupyter_output><empty_output><jupyter_text>Most Common Food PlacesOur solution relies on segmenting areas based on the most common type of food places within that area. This gives us an idea about the kind of area it is from a culinary point-of-view, and allowing us to make judgments on whether the food is ideal to our taste or not. We also want to factor in the total number of food places within an area since some places in Bahrain may not be ideal to live in if they don't even have enough places to eat.Using the dataframe `bh_food`, we form a one-hot encoding of the `Venue Category` field that produces new columns for each category. Each record in this table corresponds to a certain venue and a `1` is placed in the category field for that area. The only other field that is retained is the area name. We will call this `bh_onehot`.<jupyter_code># one hot encoding
bh_onehot = pd.get_dummies(bh_food[['Venue Category']], prefix="", prefix_sep="")

# add neighborhood column back to dataframe
bh_onehot = pd.concat([bh_food[['Area']], bh_onehot], axis=1) 

bh_onehot.head()
#hide
# And let's examine the new dataframe size.
bh_onehot.shape<jupyter_output><empty_output><jupyter_text>Now, let's group rows by area and by taking the mean of the frequency of occurrence of each category, along with the number of food places surrouding it (`NumberOfFoodPlaces`). Looking at the number of food places is significant considering that some areas have fewer restaurants, and could be a valid factor to segment, if a "foodie" is looking for a place to stay.<jupyter_code>bh_grouped = bh_onehot.groupby(['Area']).mean().reset_index()
bh_grouped['NumberOfFoodPlaces'] = bh_onehot[['Area']].value_counts(sort=False).values
bh_grouped.head()<jupyter_output><empty_output><jupyter_text>Let's call this this `bh_grouped`. Now that we have this processed information, we can analyze this data more clearly by reordering it so that only the 10 most common type of food places for an area are retained.<jupyter_code>#hide
# Let's confirm the new size
bh_grouped.shape
# Function to sort venues by most common ones
def return_most_common_venues(row, num_top_venues):
    row_categories = row.iloc[1:-1]
    row_categories_sorted = row_categories.sort_values(ascending=False)
    
    return row_categories_sorted.index.values[0:num_top_venues]

num_top_venues = 10

indicators = ['st', 'nd', 'rd']

# create columns according to number of top venues
columns = ['Area', 'NumberOfFoodPlaces']
for ind in np.arange(num_top_venues):
    try:
        columns.append('{}{} Most Common Food Place'.format(ind+1, indicators[ind]))
    except:
        columns.append('{}th Most Common Food Place'.format(ind+1))

# create a new dataframe
foods_sorted = pd.DataFrame(columns=columns)
foods_sorted[['Area','NumberOfFoodPlaces']] = bh_grouped[['Area','NumberOfFoodPlaces']]

for ind in np.arange(bh_grouped.shape[0]):
    foods_sorted.iloc[ind, 2:] = return_most_common_venues(bh_grouped.iloc[ind, :], num_top_venues)

# Get the count    
foods_sorted.head()<jupyter_output><empty_output><jupyter_text>Let's call this table `foods_sorted`.  Cluster Areas Now we are ready for further analysis and clustering. We will use the `bh_grouped` dataframe since it contains the necessary numerical values for machine learning.Our feature set is comprised of all the food categories (10 features).We are excluding the `NumberOfFoodPlaces` feature as input to the ML model, since our problem requires segmenting areas by the type of food available. This quantity is only relevant to us to finally decide whether to live in an area or not.A more concrete reason to exclude it, is the fact that there are all sorts of factors involved that we're **neglecting** due to lack of data, such as living costs, access to public transport etc. This is a foodie's guide to finding a place, and this venture shouldn't be bogged-down by the fact that there are sometimes fewer number of restaurants than one would expect.Our target value will be **cluster labels**.For our machine learning analysis, we will use the simplest clustering algorithm to separate the areas which is **K-Means Clustering**; an unsupervised machine learning approach to serve our purpose. We'll use the popular machine learning library Sci-Kit Learn to do that in python. We'll run _k_-means to group the areas into 5 clusters. We pick this number for the sake of examination. We'll fit the model on the entire data to learn these clusters.<jupyter_code># set number of clusters
kclusters = 5

bh_grouped_clustering = bh_grouped.drop(['Area','NumberOfFoodPlaces'], 1)

# run k-means clustering
kmeans = KMeans(n_clusters=kclusters, random_state=0).fit(bh_grouped_clustering)

# check cluster labels generated for each row in the dataframe
kmeans.labels_[0:10]<jupyter_output><empty_output><jupyter_text>Let's create a new dataframe `bh_merged` that includes the cluster as well as the top 10 food places for each area.<jupyter_code># add clustering labels
try:
    foods_sorted.insert(0, 'Cluster Labels', kmeans.labels_)
except:
    # Allows me to retry if the Cluster Labels column exists
    foods_sorted['Cluster Labels'] = kmeans.labels_

bh_merged = bh_data

# merge bh_grouped with bh_data to add latitude/longitude for each neighborhood
bh_merged = bh_merged.join(foods_sorted.set_index('Area'), on='Area')
bh_merged.dropna(how='any', axis=0, inplace=True)
bh_merged['Cluster Labels'] = bh_merged['Cluster Labels'].astype(np.int32)
bh_merged.head() # check the last columns!<jupyter_output><empty_output><jupyter_text>Finally, let's visualize the resulting clusters<jupyter_code>#hide_input
# create map
map_clusters = folium.Map(location=[latitude, longitude], zoom_start=11)

# set color scheme for th12e clusters
x = np.arange(kclusters)
ys = [i + x + (i*x)**2 for i in range(kclusters)]
colors_array = cm.rainbow(np.linspace(0, 1, len(ys)))
rainbow = [colors.rgb2hex(i) for i in colors_array]

# add markers to the map
markers_colors = []
for lat, lon, poi, cluster in zip(bh_merged['Latitude'], bh_merged['Longitude'], bh_merged['Area'], bh_merged['Cluster Labels']):
    label = folium.Popup(str(poi) + ' Cluster ' + str(cluster), parse_html=True)
    folium.CircleMarker(
        [lat, lon],
        radius=5,
        popup=label,
        color=rainbow[cluster-1],
        fill=True,
        fill_color=rainbow[cluster-1],
        fill_opacity=0.7).add_to(map_clusters)
       
render(map_clusters)
#hide
map_clusters<jupyter_output><empty_output><jupyter_text>Examine Clusters & Final Conclusion Now, we can examine & determine the discriminating characteristics of each cluster.  Cluster 1<jupyter_code>cluster1 = bh_merged.loc[bh_merged['Cluster Labels'] == 0, bh_merged.columns[[0] + list(range(4, bh_merged.shape[1]))]]
cluster1
#hide_input
md(f"This cluster has {cluster1.shape[0]} areas")<jupyter_output><empty_output><jupyter_text>Cluster 2<jupyter_code>cluster2 = bh_merged.loc[bh_merged['Cluster Labels'] == 1, bh_merged.columns[[0] + list(range(4, bh_merged.shape[1]))]]
cluster2
#hide_input
md(f"This cluster has {cluster2.shape[0]} areas")<jupyter_output><empty_output><jupyter_text>Cluster 3<jupyter_code>cluster3 = bh_merged.loc[bh_merged['Cluster Labels'] == 2, bh_merged.columns[[0] + list(range(4, bh_merged.shape[1]))]]
cluster3
#hide_input
md(f"This cluster has {cluster3.shape[0]} areas")<jupyter_output><empty_output><jupyter_text>Cluster 4<jupyter_code>cluster4 = bh_merged.loc[bh_merged['Cluster Labels'] == 3, bh_merged.columns[[0] + list(range(4, bh_merged.shape[1]))]]
cluster4
#hide_input
md(f"This cluster has {cluster4.shape[0]} area")<jupyter_output><empty_output><jupyter_text>Cluster 5<jupyter_code>cluster5 = bh_merged.loc[bh_merged['Cluster Labels'] == 4, bh_merged.columns[[0] + list(range(4, bh_merged.shape[1]))]]
cluster5
#hide_input
md(f"This cluster has {cluster5.shape[0]} areas")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09879638916750251, 'fraction_numerical': 0.013206285523236376, 'mean_word_length': 4.117408041060735, 'pattern_counts': {'":': 1, '<': 104, '<?xml version=': 0, '>': 105, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '201691', 'license': 'Apache-2.0', 'n_tokens_mistral': 7797, 'n_tokens_neox': 7259, 'n_words': 2556, 'path': '_notebooks/2021-09-10-bahrain-area-segmentation.ipynb', 'repo_name': 'isados/blog', 'size': '23990'}