<jupyter_start><jupyter_text>Neighborhoods in TorontoApplied Data Science Capstone Week 3 Assignment  1.  First Part of the Assginment: Scrape Wikipedia  Import the relevant libraries<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from urllib.request import urlopen
from bs4 import BeautifulSoup
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Fetch the HTML content of the url and parse it with BeautifulSoup, find and reference the table with the codes<jupyter_code>url = 'https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M'
html = urlopen(url)
soup = BeautifulSoup(html, 'lxml')
table = soup.find_all('table', class_='wikitable')[0]<jupyter_output><empty_output><jupyter_text>Fill the initial dataframe with all uncleansed table rowsI know this causes triple iteration through the data rows. But it is more readable than a big for loop.- Preventing the recursive search in `findChildren` will only return direct children. Not the anchors **a** contained in links.- `get_text()` strips the HTML tags- `strip()` strips whitespaces from the remaining string<jupyter_code>df = pd.DataFrame({
    'PostalCode': [row.findChildren(recursive=False)[0].get_text().strip() for row in table.select('tr')],
    'Borough': [row.findChildren(recursive=False)[1].get_text().strip() for row in table.select('tr')],
    'Neighborhood': [row.findChildren(recursive=False)[2].get_text().strip() for row in table.select('tr')]
})<jupyter_output><empty_output><jupyter_text>Drop the first row containing the header information<jupyter_code>df = df.iloc[1:]<jupyter_output><empty_output><jupyter_text>Drop the 'Not assigned' boroughs:<jupyter_code>df = df[df['Borough']!='Not assigned']<jupyter_output><empty_output><jupyter_text>Replace the Neighborhood if it is not assigned<jupyter_code>df['Neighborhood'] = df.apply(lambda x: x['Borough'] if x['Neighborhood'] == 'Not assigned' else x['Neighborhood'], axis=1)<jupyter_output><empty_output><jupyter_text>Group By Postal Code and Borough and concatenate Neighborhood<jupyter_code>df = df.groupby(['PostalCode', 'Borough'])['Neighborhood'].apply(lambda x: ', '.join(x)).reset_index()
df.head(12)
df.shape<jupyter_output><empty_output><jupyter_text>2. Second part of the Assignment: Fetch Geo location data  Unable to fetch locations from geocoder... therefore I had to use the CSV     !pip install geocoder     import geocoder  import geocoder          def fetchFromGeoCoder(pc):         lat_lng_coords = None         while(lat_lng_coords is None):             g = geocoder.google('{}, Toronto, Ontario'.format(pc))             lat_lng_coords = g.latlng         return lat_lng_coords          df['latlng'] = df.apply(lambda x: fetchFromGeoCoder(x['PostalCode']), axis=1)  Fetch the geospatial data<jupyter_code>locations = pd.read_csv('http://cocl.us/Geospatial_data')<jupyter_output><empty_output><jupyter_text>Prepare the PC column name for the merge<jupyter_code>locations.rename(columns={'Postal Code':'PostalCode'}, inplace=True)<jupyter_output><empty_output><jupyter_text>Merge the two data sets<jupyter_code>df = pd.merge(df, locations, how='left', on='PostalCode')
df.head(12)<jupyter_output><empty_output><jupyter_text>3. Third part of the Assignment: Clustering the data<jupyter_code>import matplotlib.cm as cm
import matplotlib.colors as colors
from sklearn.cluster import KMeans
!conda install -c conda-forge folium=0.5.0 geopy --yes
import folium
from geopy.geocoders import Nominatim
import requests<jupyter_output>Fetching package metadata .............
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /opt/conda/envs/DSX-Python35:
#
folium                    0.5.0                      py_0    conda-forge
geopy                     1.18.1                     py_0    conda-forge<jupyter_text>3.1 Download and Explore the Dataset  Fetch the center of Toronto<jupyter_code>geolocator = Nominatim(user_agent="ds-learning")
location = geolocator.geocode('Toronto, Ontario')
latitude = location.latitude
longitude = location.longitude
print('The geograpical coordinate of Toronto are {}, {}.'.format(latitude, longitude))<jupyter_output>The geograpical coordinate of Toronto are 43.653963, -79.387207.<jupyter_text>Display the map of Toronto<jupyter_code>map_toronto = folium.Map(location=[latitude, longitude], zoom_start=10)

for index, row in df.iterrows():
    label = '{}: {} ({})'.format(row['Borough'], row['Neighborhood'], row['PostalCode'])
    label = folium.Popup(label, parse_html=True)
    folium.CircleMarker(
        [row['Latitude'], row['Longitude']],
        radius=5,
        popup=label,
        color='blue',
        fill=True,
        fill_color='#3186cc',
        fill_opacity=0.7,
        parse_html=False).add_to(map_toronto)  
    
map_toronto<jupyter_output><empty_output><jupyter_text>Subselection of Boroughs containing Toronto in their names<jupyter_code>df['Borough'].unique()
toronto_data = df[df['Borough'].str.contains('Toronto')].reset_index(drop=True)
toronto_data.head()
map_toronto_center = folium.Map(location=[latitude, longitude], zoom_start=12)

for index, row in toronto_data.iterrows():
    label = folium.Popup(row['Neighborhood'], parse_html=True)
    folium.CircleMarker(
        [row['Latitude'], row['Longitude']],
        radius=5,
        popup=label,
        color='blue',
        fill=True,
        fill_color='#3186cc',
        fill_opacity=0.7,
        parse_html=False).add_to(map_toronto_center)  
    
map_toronto_center<jupyter_output><empty_output><jupyter_text>3.2 Explore Neighborhoods in Toronto  Define secret Foursquare API parameters<jupyter_code># The code was removed by Watson Studio for sharing.<jupyter_output><empty_output><jupyter_text>Define not so secret Foursquare API parameters<jupyter_code>LIMIT = 100
VERSION = '20180605'<jupyter_output><empty_output><jupyter_text>Define previously Foursquare nearby venues fetching function<jupyter_code>def getNearbyVenues(names, latitudes, longitudes, radius=500):
    
    venues_list=[]
    for name, lat, lng in zip(names, latitudes, longitudes):
        # create the API request URL
        url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}'.format(
            CLIENT_ID, 
            CLIENT_SECRET, 
            VERSION, 
            lat, 
            lng, 
            radius, 
            LIMIT)
            
        # make the GET request
        results = requests.get(url).json()["response"]['groups'][0]['items']
        
        # return only relevant information for each nearby venue
        venues_list.append([(
            name, 
            lat, 
            lng, 
            v['venue']['name'], 
            v['venue']['location']['lat'], 
            v['venue']['location']['lng'],  
            v['venue']['categories'][0]['name']) for v in results])

    nearby_venues = pd.DataFrame([item for venue_list in venues_list for item in venue_list])
    nearby_venues.columns = ['Neighborhood', 
                  'Neighborhood Latitude', 
                  'Neighborhood Longitude', 
                  'Venue', 
                  'Venue Latitude', 
                  'Venue Longitude', 
                  'Venue Category']
    
    return(nearby_venues)
toronto_venues = getNearbyVenues(names=toronto_data['Neighborhood'],
                                   latitudes=toronto_data['Latitude'],
                                   longitudes=toronto_data['Longitude']
                                  )
print(toronto_venues.shape)
toronto_venues.head()
toronto_venues.groupby('Neighborhood').count()
print('There are {} uniques categories.'.format(len(toronto_venues['Venue Category'].unique())))<jupyter_output>There are 239 uniques categories.<jupyter_text>3.3 Analyze Each Neighborhood<jupyter_code># one hot encoding
toronto_onehot = pd.get_dummies(toronto_venues[['Venue Category']], prefix="", prefix_sep="")

# There seems to be a category called 'Neighborhood':
toronto_onehot.rename(columns={'Neighborhood':'Neighborhood_Cat'}, inplace=True)

toronto_onehot = pd.concat([toronto_venues['Neighborhood'], toronto_onehot], axis=1)
toronto_onehot.head()
toronto_onehot.shape
toronto_grouped = toronto_onehot.groupby('Neighborhood').mean().reset_index()
toronto_grouped
toronto_grouped.shape<jupyter_output><empty_output><jupyter_text>Let's print each neighborhood along with the top 5 most common venues<jupyter_code>num_top_venues = 5

for hood in toronto_grouped['Neighborhood']:
    print("----"+hood+"----")
    temp = toronto_grouped[toronto_grouped['Neighborhood'] == hood].T.reset_index()
    temp.columns = ['venue','freq']
    temp = temp.iloc[1:]
    temp['freq'] = temp['freq'].astype(float)
    temp = temp.round({'freq': 2})
    print(temp.sort_values('freq', ascending=False).reset_index(drop=True).head(num_top_venues))
    print('\n')<jupyter_output>----Adelaide, King, Richmond----
                 venue  freq
0          Coffee Shop  0.06
1                 Café  0.05
2      Thai Restaurant  0.04
3           Steakhouse  0.04
4  American Restaurant  0.04


----Berczy Park----
          venue  freq
0   Coffee Shop  0.07
1  Cocktail Bar  0.05
2    Restaurant  0.05
3        Bakery  0.04
4    Steakhouse  0.04


----Brockton, Exhibition Place, Parkdale Village----
                  venue  freq
0           Coffee Shop  0.14
1        Breakfast Spot  0.10
2                  Café  0.10
3                   Gym  0.05
4  Gym / Fitness Center  0.05


----Business reply mail Processing Centre969 Eastern----
                venue  freq
0  Light Rail Station  0.11
1         Yoga Studio  0.05
2       Garden Center  0.05
3          Smoke Shop  0.05
4       Auto Workshop  0.05


----CN Tower, Bathurst Quay, Island airport, Harbourfront West, King and Spadina, Railway Lands, South Niagara----
              venue  freq
0  Airport Terminal  0.14
1    Air[...]<jupyter_text>Let's put that into a pandas dataframeFirst, let's write a function to sort the venues in descending order.<jupyter_code>def return_most_common_venues(row, num_top_venues):
    row_categories = row.iloc[1:]
    row_categories_sorted = row_categories.sort_values(ascending=False)
    
    return row_categories_sorted.index.values[0:num_top_venues]<jupyter_output><empty_output><jupyter_text>Now let's create the new dataframe and display the top 10 venues for each neighborhood.<jupyter_code>num_top_venues = 10

indicators = ['st', 'nd', 'rd']

# create columns according to number of top venues
columns = ['Neighborhood']
for ind in np.arange(num_top_venues):
    try:
        columns.append('{}{} Most Common Venue'.format(ind+1, indicators[ind]))
    except:
        columns.append('{}th Most Common Venue'.format(ind+1))

# create a new dataframe
neighborhoods_venues_sorted = pd.DataFrame(columns=columns)
neighborhoods_venues_sorted['Neighborhood'] = toronto_grouped['Neighborhood']

for ind in np.arange(toronto_grouped.shape[0]):
    neighborhoods_venues_sorted.iloc[ind, 1:] = return_most_common_venues(toronto_grouped.iloc[ind, :], num_top_venues)

neighborhoods_venues_sorted.head()<jupyter_output><empty_output><jupyter_text>3.4 Cluster Neighborhoods  First find best k<jupyter_code>toronto_grouped_clustering = toronto_grouped.drop('Neighborhood', 1)
ssq = []
K = range(1, 15)
for k in K:
    kmeans = KMeans(n_clusters=k, random_state=3).fit(toronto_grouped_clustering)
    ssq.append(kmeans.inertia_)

plt.plot(K, ssq, 'bx-')
plt.xlabel('k')
plt.ylabel('Sum of squared distances')
plt.title('Elbow Method For Optimal k')
plt.show()<jupyter_output><empty_output><jupyter_text>k=6 seems to yield good enough results, re-run with k=6<jupyter_code>kclusters = 6
toronto_grouped_clustering = toronto_grouped.drop('Neighborhood', 1)
kmeans = KMeans(n_clusters=kclusters, random_state=3).fit(toronto_grouped_clustering)
print(kmeans.inertia_)
kmeans.labels_[0:10]<jupyter_output>1.32581617378<jupyter_text>Rejoin the datasets to create the map<jupyter_code>toronto_merged = toronto_data
toronto_merged['Cluster Labels'] = kmeans.labels_
toronto_merged = toronto_merged.join(neighborhoods_venues_sorted.set_index('Neighborhood'), on='Neighborhood')
toronto_merged.head() # check the last columns!<jupyter_output><empty_output><jupyter_text>Finally, let's visualize the resulting clusters<jupyter_code># create map
map_clusters = folium.Map(location=[latitude, longitude], zoom_start=12)

# set color scheme for the clusters
x = np.arange(kclusters)
ys = [i+x+(i*x)**2 for i in range(kclusters)]
colors_array = cm.rainbow(np.linspace(0, 1, len(ys)))
rainbow = [colors.rgb2hex(i) for i in colors_array]

# add markers to the map
markers_colors = []
for lat, lon, poi, cluster in zip(toronto_merged['Latitude'], toronto_merged['Longitude'], toronto_merged['Neighborhood'], toronto_merged['Cluster Labels']):
    label = folium.Popup(str(poi) + ' Cluster ' + str(cluster), parse_html=True)
    folium.CircleMarker(
        [lat, lon],
        radius=5,
        popup=label,
        color=rainbow[cluster-1],
        fill=True,
        fill_color=rainbow[cluster-1],
        fill_opacity=0.7).add_to(map_clusters)
       
map_clusters<jupyter_output><empty_output><jupyter_text>3.5 Examine Clusters<jupyter_code>toronto_merged.loc[toronto_merged['Cluster Labels'] == 0, toronto_merged.columns[[1] + list(range(5, toronto_merged.shape[1]))]]
toronto_merged.loc[toronto_merged['Cluster Labels'] == 1, toronto_merged.columns[[1] + list(range(5, toronto_merged.shape[1]))]]
toronto_merged.loc[toronto_merged['Cluster Labels'] == 2, toronto_merged.columns[[1] + list(range(5, toronto_merged.shape[1]))]]
toronto_merged.loc[toronto_merged['Cluster Labels'] == 3, toronto_merged.columns[[1] + list(range(5, toronto_merged.shape[1]))]]
toronto_merged.loc[toronto_merged['Cluster Labels'] == 4, toronto_merged.columns[[1] + list(range(5, toronto_merged.shape[1]))]]
toronto_merged.loc[toronto_merged['Cluster Labels'] == 5, toronto_merged.columns[[1] + list(range(5, toronto_merged.shape[1]))]]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12192354067664374, 'fraction_numerical': 0.01822824313781119, 'mean_word_length': 4.280898876404494, 'pattern_counts': {'":': 0, '<': 100, '<?xml version=': 0, '>': 100, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '338037', 'license': 'MIT', 'n_tokens_mistral': 4889, 'n_tokens_neox': 4602, 'n_words': 991, 'path': 'Neighborhoods in Toronto.ipynb', 'repo_name': 'dominicbosch/Coursera_Capston', 'size': '14099'}