<jupyter_start><jupyter_text>Neighborhoods in Toronto Before we get the data and start exploring it, let's download all the dependencies that we will need.<jupyter_code>import numpy as np # library to handle data in a vectorized manner

import pandas as pd # library for data analsysis
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

import json # library to handle JSON files

!conda install -c conda-forge geopy --yes # uncomment this line if you haven't completed the Foursquare API lab
from geopy.geocoders import Nominatim # convert an address into latitude and longitude values

import requests # library to handle requests
from pandas.io.json import json_normalize # tranform JSON file into a pandas dataframe

# Matplotlib and associated plotting modules
import matplotlib.cm as cm
import matplotlib.colors as colors

# import k-means from clustering stage
from sklearn.cluster import KMeans

!conda install -c conda-forge folium=0.5.0 --yes # uncomment this line if you haven't completed the Foursquare API lab
import folium # map rendering library

print('Libraries imported.')<jupyter_output>Solving environment: done

## Package Plan ##

  environment location: /opt/conda/envs/Python36

  added / updated specs: 
    - geopy


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    python_abi-3.6             |          1_cp36m           4 KB  conda-forge
    geographiclib-1.50         |             py_0          34 KB  conda-forge
    ca-certificates-2020.4.5.2 |       hecda079_0         147 KB  conda-forge
    openssl-1.1.1g             |       h516909a_0         2.1 MB  conda-forge
    certifi-2020.4.5.2         |   py36h9f0ad1d_0         152 KB  conda-forge
    geopy-1.22.0               |     pyh9f0ad1d_0          63 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         2.5 MB

The following NEW packages will be INSTALLED:

    geographiclib:   1.50-py_0           conda-forge
    geopy:          [...]<jupyter_text>1. Download and Explore Dataset Collecting data (webscraping and CSV)<jupyter_code>#Merge two tables (webscrape and CSV) into a new one which include Neighbourhood - and longitude/latitude/postalcode data 
#Import Library
from bs4 import BeautifulSoup

#web scraping link
website_text = requests.get('https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M').text
soup = BeautifulSoup(website_text,'xml')

#reference data html table
table = soup.find('table',{'class':'wikitable sortable'})
table_rows = table.find_all('tr')

#collect records from table into dataframe
data = []
for row in table_rows:
    data.append([t.text.strip() for t in row.find_all('td')])

df = pd.DataFrame(data, columns=['PostalCode', 'Borough', 'Neighbourhood'])
df = df[~df['PostalCode'].isnull()]  # to filter out bad rows

#Only process the cells that have an assigned borough. Ignore cells with a borough that is Not assigned
df=df[df.Borough != 'Not assigned']
#df.shape
#Table with Postal Code combined with latitude and longitude
path='http://cocl.us/Geospatial_data'
df1 = pd.read_csv(path)

#rename the column name so its equal to the 'df' table 
df1.rename(columns={'Postal Code': 'PostalCode'}, inplace=True)

#df1.head()

#Merge tables
df3 = pd.merge(df, df1, on=['PostalCode', 'PostalCode'])

#display first 10 records
df3.head(10)
df4=df3
# We decide to work with only boroughs that contain the word "Toronto"


Toronto = df4[df4['Borough'].astype(str).str.contains('Toronto')]

Toronto.head(10)
Toronto.shape<jupyter_output><empty_output><jupyter_text>Check the dataset how many boroughs and neighborhoods there are<jupyter_code>print('The dataframe has {} boroughs and {} Toronto.'.format(
        len(Toronto['Borough'].unique()),
        Toronto.shape[0]
    )
)<jupyter_output>The dataframe has 4 boroughs and 39 Toronto.<jupyter_text>Use geopy library to get the latitude and longitude values of Toronto. In order to define an instance of the geocoder, we need to define a user_agent. We will name our agent ny_explorer, as shown below.<jupyter_code>address = 'Toronto'

geolocator = Nominatim(user_agent="ny_explorer")
location = geolocator.geocode(address)
latitude = location.latitude
longitude = location.longitude
print('The geograpical coordinate of Toronto are {}, {}.'.format(latitude, longitude))<jupyter_output>The geograpical coordinate of Toronto are 43.6534817, -79.3839347.<jupyter_text>Create a map of Toronto with neighborhoods superimposed on top.<jupyter_code># create map of Toronto using latitude and longitude values
map_toronto = folium.Map(location=[latitude, longitude], zoom_start=10)

# add markers to map
for lat, lng, borough, Neighbourhood in zip(Toronto['Latitude'], Toronto['Longitude'], Toronto['Borough'], Toronto['Neighbourhood']):
    label = '{}, {}'.format(Neighbourhood, borough)
    label = folium.Popup(label, parse_html=True)
    folium.CircleMarker(
        [lat, lng],
        radius=5,
        popup=label,
        color='blue',
        fill=True,
        fill_color='#3186cc',
        fill_opacity=0.7,
        parse_html=False).add_to(map_toronto)  
    
map_toronto<jupyter_output><empty_output><jupyter_text>**Folium** is a great visualization library. Feel free to zoom into the above map, and click on each circle mark to reveal the name of the neighborhood and its respective borough.<jupyter_code>Toronto.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11049127226921, 'fraction_numerical': 0.022314198308439805, 'mean_word_length': 3.4286852589641432, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '98815', 'license': 'MIT', 'n_tokens_mistral': 1801, 'n_tokens_neox': 1681, 'n_words': 526, 'path': 'Capstone Notebook_C.ipynb', 'repo_name': 'frits8l/Coursera_Capstone', 'size': '5557'}