<jupyter_start><jupyter_text>Now that we have built a dataframe of the postal code of each neighborhood  along with the borough name and neighborhood name,  in order to utilize the Foursquare location data,  we need to get the latitude and the longitude coordinates of each neighborhood.  Part 0: Extracting the table again<jupyter_code>import pandas as pd
import numpy as np
from bs4 import BeautifulSoup
import requests

wikipedia_link='https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M'
raw_wikipedia_page= requests.get(wikipedia_link).text

soup = BeautifulSoup(raw_wikipedia_page,'lxml')
#print(soup.prettify())

table = soup.find('table')

Postcode      = []
Borough       = []
Neighbourhood = []

# print(table)

for tr_cell in table.find_all('tr'):
    
    counter = 1
    Postcode_var      = -1
    Borough_var       = -1
    Neighbourhood_var = -1
    
    for td_cell in tr_cell.find_all('td'):
        if counter == 1: 
            Postcode_var = td_cell.text
        if counter == 2: 
            Borough_var = td_cell.text
            tag_a_Borough = td_cell.find('a')
            
        if counter == 3: 
            Neighbourhood_var = str(td_cell.text).strip()
            tag_a_Neighbourhood = td_cell.find('a')
            
        counter +=1
        
    if (Postcode_var == 'Not assigned' or Borough_var == 'Not assigned' or Neighbourhood_var == 'Not assigned'): 
        continue
    try:
        if ((tag_a_Borough is None) or (tag_a_Neighbourhood is None)):
            continue
    except:
        pass
    if(Postcode_var == -1 or Borough_var == -1 or Neighbourhood_var == -1):
        continue
        
    Postcode.append(Postcode_var)
    Borough.append(Borough_var)
    Neighbourhood.append(Neighbourhood_var)
    
unique_p = set(Postcode)
# print('num of unique Postal codes:', len(unique_p))
Postcode_u      = []
Borough_u       = []
Neighbourhood_u = []


for postcode_unique_element in unique_p:
    p_var = ''; b_var = ''; n_var = ''; 
    for postcode_idx, postcode_element in enumerate(Postcode):
        if postcode_unique_element == postcode_element:
            p_var = postcode_element;
            b_var = Borough[postcode_idx]
            if n_var == '': 
                n_var = Neighbourhood[postcode_idx]
            else:
                n_var = n_var + ', ' + Neighbourhood[postcode_idx]
    Postcode_u.append(p_var)
    Borough_u.append(b_var)
    Neighbourhood_u.append(n_var)<jupyter_output><empty_output><jupyter_text>Part 1: Pre-processing - Installing geocoder<jupyter_code># !pip install geocoder
"""
Collecting geocoder
  Downloading https://files.pythonhosted.org/packages/4f/6b/13166c909ad2f2d76b929a4227c952630ebaf0d729f6317eb09cbceccbab/geocoder-1.38.1-py2.py3-none-any.whl (98kB)
Requirement already satisfied: click in c:\programdata\anaconda3\lib\site-packages (from geocoder) (6.7)
Requirement already satisfied: six in c:\programdata\anaconda3\lib\site-packages (from geocoder) (1.11.0)
Collecting ratelim (from geocoder)
  Downloading https://files.pythonhosted.org/packages/f2/98/7e6d147fd16a10a5f821db6e25f192265d6ecca3d82957a4fdd592cad49c/ratelim-0.1.6-py2.py3-none-any.whl
Requirement already satisfied: requests in c:\programdata\anaconda3\lib\site-packages (from geocoder) (2.19.1)
Requirement already satisfied: future in c:\programdata\anaconda3\lib\site-packages (from geocoder) (0.16.0)
Requirement already satisfied: decorator in c:\programdata\anaconda3\lib\site-packages (from ratelim->geocoder) (4.3.0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\programdata\anaconda3\lib\site-packages (from requests->geocoder) (3.0.4)
Requirement already satisfied: idna<2.8,>=2.5 in c:\programdata\anaconda3\lib\site-packages (from requests->geocoder) (2.7)
Requirement already satisfied: urllib3<1.24,>=1.21.1 in c:\programdata\anaconda3\lib\site-packages (from requests->geocoder) (1.23)
Requirement already satisfied: certifi>=2017.4.17 in c:\programdata\anaconda3\lib\site-packages (from requests->geocoder) (2018.8.24)
Installing collected packages: ratelim, geocoder
Successfully installed geocoder-1.38.1 ratelim-0.1.6
"""

print('geocoder has been installed before.')
import geocoder
print('geocoder has been successfully imported.')<jupyter_output>geocoder has been installed before.
geocoder has been successfully imported.<jupyter_text>Part 2: Processing - Extracting Latitude and Longtitude for each Postal code in Toronto<jupyter_code>latitude = []
longitude = []
for elem in Postcode_u:
# initialize your variable to None
    lat_lng_coords = None

# loop until you get the coordinates
    while (lat_lng_coords is None):
        g = geocoder.google('{}, Toronto, Ontario'.format(elem))
        lat_lng_coords = g.latlng
        # print(lat_lng_coords)

    latitude.append(lat_lng_coords[0])
    longitude.append(lat_lng_coords[1])
    print(elem, 'is RECEIVED.')
    # print(lat_lng_coords[0])
    # print(lat_lng_coords[1])<jupyter_output>M4X is RECEIVED.
M4G is RECEIVED.
M9M is RECEIVED.
M1W is RECEIVED.
M1J is RECEIVED.
M9P is RECEIVED.
M1G is RECEIVED.
M4J is RECEIVED.
M8Y is RECEIVED.
M5A is RECEIVED.
M1C is RECEIVED.
M5L is RECEIVED.
M8V is RECEIVED.
M6R is RECEIVED.
M8Z is RECEIVED.
M5M is RECEIVED.
M1N is RECEIVED.
M2H is RECEIVED.
M4C is RECEIVED.
M3K is RECEIVED.
M6P is RECEIVED.
M9C is RECEIVED.
M4B is RECEIVED.
M4E is RECEIVED.
M4K is RECEIVED.
M9R is RECEIVED.
M6L is RECEIVED.
M4V is RECEIVED.
M5X is RECEIVED.
M4L is RECEIVED.
M9L is RECEIVED.
M1R is RECEIVED.
M6A is RECEIVED.
M3C is RECEIVED.
M1V is RECEIVED.
M5K is RECEIVED.
M1H is RECEIVED.
M3H is RECEIVED.
M9N is RECEIVED.
M6B is RECEIVED.
M6M is RECEIVED.
M1T is RECEIVED.
M5B is RECEIVED.
M8W is RECEIVED.
M5P is RECEIVED.
M5T is RECEIVED.
M3L is RECEIVED.
M3A is RECEIVED.
M9A is RECEIVED.
M1M is RECEIVED.
M4H is RECEIVED.
M5C is RECEIVED.
M6J is RECEIVED.
M6S is RECEIVED.
M1E is RECEIVED.
M5S is RECEIVED.
M4T is RECEIVED.
M5H is RECEIVED.
M9B is RECEIVE[...]<jupyter_text>Part 3: Post-processing - Creating Pandas Dataframe<jupyter_code>toronto_dict = {'Postcode':Postcode_u, 'Borough':Borough_u, 'Neighbourhood':Neighbourhood_u,
              'Latitude': latitude, 'Longitude':longitude}
df_toronto = pd.DataFrame.from_dict(toronto_dict)
df_toronto.to_csv('toronto_base.csv')
df_toronto.head(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09969979459630274, 'fraction_numerical': 0.03823668826038869, 'mean_word_length': 3.7558226897069873, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 24, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '329704', 'license': 'MIT', 'n_tokens_mistral': 2531, 'n_tokens_neox': 2437, 'n_words': 557, 'path': 'Week 3 Capstone - 2.ipynb', 'repo_name': 'Soumya44/Coursera_Capstone', 'size': '6329'}