<jupyter_start><jupyter_text>Applied Data Science Capstone Assignment 2 :Segmenting and Clustering Neighborhoods in Toronto - Part 1  For this assignment, you will be required to explore and cluster the neighborhoods in Toronto.1. Start by creating a new Notebook for this assignment.2. Use the Notebook to build the code to scrape the following Wikipedia page, https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M, in order to obtain the data that is in the table of postal codes and to transform the data into a pandas dataframe like the one shown below:<jupyter_code>import requests # library to handle requests
import lxml.html as lh
import bs4 as bs
import urllib.request

import numpy as np # library to handle data in a vectorized manner
import pandas as pd # library for data analsysis
url   = "https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M"<jupyter_output><empty_output><jupyter_text>The following two functions below allows us to scrape html tables using**Python BeautifulSoup** - python library for pulling data out of HTML and XML files, or**Python lxml** - simmilar library that supports scraping with xpath<jupyter_code>def scrape_table_bs4(cname,cols):
    page  = urllib.request.urlopen(url).read()
    soup  = bs.BeautifulSoup(page,'lxml')
    table = soup.find("table",class_=cname)
    header = [head.findAll(text=True)[0].strip() for head in table.find_all("th")]
    data   = [[td.findAll(text=True)[0].strip() for td in tr.find_all("td")]
              for tr in table.find_all("tr")]
    data    = [row for row in data if len(row) == cols]
    # Store data to this temporary dataframe
    raw_df = pd.DataFrame(data,columns=header)
    return raw_df

def scrape_table_lxml(XPATH,cols):
    page = requests.get(url)
    doc = lh.fromstring(page.content)
    table_content = doc.xpath(XPATH)
    for table in table_content:
        headers = [th.text_content().strip() for th in table.xpath('//th')]
        headers = headers[0:3]
        data    = [[td.text_content().strip() for td in tr.xpath('td')] 
                   for tr in table.xpath('//tbody/tr')]
        data    = [row for row in data if len(row) == cols]
        raw_df = pd.DataFrame(data,columns=headers)
        return raw_df
raw_TorontoPostalCodes = scrape_table_bs4("wikitable",3)
print(raw_TorontoPostalCodes.info(verbose=True))<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 288 entries, 0 to 287
Data columns (total 3 columns):
Postcode         288 non-null object
Borough          288 non-null object
Neighbourhood    288 non-null object
dtypes: object(3)
memory usage: 6.8+ KB
None<jupyter_text>Data Cleanup and re-grouping. The scraped wikipedia table contains some un-wanted entries and needs some cleanup. The following tasks will be performed:- Drop/ignore cells with un-assigned boroughs.- If a cell has a borough but a Not assigned neighborhood, then the neighborhood will be the same as the borough.- Group the table by PostalCode/Borough, Neighbourhood belonging to same borough will be combined in 'Neighbourhood' column as separated with 'comma'.<jupyter_code>TorontoPostalCodes=raw_TorontoPostalCodes[~raw_TorontoPostalCodes['Borough'].isin(['Not assigned'])]
TorontoPostalCodes=TorontoPostalCodes.sort_values(by=['Postcode','Borough','Neighbourhood'], ascending=[1,1,1]).reset_index(drop=True)
TorontoPostalCodes.loc[TorontoPostalCodes['Neighbourhood'] == 'Not assigned', ['Neighbourhood']] = TorontoPostalCodes['Borough']
check_unassigned_post_state_sample = TorontoPostalCodes.loc[TorontoPostalCodes['Borough'] == 'Queen\'s Park']
TorontoPostalCodes = TorontoPostalCodes.groupby(['Postcode','Borough'])['Neighbourhood'].apply(', '.join).reset_index()
TorontoPostalCodes.head()<jupyter_output><empty_output><jupyter_text>**Shape**<jupyter_code>TorontoPostalCodes.shape<jupyter_output><empty_output><jupyter_text>**Saving File**<jupyter_code>TorontoPostalCodes.to_csv('data/Toronto_part_1.csv',index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10423452768729642, 'fraction_numerical': 0.009020295665246805, 'mean_word_length': 5.218068535825545, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '448540', 'license': 'MIT', 'n_tokens_mistral': 1285, 'n_tokens_neox': 1250, 'n_words': 352, 'path': 'Week 2 - Segmentation and Clustering of Neighborhoods in Toronto_Part 1.ipynb', 'repo_name': 'ms22dn/Coursera_Capstone', 'size': '3991'}