<jupyter_start><jupyter_text>Coursera Capstone Notebook: Clustering of Rome Subway stations  IntroductionThe Rome Metro (Metropolitana di Roma) started operation in 1955. It has three lines: A (orange), B (blue) and C (green). It has 73 stations, the length of it is about 60 km. The lines A and B intersect at Termini Station, which is also the main train station in Rome. Annually it serves for 279 million of people (2012).We want to look at the areas surrounding metro stations and cluster them. The areas in the historical center of Rome have a lot of touristic attractions (Vatican, Colosseum, Trevi Fountain, Spanish Steps) and hotels. Some neighborhoods are in residential areas, other in commercial. The venues near to the station determine how people use it.Analysis of data can show the primary purpose of the station. This data is useful for commercial business (location to open a new business), for local government (can help them in city planning), and for subway development in the future (where open new stations).<jupyter_code>import pandas as pd
import requests
!pip install folium
import folium # map rendering library
from bs4 import BeautifulSoup
import html5lib
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline<jupyter_output>Requirement already satisfied: folium in /opt/conda/envs/Python36/lib/python3.6/site-packages (0.11.0)
Requirement already satisfied: numpy in /opt/conda/envs/Python36/lib/python3.6/site-packages (from folium) (1.15.4)
Requirement already satisfied: requests in /opt/conda/envs/Python36/lib/python3.6/site-packages (from folium) (2.21.0)
Requirement already satisfied: branca>=0.3.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from folium) (0.4.1)
Requirement already satisfied: jinja2>=2.9 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from folium) (2.10)
Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests->folium) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests->folium) (2020.4.5.1)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests->folium) (3.0.4)
Req[...]<jupyter_text>Data<jupyter_code>#get list of station from Wikipedia, create dataframe from it
#https://beenje.github.io/blog/posts/parsing-html-tables-in-python-with-pandas/
res = requests.get("https://it.wikipedia.org/wiki/Stazioni_della_metropolitana_di_Roma")
#if you don't have 'lxml' parser use default: 'html.parser'
soup = BeautifulSoup(res.content,'lxml')
table = soup.find_all('table')[0] 
df_stations = pd.read_html(str(table), header=0)

#create dataframe
df_stations = df_stations[0]

#drop rows Borough == Not assigned
#df_stations = df_stations[df_stations.Borough != 'Not assigned']

# Select all duplicate rows based on Postal code
#duplicateRowsDF = df[df.duplicated(['Postal code'])]
#duplicateRowsDF

# replace / to ,
#df = df.replace(to_replace=r'\s/', value=',', regex=True)

#drop columns Interscambi and Note
df_stations.drop(columns=['Linea', 'Comune', 'Municipio', 'Interscambi', 'Note'], inplace=True)

#rename column titles to English
df_stations.rename(columns={"Stazione": "Station", "Data apertura": "Opened", "Tipologia": "Type"}, inplace=True)

#save dataframe to csv file for easy access
df_stations.to_csv('Stations_wiki.csv')

df_stations.head(5)
#read our saved csv file and convert it to dataframe
!wget -O 'Stations_wiki.csv' https://raw.githubusercontent.com/riccione/Coursera_Capstone/master/Stations_wiki.csv
file = 'Stations_wiki.csv'
df_stations = pd.read_csv(file, index_col=0)
df_stations.head(5)
#remove day and month from Open Date
df_stations.replace({'Opened': r'^[0-9]+\º*\s[a-z]+\s'}, {'Opened': ''}, regex=True, inplace=True)

#clean some dates
df_stations.replace({'Opened': r'\[1\]'}, {'Opened': ''}, regex=True, inplace=True)
df_stations.replace({'Opened': '1980 (MA)12 maggio 2018 (MC)'}, '1980', inplace=True)
df_stations.replace({'Opened': '1955 (MB)16 febbraio 1980 (MA)'}, '1955', inplace=True)
df_stations.replace({'Station': r'Manzoni\s.*'}, {'Station': 'Manzoni'}, regex=True, inplace=True)
df_stations.replace({'Station': r'Furio\sCamillo.*'}, {'Station': 'Furio Camillo'}, regex=True, inplace=True)
df_stations.replace({'Station': r'Colli\sAlbani.*'}, {'Station': 'Colli Albani'}, regex=True, inplace=True)
df_stations.replace({'Station': r'Porta\sFurba.*'}, {'Station': 'Porta Furba - Quadraro'}, regex=True, inplace=True)
df_stations.replace({'Station': r'Sant\'Agnese\/Annibaliano'}, {'Station': 'Sant Agnese - Annibaliano'}, regex=True, inplace=True)

#translate types of stations
df_stations.replace({'Type': 'sotterranea'}, 'underground', inplace=True)
df_stations.replace({'Type': 'in superficie'}, 'surface', inplace=True)
df_stations.replace({'Type': 'in galleria artificiale'}, 'in artificial gallery', inplace=True)
df_stations.replace({'Type': 'sopraelevata'}, 'overground', inplace=True)

df_stations.head(5)
df_stations.shape
#csv = df_stations.to_csv(index=False)
#print(csv)
#get geo data for Rome subway stations
!wget -O 'Rome_Metro.csv' https://raw.githubusercontent.com/riccione/Coursera_Capstone/master/Rome_Metro.csv
file = 'Rome_Metro.csv'
df_geo = pd.read_csv(file,sep=",")
df_geo.columns = ['Station', 'Opened', 'Line', 'Latitude', 'Longitude']
#drop columns Opened
df_geo.drop(columns=['Opened'], inplace=True)
df_geo.head(5)
#sort dataframe
df_geo.sort_values(by=['Station'], inplace=True)
df_geo.reset_index(drop=True, inplace=True)
df_geo.head(5)
index = 0
for t in df_geo.duplicated('Station'):
        if (t):
            print(index, t)
        index += 1
#rename Basilica S. Paolo to Basilica San Paolo
df_geo.replace({'Station': 'Basilica S. Paolo'}, 'Basilica San Paolo', inplace=True)
#rename Manzoni – Museo della Liberazione to Manzoni
df_geo.replace({'Station': r'Manzoni\s.*'}, {'Station': 'Manzoni'}, regex=True, inplace=True)
df_geo.replace({'Station': r'Ponte\sLungo.*'}, {'Station': 'Ponte Lungo'}, regex=True, inplace=True)
df_geo.replace({'Station': r'Furio\sCamillo.*'}, {'Station': 'Furio Camillo'}, regex=True, inplace=True)
df_geo.replace({'Station': r'Colli\sAlbani.*'}, {'Station': 'Colli Albani'}, regex=True, inplace=True)
df_geo.replace({'Station': r'Numidio\sQuadrato.*'}, {'Station': 'Numidio Quadrato'}, regex=True, inplace=True)
df_geo.replace({'Station': r'Lucio\sSestio.*'}, {'Station': 'Lucio Sestio'}, regex=True, inplace=True)
df_geo.replace({'Station': r'Sant\'Agnese\s.\sAnnibaliano'}, {'Station': 'Sant Agnese - Annibaliano'}, regex=True, inplace=True)
df_geo[50:70]
#df_stations[20:30]
#remove San Giovanni and Termini, basically they are not duplicates
#these stations duplicates, because they exist on the crossing of two lines: San Giovanni A and C; Termini A and B. 
#The exits to different lines were build in different periods of time, however it is not important for our study.
df_geo[df_geo['Station'] == "San Giovanni"]
df_geo.iat[58,1] = 'AC'
counter = len(df_geo[df_geo['Station'] == "San Giovanni"])
if (counter > 1):
    df_geo.drop(df_geo.index[59], inplace=True)
df_geo[df_geo['Station'] == "San Giovanni"]
df_geo.reset_index(drop=True, inplace=True)
df_geo[df_geo['Station'] == 'Termini']
df_geo.iat[64,1] = 'AB'
counter = len(df_geo[df_geo['Station'] == "Termini"])
print(counter)
if (counter > 1):
    df_geo.drop(df_geo.index[65], inplace=True)
df_geo[df_geo['Station'] == "Termini"]
df_geo.shape
df_geo['Station'].value_counts()
df = df_stations.merge(df_geo, on="Station", how = 'inner')
df[20:30]
df.shape
line_a = df[df['Line']=='A']
line_ab = df[df['Line']=='AB']
line_ac = df[df['Line']=='AC']
line_a = line_a.append(line_ab)
line_a = line_a.append(line_ac)
line_a 
#reorder line_a stations in a right order
line_a = line_a.reindex([6,18,3,71,13,45,34,24,61,4,57,64,72,39,58,55,51,25,15,2,53,44,37,29,62,12,1])
line_a
line_b = df[df['Line']=='B']
line_b = line_b.append(line_ab)
line_b_left = line_b[line_b['Station'] != 'Rebibbia']
line_b_left = line_b[line_b['Station'] != 'Ponte Mammolo']
line_b_left = line_b[line_b['Station'] != 'Pietralata']
line_b_left = line_b[line_b['Station'] != 'Monti Tiburtini']
line_b_left = line_b[line_b['Station'] != 'Quintiliani']
line_b_left = line_b[line_b['Station'] != 'Tiburtina']
line_b_left
#reorder line_a stations in a right order
line_b_left = line_b_left.reindex([32,17,35,60,7,50,10,64,11,16,14,49,26,5,40,21,22,20,33])
line_b_left
def remove_stations_from_line(dataset, station_list):
    for station in station_list:
        dataset = dataset[dataset['Station'] != station]
    return dataset
line_b_right = remove_stations_from_line(line_b,['Jonio','Conca d\'Oro','Libia','Sant Agnese - Annibaliano','Policlinico','Castro Pretorio',
                                                 'Termini','Cavour','Colosseo','Circo Massimo','Piramide','Garbatella','Basilica San Paolo',
                                                 'Marconi','EUR Magliana','EUR Palasport','EUR Fermi','Laurentina'])
line_b_right
#reorder line_a stations in a right order
line_b_right = line_b_right.reindex([56,52,59,47,43,54,65,7])
line_b_right
line_c = df[df['Line']=='C']
line_c = line_c.append(line_ac)
line_c
#reorder line_a stations in a right order
line_c = line_c.reindex([58,36,48,38,63,27,41,46,0,69,68,28,70,66,67,31,19,9,8,23,30,42])
line_c
#lets get the point of city center: Barberini - Fontana di Trevi
latitude = df.loc[4,'Latitude']
longitude = df.loc[4, 'Longitude']

print(latitude)
print(longitude)
type(latitude)
type(longitude)<jupyter_output>41.903889
12.488889<jupyter_text>![Fontana di Trevi](https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/Panorama_of_Trevi_fountain_2015.jpg/640px-Panorama_of_Trevi_fountain_2015.jpg)<jupyter_code>rome_lat_long = [latitude, longitude]

# create map of Toronto using latitude and longitude values
rome_map = folium.Map(location=rome_lat_long, zoom_start=12)

#colors for markers
color_A = '#FFA500' #official color is orange
color_B = '#0000FF' #official color is blue
color_C = '#008000' #official color is green
color_other = '#FF0000'

#create empty list
lines=[]

#add markers to map
for lat, lng, label, line in zip(df['Latitude'],df['Longitude'],df['Station'],df['Line']):
    label = folium.Popup(label, parse_html=True)
    if(line=='A'):
        color_marker=color_A
    elif(line=='B'):
        color_marker=color_B
    elif(line=='C'):
        color_marker=color_C
    else:
        color_marker=color_other
    folium.CircleMarker(
        [lat, lng],
        radius=5,
        popup=label,
        color=color_marker,
        fill=True,
        fill_color=color_marker,
        fill_opacity=0.75,
        parse_html=False).add_to(rome_map)
    
#draw lines function
def draw_lines(dataset):
    lines=[]
    for lat, lng in zip(dataset['Latitude'],dataset['Longitude']):
        lines.append(tuple([lat, lng]))
    return lines
        
#add lines A
folium.PolyLine(draw_lines(line_a), color=color_A, weight=2.5, opacity=1).add_to(rome_map)   
#add lines B left
folium.PolyLine(draw_lines(line_b_left), color=color_B, weight=2.5, opacity=1).add_to(rome_map)   
#add lines B right
folium.PolyLine(draw_lines(line_b_right), color=color_B, weight=2.5, opacity=1).add_to(rome_map)
#add lines C right
folium.PolyLine(draw_lines(line_c), color=color_C, weight=2.5, opacity=1).add_to(rome_map)

rome_map
rome_map.save('Rome_map.html')
#save dataset to csv file
df.to_csv('Stations.csv')<jupyter_output><empty_output><jupyter_text>Foursquare API<jupyter_code>LIMIT = 100
radius = 1000
# @hidden_cell
CLIENT_ID = ''
CLIENT_SECRET = ''
VERSION = '20200425'<jupyter_output><empty_output><jupyter_text>Analysis of venues categories<jupyter_code>categories_url = 'https://api.foursquare.com/v2/venues/categories?client_id={}&client_secret={}&v={}'.format(
            CLIENT_ID, 
            CLIENT_SECRET, 
            VERSION)
            
# make the GET request
results = requests.get(categories_url).json()
len(results['response']['categories'])
categories_list = []
# Let's print only the top-level categories and their IDs and also add them to categories_list

def print_categories(categories, level=0, max_level=0):    
    if level>max_level: return
    out = ''
    out += '-'*level
    for category in categories:
        print(out + category['name'] + ' (' + category['id'] + ')')
        print_categories(category['categories'], level+1, max_level)
        categories_list.append((category['name'], category['id']))

print_categories(results['response']['categories'], 0, 0)<jupyter_output>Arts & Entertainment (4d4b7104d754a06370d81259)
College & University (4d4b7105d754a06372d81259)
Event (4d4b7105d754a06373d81259)
Food (4d4b7105d754a06374d81259)
Nightlife Spot (4d4b7105d754a06376d81259)
Outdoors & Recreation (4d4b7105d754a06377d81259)
Professional & Other Places (4d4b7105d754a06375d81259)
Residence (4e67e38e036454776db1fb3a)
Shop & Service (4d4b7105d754a06378d81259)
Travel & Transport (4d4b7105d754a06379d81259)<jupyter_text>Query number of venues in each category<jupyter_code>from pandas.io.json import json_normalize # tranform JSON file into a pandas dataframe
def get_venues_count(latitude, longitude, radius, categoryId):
    url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&categoryId={}'.format(
                CLIENT_ID, 
                CLIENT_SECRET, 
                VERSION,
                latitude, 
                longitude,
                radius,
                categoryId)

    # make the GET request
    results = requests.get(url).json()
    return results['response']['totalResults']
#Create new dataframe to store venues data
stations_venues_df = df.copy()
for c in categories_list:
    stations_venues_df[c[0]] = 0
stations_venues_df.head()
#SKIP - use too many request for sandbox
#Request number of venues
#for station, lat, lng in zip(df['Station'],df['Latitude'],df['Longitude']):
for i, row in stations_venues_df.iterrows():
    print(i)
    for c in categories_list:
        stations_venues_df.loc[i, c[0]] = get_venues_count(row.Latitude,row.Longitude, radius=1000, categoryId=c[1])
    stations_venues_df.to_csv('Stations_venues.csv')
#get saved data for Rome subway stations venues by categories
!wget -O 'Stations_venues.csv' https://raw.githubusercontent.com/riccione/Coursera_Capstone/master/Stations_venues.csv
stations_venues_df = pd.read_csv('Stations_venues.csv', index_col=0)
stations_venues_df.head()<jupyter_output><empty_output><jupyter_text>Visualize data<jupyter_code>stations_venues_df[['Arts & Entertainment','College & University','Event','Food','Nightlife Spot','Outdoors & Recreation','Professional & Other Places','Residence','Shop & Service','Travel & Transport']].describe()
tmp_df = stations_venues_df[['Arts & Entertainment','College & University','Event','Food','Nightlife Spot','Outdoors & Recreation','Professional & Other Places','Residence','Shop & Service','Travel & Transport']]
tmp_df.head(3)
import seaborn as sns
plt.figure(figsize=(20, 10))
plt.xticks(rotation='vertical')
sns.boxplot

ax = sns.boxplot(data = tmp_df)
#ax = sns.swarmplot(data=tmp_df, color="black", edgecolor="gray")
ax.set_ylabel('Count of venues', fontsize=20)
ax.set_xlabel('Venue category', fontsize=20)
ax.tick_params(labelsize=20)
plt.xticks(rotation=45, ha='right')

plt.show()<jupyter_output><empty_output><jupyter_text>As expected for Italy, the most frequent category is food. In Italy food is very important part of life.Event and Residence are not important categories, their number is not high and we can easily remove it from our dataframe.Reference: https://developer.foursquare.com/docs/build-with-foursquare/categories/<jupyter_code>if 'Event' in tmp_df.columns:
    tmp_df.drop(columns=['Event', 'Residence'], inplace=True)
    stations_venues_df.drop(columns=['Event', 'Residence'], inplace=True)
    categories_list = list(filter(lambda x: x[0] != 'Event', categories_list))
    categories_list = list(filter(lambda x: x[0] != 'Residence', categories_list))
#normalize data
from sklearn.preprocessing import MinMaxScaler

X = tmp_df.values
cluster_dataset = MinMaxScaler().fit_transform(X)
cluster_df.describe()
cluster_df = pd.DataFrame(cluster_dataset)
cluster_df.columns = [c[0] for c in categories_list]
cluster_df.head()
plt.figure(figsize=(20, 10))
#plt.xticks(rotation='vertical')
sns.boxplot

ax = sns.boxplot(data = cluster_df)
ax.set_ylabel('Count of venues (relative)', fontsize=20)
ax.set_xlabel('Venue category', fontsize=20)
ax.tick_params(labelsize=20)
plt.xticks(rotation=45, ha='right')

plt.show()<jupyter_output><empty_output><jupyter_text>Clustering Let's use Elbow method to find the optimal number of clusters<jupyter_code>cluster_df.head()
from sklearn.cluster import KMeans
wss = []
for i in range(1,10):
    fitx = KMeans(n_clusters=i).fit(cluster_df)
    wss.append(fitx.inertia_)

plt.figure(figsize=(11,8.5))
plt.plot(range(1,10), wss, 'bx-')
plt.xlabel('Number of clusters $k$')
plt.ylabel('Inertia')
plt.title('The Elbow Method showing the optimal $k$')
plt.show()<jupyter_output><empty_output><jupyter_text>So, optimal number of clusters based on 'Elbow method' is 3 for our dataframe. Note that, the elbow method is sometimes ambiguous. An alternative is the average silhouette method (Kaufman and Rousseeuw, 1990) which can be also used with any clustering approach.Let's try to use different approach to find optimal number of clusters - Silhouette method. The Silhouette Score reaches its global maximum at the optimal k.<jupyter_code>import numpy as np
from sklearn.metrics import silhouette_score

scores = [0]
for i in range(2,10):
    fitx = KMeans(n_clusters=i).fit(cluster_df)
    score = silhouette_score(cluster_df, fitx.labels_)
    scores.append(score)
    
plt.figure(figsize=(11,8.5))
plt.plot(range(1,10), np.array(scores), 'bx-')
plt.xlabel('Number of clusters $k$')
plt.ylabel('Average Silhouette')
plt.title('The Elbow Method showing the optimal $k$')
plt.show()<jupyter_output><empty_output><jupyter_text>The Silhouette method shows, that 2 is optimal choice for number of clusters. However observation and analysis showed that 2 is not enough, so we are going to use value - 3.<jupyter_code># set number of clusters
kclusters = 3

# run k-means clustering
kmeans = KMeans(n_clusters=kclusters, random_state=0).fit(cluster_df)

# check cluster labels generated for each row in the dataframe
kmeans.labels_[0:10]

# add clustering labels
cluster_labels = 'Cluster Labels'
if cluster_labels in cluster_df:
    cluster_df.drop(columns=[cluster_labels],inplace=True)
    
cluster_df.insert(0, cluster_labels, kmeans.labels_)

cluster_df.head(5)
clustered_df = pd.merge(df, cluster_df, right_index=True, left_index=True)
clustered_df.head(5)
clustered_df_boxplot=clustered_df[['Cluster Labels','Arts & Entertainment','College & University','Food','Nightlife Spot','Outdoors & Recreation','Professional & Other Places','Shop & Service','Travel & Transport']]
import matplotlib.ticker as ticker

fig, axes = plt.subplots(1,kclusters, figsize=(20, 10), sharey=True)
axes[0].set_ylabel('Count of venues (relative)', fontsize=25)

for k in range(kclusters):
    #Set same y axis limits
    axes[k].set_ylim(0,1.1)
    axes[k].xaxis.set_label_position('top')
    axes[k].set_xlabel('Cluster ' + str(k), fontsize=25)
    axes[k].tick_params(labelsize=20)
    plt.sca(axes[k])
    plt.xticks(rotation='vertical')
    #sns.boxplot(data = clustered_df[clustered_df['Cluster Labels']==k].drop('Cluster Labels',1), ax=axes[k])
    sns.boxplot(data = clustered_df_boxplot[clustered_df_boxplot['Cluster Labels']==k].drop('Cluster Labels',1), ax=axes[k])

plt.show()
#create map of Rome with all stations
map_rome_metro = folium.Map(location=[latitude, longitude], zoom_start=12)

#add markers
for i, station, latitude, longitude, cluster in zip(clustered_df.index,
                                         clustered_df['Station'],
                                         clustered_df['Latitude'],
                                         clustered_df['Longitude'],
                                         clustered_df['Cluster Labels']):
    #colors = sns.color_palette(None, kclusters).as_hex()
    colors=['blue','green','orange','red']
    #Calculate top 3 categories for each station
    station_series = clustered_df.iloc[i]
    top_categories_dict = {}
    for cat in categories_list:
        top_categories_dict[cat[0]] = station_series[cat[0]]
    top_categories = sorted(top_categories_dict.items(), key = lambda x: x[1], reverse=True)
    popup='<b>{}</b><br>Cluster {}<br>1. {} {}<br>2. {} {}<br>3. {} {}'.format(
        station,
        cluster,
        top_categories[0][0],
        "{0:.2f}".format(top_categories[0][1]),
        top_categories[1][0],
        "{0:.2f}".format(top_categories[1][1]),
        top_categories[2][0],
        "{0:.2f}".format(top_categories[2][1]))
    folium.CircleMarker(
        [latitude,longitude],
        fill=True,
        fill_opacity=0.5,
        popup=folium.Popup(popup, max_width = 300),
        radius=5,
        color=colors[cluster]
    ).add_to(map_rome_metro)
    
#add lines A
folium.PolyLine(draw_lines(line_a), color=color_A, weight=2.5, opacity=1).add_to(map_rome_metro)   
#add lines B left
folium.PolyLine(draw_lines(line_b_left), color=color_B, weight=2.5, opacity=1).add_to(map_rome_metro)   
#add lines B right
folium.PolyLine(draw_lines(line_b_right), color=color_B, weight=2.5, opacity=1).add_to(map_rome_metro)
#add lines C right
folium.PolyLine(draw_lines(line_c), color=color_C, weight=2.5, opacity=1).add_to(map_rome_metro)
    
map_rome_metro
clustered_df.head(5)
#get top 3 categories for a station
def getTopCategories(dataset):
    c=['Max1','Max2','Max3']
    dataset_max=(dataset[['Arts & Entertainment','College & University','Food','Nightlife Spot','Outdoors & Recreation','Professional & Other Places','Shop & Service','Travel & Transport']]
                     .apply(lambda x: pd.Series(x.nlargest(3).index,index=c),axis=1))
    dataset_max = pd.merge(dataset[['Station']], dataset_max,right_index=True, left_index=True)
    return dataset_max
# Cluster 1
cluster1_df = clustered_df.loc[clustered_df['Cluster Labels'] == 0, clustered_df.columns[[0] + list(range(5, clustered_df.shape[1]))]]
cluster1_df
getTopCategories(cluster1_df)
# Cluster 2
cluster2_df=clustered_df.loc[clustered_df['Cluster Labels'] == 1, clustered_df.columns[[0] + list(range(5, clustered_df.shape[1]))]]
cluster2_df
getTopCategories(cluster2_df)
# Cluster 3
cluster3_df=clustered_df.loc[clustered_df['Cluster Labels'] == 2, clustered_df.columns[[0] + list(range(5, clustered_df.shape[1]))]]
cluster3_df
getTopCategories(cluster3_df)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14289469073068162, 'fraction_numerical': 0.032854389346416685, 'mean_word_length': 5.5095523239235815, 'pattern_counts': {'":': 3, '<': 50, '<?xml version=': 0, '>': 59, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '37095', 'license': 'MIT', 'n_tokens_mistral': 8521, 'n_tokens_neox': 7957, 'n_words': 1730, 'path': 'Capstone_final.ipynb', 'repo_name': 'riccione/Coursera_Capstone', 'size': '22828'}