<jupyter_start><jupyter_text>**Problem Statement**Identify top countries that are in direst need of aid. Your job is to categorize the countries using some socio-economic and health factors that determine the overall development of the country. Then you need to suggest the countries which the CEO needs to focus on the most. **Steps:**    1. Reading and understanding data2. Data Formatting3. Performing EDA        4. Data Preparation for clustering  - Outlier Treatment  - Performing Scaling  - Hopkins Check    5. Clustering using K-Means and Hierarchical Clustering  *   K-Means Clustering:    - Choose optimal value of K    - Run K-means using optimal K    - Visualize Clusters    - Cluster profiling using "gdpp","child_mort" & "income"  * Hierarchical Clustering:    - Plot dendrogram using single and complete linkage    - Choose optimal linkage    - Visualize Clusters    - Cluster profiling using "gdpp","child_mort" & "income"      6. Country Recommendation  **1. Reading and understanding data**<jupyter_code># Importing libraries
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import warnings

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

from scipy.cluster.hierarchy import linkage
from scipy.cluster.hierarchy import dendrogram
from scipy.cluster.hierarchy import cut_tree
%matplotlib inline

pd.set_option("display.max_rows", None,"display.max_columns", None)

warnings.simplefilter(action='ignore')
plt.style.use('seaborn')
# Importing Country-data.csv
country_df = pd.read_csv('/content/drive/MyDrive/Colab Notebooks/Upgrad_NGO_Clustering_Assignment/Country-data.csv')
# Reading head of the dataset
country_df.head()
# Reading data dictionary
data_dict = pd.read_csv('/content/drive/MyDrive/Colab Notebooks/Upgrad_NGO_Clustering_Assignment/data-dictionary+.csv')
data_dict<jupyter_output><empty_output><jupyter_text>Based on data dictionary "exports","health" and "import" spendings are mentioned as percentages of "gdpp". For consistency we will be converting these to quantative measures later.<jupyter_code># Columns summary
country_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 167 entries, 0 to 166
Data columns (total 10 columns):
 #   Column      Non-Null Count  Dtype  
---  ------      --------------  -----  
 0   country     167 non-null    object 
 1   child_mort  167 non-null    float64
 2   exports     167 non-null    float64
 3   health      167 non-null    float64
 4   imports     167 non-null    float64
 5   income      167 non-null    int64  
 6   inflation   167 non-null    float64
 7   life_expec  167 non-null    float64
 8   total_fer   167 non-null    float64
 9   gdpp        167 non-null    int64  
dtypes: float64(7), int64(2), object(1)
memory usage: 13.2+ KB<jupyter_text>Based on above summary "dtypes" seems normal<jupyter_code># Numerical columns statistical attributes
country_df.describe()
country_df.shape
# Checking for Null values
country_df.isnull().sum()<jupyter_output><empty_output><jupyter_text>There are no missing entries present in the dataset.  **2. Data formatting**<jupyter_code># Converting "exports","health" & "imports" into quantitive measures
country_df['exports'] = country_df['exports']*country_df['gdpp']/100
country_df['health'] = country_df['health']*country_df['gdpp']/100
country_df['imports'] = country_df['imports']*country_df['gdpp']/100
country_df.head()<jupyter_output><empty_output><jupyter_text>**3. Performing EDA**  **a) Univariate Analysis**<jupyter_code># Visualizing Numerical columns using Boxplots
cols = country_df.columns[1:].tolist()
r = c = 0
fig,ax = plt.subplots(5,2,figsize=(15,18))

for n,i in enumerate(cols):
    sns.boxplot(x=cols[n], data=country_df,ax=ax[r,c])
    c+=1
    if (n+1)%2==0:
        r+=1
        c=0
ax[r,c].axis("off")
plt.show()
# Visualizing Numerical columns using Distribution Plot 
cols = country_df.columns[1:].tolist()
r = c = 0
fig,ax = plt.subplots(5,2,figsize=(15,18))

for n,i in enumerate(cols):
    sns.distplot(country_df[cols[n]],ax=ax[r,c])
    c+=1
    if (n+1)%2==0:
        r+=1
        c=0
ax[r,c].axis("off")
plt.show()<jupyter_output><empty_output><jupyter_text>**Inferences**:- Since health, import and export expenses are proportions of gdpp, all of them follow similar trend.- There couple of countries which have child_mort greater than 150 and life_excpec below 50. We will perform bivariate analysis later to find if there is any correlation.- Based on above plots 'health','imports','exports','income' & 'gdpp' are having outliers, which need to be treated. As outliers impact cluster formation.  **c) Bivariate Analysis**<jupyter_code># Since all columns are numerical, we can plot scatterplots to gain insights.
cols = ["child_mort","gdpp","income","life_expec","total_fer","inflation"]
sns.pairplot(country_df[cols])
plt.show()<jupyter_output><empty_output><jupyter_text>**Inferences:**- "child_mort" and "gdpp" are showing an inverse relationship, which is expected due to better healthcare service. As healthcare expenses are directly proportional to GDP.- As can be seen "life_expec" and "child_mort" are having an inverse linear relationship, as smaller value of "child_mort" leads to better "life_expec".- Usually countries with higher GDP have better standard of living, which explains the linear relationship between "income" and "gdpp".  **4. Data Preparation for clustering**  **a) Outlier Treatment** Clustering algorithms use distance as a metric for cluster formation. Thus presence of outlier may drastically vary the final outcome. So, before proceeding any further we need to treat these outliers by capping them.<jupyter_code># Checking 0th, 5th, 95th, 99th and 100th percentile of numerical columns
country_df.quantile([0,0.05,0.95,0.99,1])<jupyter_output><empty_output><jupyter_text>As observed outliers are present in 'child_mort', 'inflation', 'health' 'imports', 'exports', 'income' & 'gdpp'. Since our aim is to identify the countries which need aid, we cannot aggrissively cap variables such as  'child_mort' & 'inflation' as it will affect our analysis.<jupyter_code># Capping 'health','imports','exports','income','gdpp' to 0.95 percentile

high_cap = ['health','imports','exports','income','gdpp']
cap = 0.95
for col in high_cap:
  country_df[col][country_df[col] > country_df[col].quantile(cap)] = country_df[col].quantile(cap)
country_df[["child_mort","inflation"]].quantile([0.99,1])
# We are capping "child_mort" & "inflation" only at 99th percentile for better cluster formation
country_df["inflation"][country_df["inflation"] > country_df["inflation"].quantile(0.99)] = country_df["inflation"].quantile(0.99)
country_df["child_mort"][country_df["child_mort"] > country_df["child_mort"].quantile(0.99)] = country_df["child_mort"].quantile(0.99)<jupyter_output><empty_output><jupyter_text>**Plotting boxplots after treating outliers**<jupyter_code># Visualizing Numerical columns using Boxplots
cols = country_df.columns[1:].tolist()
r = c = 0
fig,ax = plt.subplots(5,2,figsize=(15,18))

for n,i in enumerate(cols):
    sns.boxplot(x=cols[n], data=country_df,ax=ax[r,c])
    c+=1
    if (n+1)%2==0:
        r+=1
        c=0
ax[r,c].axis("off")
plt.show()<jupyter_output><empty_output><jupyter_text>**b) Feature Scaling**<jupyter_code># Performing scaling using Standard Scaler
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
country_df_scaled = scaler.fit_transform(country_df[country_df.columns[1:]])
country_df_scaled = pd.DataFrame(country_df_scaled,columns=country_df.columns[1:])
country_df_scaled.head()<jupyter_output><empty_output><jupyter_text>**c) Hopkins Test** The Hopkins statistic, is a statistic which gives a value which indicates the cluster tendency, in other words: how well the data can be clustered.- If the value is between {0.01, ...,0.3}, the data is regularly spaced.- If the value is around 0.5, it is random.- If the value is between {0.7, ..., 0.99}, it has a high tendency to cluster.<jupyter_code>#Calculating the Hopkins statistic
from sklearn.neighbors import NearestNeighbors
from random import sample
from numpy.random import uniform
import numpy as np
from math import isnan
 
def hopkins(X):
    d = X.shape[1]
    #d = len(vars) # columns
    n = len(X) # rows
    m = int(0.1 * n) 
    nbrs = NearestNeighbors(n_neighbors=1).fit(X.values)
 
    rand_X = sample(range(0, n, 1), m)
 
    ujd = []
    wjd = []
    for j in range(0, m):
        u_dist, _ = nbrs.kneighbors(uniform(np.amin(X,axis=0),np.amax(X,axis=0),d).reshape(1, -1), 2, return_distance=True)
        ujd.append(u_dist[0][1])
        w_dist, _ = nbrs.kneighbors(X.iloc[rand_X[j]].values.reshape(1, -1), 2, return_distance=True)
        wjd.append(w_dist[0][1])
 
    H = sum(ujd) / (sum(ujd) + sum(wjd))
    if isnan(H):
        print(ujd, wjd)
        H = 0
 
    return H
for i in range(5):
  print('Hopkins statistic value is:',round(hopkins(country_df_scaled),3))<jupyter_output>Hopkins statistic value is: 0.885
Hopkins statistic value is: 0.866
Hopkins statistic value is: 0.869
Hopkins statistic value is: 0.863
Hopkins statistic value is: 0.856<jupyter_text>As outcome of Hopkins test is greater than 0.7, thus the dataset has high clustering tendency.  **5. Clustering using K-Means and Hierarchical Clustering**  **i) K-means Clustering**  **a) Finding Optimal value of K**  **Elbow Method**<jupyter_code># elbow-curve/SSD
ssd = []
range_n_clusters = [2, 3, 4, 5, 6, 7, 8]
for num_clusters in range_n_clusters:
    kmeans = KMeans(n_clusters=num_clusters, max_iter=50)
    kmeans.fit(country_df_scaled)
    ssd.append(kmeans.inertia_)

plt.plot(range_n_clusters,ssd)
plt.xlabel('Number of clusters(k)')
plt.ylabel('SSD')
plt.show()<jupyter_output><empty_output><jupyter_text>Based on above "elbow" curve, **k=3** seems to be the optimal number of clusters.  **Silhouette Analysis**<jupyter_code># silhouette analysis
range_n_clusters = [2, 3, 4, 5, 6, 7, 8]

for num_clusters in range_n_clusters:
    
    # intialise kmeans
    kmeans = KMeans(n_clusters=num_clusters, max_iter=50)
    kmeans.fit(country_df_scaled)
    
    cluster_labels = kmeans.labels_
    
    # silhouette score
    silhouette_avg = silhouette_score(country_df_scaled, cluster_labels)
    print("For n_clusters={0}, the silhouette score is {1}".format(num_clusters, round(silhouette_avg,2)))<jupyter_output>For n_clusters=2, the silhouette score is 0.48
For n_clusters=3, the silhouette score is 0.42
For n_clusters=4, the silhouette score is 0.38
For n_clusters=5, the silhouette score is 0.38
For n_clusters=6, the silhouette score is 0.29
For n_clusters=7, the silhouette score is 0.31
For n_clusters=8, the silhouette score is 0.3<jupyter_text>Using silhouette analysis k=2 seems to be the optimal number of clusters. However, as per business requirement having 3 clusters would make more sense. Since we can segment the countries into high, moderate and low based on their development.  **b) Run K-means using optimal K**<jupyter_code># final model with k=3
kmeans = KMeans(n_clusters=3, max_iter=50,random_state=1)
kmeans.fit(country_df_scaled)
# Adding cluster labels to master dataframe
country_df['cluster_id'] = kmeans.labels_
# Number of countries per cluster
country_df['cluster_id'].value_counts()<jupyter_output><empty_output><jupyter_text>**c) Visualizing Clusters** Visualizing Clusters characterstics using boxplots.<jupyter_code># Visualizing Numerical columns using Boxplots
cols = country_df.columns[1:-1].tolist()
r = c = 0
fig,ax = plt.subplots(5,2,figsize=(15,25))

for n,i in enumerate(cols):
    sns.boxplot(x='cluster_id',y=cols[n], data=country_df,ax=ax[r,c])
    c+=1
    if (n+1)%2==0:
        r+=1
        c=0
ax[r,c].axis("off")
plt.show()<jupyter_output><empty_output><jupyter_text>**Inferences:** Above plots suggests that **cluster_2** has lowest "life_expec", "income", "gdpp" and highest "child_mort" among all three clusters. Thus its clearly visible that countries which falls under **cluster_2** are in need of aid the most. Meanwhile, **cluster_0** and **cluster_1** are moderately and highly self sufficient respectively.  **d) Clustering profiling using “gdpp" , "child_mort" and "income”**<jupyter_code>fig = plt.figure(figsize=[20,8])

plt.subplot(1,3,1)
sns.scatterplot(data=country_df,x="gdpp",y="child_mort",hue="cluster_id",size="cluster_id",palette="Set1")
plt.subplot(1,3,2)
sns.scatterplot(data=country_df,x="income",y="child_mort",hue="cluster_id",size="cluster_id",palette="Set1")
plt.subplot(1,3,3)
sns.scatterplot(data=country_df,x="income",y="gdpp",hue="cluster_id",size="cluster_id",palette="Set1")
plt.show()<jupyter_output><empty_output><jupyter_text>**Based on k-means clustering following are the countries which are in dire need of aid.**<jupyter_code># Top 5 under developed countries ordered by "child_mort","gdpp","income"
country_df[country_df['cluster_id'] == 2].sort_values(["child_mort","gdpp","income"],ascending=[False,True,False]).head(5)<jupyter_output><empty_output><jupyter_text>**ii) Hierarchical Clustering**  **a) Plot dendrogram using single and complete linkage**   **Dendrogram using single linkage**<jupyter_code>single_den = linkage(country_df_scaled, method="single", metric='euclidean')
dendrogram(single_den)
plt.show()<jupyter_output><empty_output><jupyter_text>**Dendrogram using complete linkage**<jupyter_code>comp_dend = linkage(country_df_scaled, method="complete", metric='euclidean')
dendrogram(comp_dend)
plt.show()<jupyter_output><empty_output><jupyter_text>**b) Choosing Optimal Linkage**<jupyter_code># Single Linkage
cluster_labels = cut_tree(single_den, n_clusters=4).reshape(-1, )
country_df['cluster_id'] = cluster_labels
country_df['cluster_id'].value_counts()
# Complete Linkage
cluster_labels = cut_tree(comp_dend, n_clusters=4).reshape(-1, )
country_df['cluster_id'] = cluster_labels
country_df['cluster_id'].value_counts()<jupyter_output><empty_output><jupyter_text>**Inferences:**- Based on above cluster distribution **"complete linkage"** seems to do better clustering compared to "single linkage", as the countries are more uniformly distributed per cluster. - Unlike k-means, for hierarchical clustering **k=4** seems to provide better results.  **c) Visualizing clusters**<jupyter_code># Visualizing Numerical columns using Boxplots
cols = country_df.columns[1:-1].tolist()
r = c = 0
fig,ax = plt.subplots(5,2,figsize=(15,25))

for n,i in enumerate(cols):
    sns.boxplot(x='cluster_id',y=cols[n], data=country_df,ax=ax[r,c])
    c+=1
    if (n+1)%2==0:
        r+=1
        c=0
ax[r,c].axis("off")
plt.show()<jupyter_output><empty_output><jupyter_text>**Inferences:** Above plots suggests that **cluster_0** has lowest "life_expec", "income", "gdpp" and highest "child_mort" among all four clusters. Thus its clearly visible that countries which falls under **cluster_0** are in need of aid the most.  **d) Clustering profiling using “gdpp" , "child_mort" and "income”**<jupyter_code>fig = plt.figure(figsize=[20,8])

plt.subplot(1,3,1)
sns.scatterplot(data=country_df,x="gdpp",y="child_mort",hue="cluster_id",size="cluster_id",palette="Set1")
plt.subplot(1,3,2)
sns.scatterplot(data=country_df,x="income",y="child_mort",hue="cluster_id",size="cluster_id",palette="Set1")
plt.subplot(1,3,3)
sns.scatterplot(data=country_df,x="income",y="gdpp",hue="cluster_id",size="cluster_id",palette="Set1")
plt.show()<jupyter_output><empty_output><jupyter_text>**Based on Hierarchical clustering following are the countries which are in dire need of aid.**<jupyter_code>country_df[country_df['cluster_id'] == 0].sort_values(["child_mort","gdpp","income"],ascending=[False,True,False]).head(5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12678741658722592, 'fraction_numerical': 0.024022878932316492, 'mean_word_length': 5.7507507507507505, 'pattern_counts': {'":': 0, '<': 91, '<?xml version=': 0, '>': 94, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '346527', 'license': 'MIT', 'n_tokens_mistral': 5685, 'n_tokens_neox': 5261, 'n_words': 1388, 'path': 'Clustering_Assignment.ipynb', 'repo_name': 'rppradhan08/Socio-Economic-Countries-Clustering', 'size': '15735'}