<jupyter_start><jupyter_text>Homework 9 - Dimensionality ReductionIn machine learning, “dimensionality” simply refers to the number of features (i.e. input variables) in your dataset.When the number of features is very large relative to the number of observations in your dataset, certain algorithms struggle to train effective models. This is called the “Curse of Dimensionality,” and it’s especially relevant for clustering algorithms that rely on distance calculations. [1] Why is Dimensionality Reduction important in Machine Learning and Predictive Modeling?An intuitive example of dimensionality reduction can be discussed through a simple e-mail classification problem, where we need to classify whether the e-mail is spam or not. This can involve a large number of features, such as whether or not the e-mail has a generic title, the content of the e-mail, whether the e-mail uses a template, etc. However, some of these features may overlap. In another condition, a classification problem that relies on both humidity and rainfall can be collapsed into just one underlying feature, since both of the aforementioned are correlated to a high degree. Hence, we can reduce the number of features in such problems. A 3-D classification problem can be hard to visualize, whereas a 2-D one can be mapped to a simple 2 dimensional space, and a 1-D problem to a simple line. The below figure illustrates this concept, where a 3-D feature space is split into two 1-D feature spaces, and later, if found to be correlated, the number of features can be reduced even further.[2][2] Components of Dimensionality ReductionThere are two components of dimensionality reduction:1. Feature selection: In this, we try to find a subset of the original set of variables, or features, to get a smaller subset which can be used to model the problem. It usually involves three ways:    a. Filter    b. Wrapper    c. Embedded2. Feature extraction: This reduces the data in a high dimensional space to a lower dimension space, i.e. a space with lesser no. of dimensions. Methods of Dimensionality ReductionThe various methods used for dimensionality reduction include:1. Principal Component Analysis (PCA)2. Linear Discriminant Analysis (LDA)3. Generalized Discriminant Analysis (GDA)Dimensionality reduction may be both linear or non-linear, depending upon the method used. The prime linear method, called Principal Component Analysis, or PCA, is discussed below. Principal Component AnalysisThis method was introduced by <NAME>. It works on a condition that while the data in a higher dimensional space is mapped to data in a lower dimension space, the variance of the data in the lower dimensional space should be maximum.[2] REFERENCES:[1] Elite Data Science, "Dimensionality Reduction Algorithms". URL - https://elitedatascience.com/dimensionality-reduction-algorithms[2] Geek for Geeks, "Dimensionality Reduction". URL - https://www.geeksforgeeks.org/dimensionality-reduction/  Data SetupLoading the file.File Obtained from http://www.who.int/tb/country/data/download/en/<jupyter_code>local_tb_existing_file = 'burden_estimates.csv'<jupyter_output><empty_output><jupyter_text>Reading csv file with 'country' column as index.<jupyter_code>import pandas as pd
    
existing_df = pd.read_csv(
    local_tb_existing_file, 
    index_col = 0)
existing_df.index.names = ['country']
existing_df.columns.names = ['variables']
existing_df.head()<jupyter_output><empty_output><jupyter_text>For the purpose of PCA, we will only be keeping columns with float type values - hence, removing columns that do not have float type values.<jupyter_code>import numpy as np
# check type of each column
df = existing_df
df.dtypes
# columns that are floats
msk = df.dtypes == np.float64
# keep those that are floats
active_df = df.loc[:, msk]
active_df.head(5)<jupyter_output><empty_output><jupyter_text>We also need to avoid NaN values from the dataframe. So removing columns with NaN values.<jupyter_code>active_df = active_df.dropna()
active_df.head()<jupyter_output><empty_output><jupyter_text>Implementing PCA on the DatasetFollowing the tutorial at: https://www.codementor.io/jadianes/data-science-python-pandas-r-dimensionality-reduction-du1081aka  pip install sklean<jupyter_code>from sklearn.decomposition import PCA
    
pca = PCA(n_components=2)
pca.fit(active_df)
data_2d = pca.transform(active_df)<jupyter_output><empty_output><jupyter_text>Creating a new dataframe with country as index, and the two principal components as the two columns.<jupyter_code>data_2d = pd.DataFrame(data_2d)
data_2d.index = active_df.index
data_2d.columns = ['PC1','PC2']
data_2d.head()<jupyter_output><empty_output><jupyter_text>Variances of PC1, PC2<jupyter_code>print(pca.explained_variance_ratio_)<jupyter_output>[0.995717   0.00427129]<jupyter_text>We see that the first PC already explains almost 99.6% of the variance, while the second one accounts for another 0.4% between the two of them. So the most varience will occur on the y axis Let's visualize it the graph below<jupyter_code>#rounding up values to 2 decimal points for graph
for index, row in data_2d.iterrows():
    row['PC1'] = round(row['PC1'],2)
    row['PC2'] = round(row['PC2'],2)
ax = data_2d.plot(kind='scatter', x='PC2', y='PC1', figsize=(16,8))
    
for i, country in enumerate(active_df.index):
    ax.annotate(
        country.encode('utf-8'), (data_2d.iloc[i].PC2, data_2d.iloc[i].PC1)
    )<jupyter_output><empty_output><jupyter_text>Normalize Data Mean<jupyter_code>from sklearn.preprocessing import normalize
    
data_2d['country_mean'] = pd.Series(active_df.mean(axis=1), index=data_2d.index)
country_mean_max = data_2d['country_mean'].max()
country_mean_min = data_2d['country_mean'].min()
country_mean_scaled = (data_2d.country_mean-country_mean_min) / country_mean_max
data_2d['country_mean_scaled'] = pd.Series(country_mean_scaled, index=data_2d.index) 
data_2d.head()
data_2d.plot(
    kind='scatter', 
    x='PC2', 
    y='PC1', 
    s=data_2d['country_mean_scaled']*100, 
    figsize=(16,8))<jupyter_output><empty_output><jupyter_text>Sum<jupyter_code>data_2d['country_sum'] = pd.Series(active_df.sum(axis=1), index=data_2d.index)
country_sum_max = data_2d['country_sum'].max()
country_sum_min = data_2d['country_sum'].min()
country_sum_scaled = (data_2d.country_sum-country_sum_min) / country_sum_max
data_2d['country_sum_scaled'] = pd.Series(country_sum_scaled, index=data_2d.index)
data_2d.head()
data_2d.plot(kind='scatter', x='PC2', y='PC1', s=data_2d['country_sum_scaled']*100, figsize=(16,8))<jupyter_output><empty_output><jupyter_text>What can we learn from the above graphs?From the above plots of the pca results and on the normalized data, we can confirm that the most variation happens along the y axis, which we have assigned to PC1. We saw that the first PC already explains almost 99.6% of the variance, while the second one accounts for another 0.4% between the two of them. The countries are spread in the graph but there is a concentration at the bottom center. Performing a clustering algorithm on our findings can further categorize the countries and group them in clusters. So performing K-Means Clustering.  K-Means ClusteringUsing sklearn's K-Means Clustering Implementation<jupyter_code>from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
from itertools import cycle, islice
from matplotlib import cm


#set the number of clusters: 3
kmeans = KMeans(n_clusters=3)
clusters = kmeans.fit(active_df)
data_2d['cluster'] = pd.Series(clusters.labels_, index=data_2d.index)
color = cm.inferno_r(data_2d.cluster.astype(np.float)/4)

data_2d.plot(kind='scatter',x='PC2',y='PC1',c=color, figsize=(12,6))<jupyter_output><empty_output><jupyter_text>The above graph shows 3 clusters.Cluster Top - OrangeCluster Center - MaroonCluster Bottom - YellowLet us see which countries belong to which clusters.<jupyter_code>labels = kmeans.predict(active_df)
clusters = {}
n = 0

# associate each state to a cluster
for item in labels:
    if item in clusters:
        clusters[item].append(active_df.index[n])
    else:
        clusters[item] = [active_df.index[n]]
    n +=1

# print the cluster
for item in clusters:
    print "Cluster ", item
    for i in clusters[item]:
        print i
    print<jupyter_output>Cluster  0
Afghanistan
Albania
Algeria
American Samoa
Andorra
Angola
Antigua and Barbuda
Argentina
Armenia
Aruba
Australia
Austria
Azerbaijan
Bahamas
Bahrain
Bangladesh
Belarus
Belgium
Belize
Benin
Bermuda
Bhutan
Bolivia (Plurinational State of)
Botswana
Brazil
Bulgaria
Burkina Faso
Burundi
Cabo Verde
Cambodia
Cameroon
Cayman Islands
Central African Republic
Chad
Chile
China, Hong Kong SAR
China, Macao SAR
Colombia
Congo
Costa Rica
Côte d'Ivoire
Cuba
Curaçao
Cyprus
Czechia
Democratic People's Republic of Korea
Democratic Republic of the Congo
Denmark
Djibouti
Dominica
Dominican Republic
Ecuador
Egypt
El Salvador
Equatorial Guinea
Eritrea
Estonia
Eswatini
Ethiopia
Fiji
Finland
France
French Polynesia
Gabon
Gambia
Georgia
Germany
Ghana
Greece
Greenland
Grenada
Guam
Guatemala
Guinea
Guinea-Bissau
Guyana
Haiti
Honduras
Hungary
Iran (Islamic Republic of)
Iraq
Ireland
Israel
Italy
Jamaica
Japan
Jordan
Kazakhstan
Kenya
Kiribati
Kuwait
Kyrgyzstan
Lao People's Democratic Republic
Latvia
Lebanon[...]
{'fraction_non_alphanumeric': 0.07751937984496124, 'fraction_numerical': 0.01475021533161068, 'mean_word_length': 5.6302640970735185, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 3, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '325662', 'license': 'MIT', 'n_tokens_mistral': 3039, 'n_tokens_neox': 2882, 'n_words': 1017, 'path': 'Homeworks/Homework9/Maharjan_HW9_DimensionalityReduction_fall2018.ipynb', 'repo_name': 'jenishmaharjan/big-data-python-class', 'size': '9294'}