<jupyter_start><jupyter_text>Why we want to reduce the dimentionality:In the era of big data, when more is axiomatically better, we have re-discovered that **too many noisy or even faulty input data columns** often lead to a less than desirable algorithm performance. Removing un-informative or even worse dis-informative input attributes might help build a model on more extensive data regions, with **more general classification rules**, **the training speed much faster** and overall with **better performances on new unseen data**.<jupyter_code>import sklearn
from sklearn import svm,metrics,model_selection
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Data ImportNote: It is very significant to remember for SVM that the input data need to be normalized so that features are on the same scale and compatible.    But due to the fact that we use the features of a DL model as the input, we don't need to do the normalization<jupyter_code>X_features=pd.read_csv(r"Dataset\features\ResNet101\AwA2-features.txt",header=None,sep=' ')
X_features.head(5)
X_features.shape
Y_labels=pd.read_csv(r"Dataset\features\ResNet101\AwA2-labels.txt",header=None, sep=' ')
Y_labels.head(5)
Y_labels.shape
print(sorted(Y_labels.iloc[:,0].unique()))
X_train,X_test,Y_train,Y_test= model_selection.train_test_split(X_features,Y_labels, test_size=0.4,random_state=1234,stratify=Y_labels)
print(X_train.shape, Y_train.shape,X_test.shape,Y_test.shape)
print("Train set:",sorted(Y_train.iloc[:,0].unique()))
print("Test set:",sorted(Y_test.iloc[:,0].unique()))<jupyter_output>Train set: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50]
Test set: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50]<jupyter_text>Linear SVM  Train   --> here we just use the default SVM Hyper-parameterssvm.LinearSVC(    penalty='l2',    loss='squared_hinge',    *,    dual=True,    tol=0.0001,    C=1.0,    multi_class='ovr',    fit_intercept=True,    intercept_scaling=1,    class_weight=None,    verbose=0,    random_state=None,    max_iter=1000,)<jupyter_code># use linear separable SVM
linear_svc = svm.LinearSVC()
linear_svc.fit(X_train,Y_train)<jupyter_output>D:\anaconda\envs\AI_1\lib\site-packages\sklearn\utils\validation.py:72: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  return f(**kwargs)
D:\anaconda\envs\AI_1\lib\site-packages\sklearn\svm\_base.py:977: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
  "the number of iterations.", ConvergenceWarning)<jupyter_text>Test   -->  Sklearn.metrics.accuracy_score()<jupyter_code>pred_linear_svc = linear_svc.predict(X_test)
metrics.accuracy_score(Y_test, pred_linear_svc)<jupyter_output><empty_output><jupyter_text>Non-Linear SVM  Trainsvm.SVC(    *,    C=1.0,    kernel='rbf',    degree=3,    gamma='scale',    coef0=0.0,    shrinking=True,    probability=False,    tol=0.001,    cache_size=200,    class_weight=None,    verbose=False,    max_iter=-1,    decision_function_shape='ovr',    break_ties=False,    random_state=None,)gamma : {'scale', 'auto'} or float, default='scale'    - if ``gamma='scale'`` (default) is passed then it uses      1 / (n_features * X.var()) as value of gamma,    - if 'auto', uses 1 / n_features.<jupyter_code># use non-linear SVM, set the kernel function as RBF
nolinear_svc = svm.SVC(kernel='rbf')   # radial basis function
nolinear_svc.fit(X_train,Y_train)<jupyter_output>D:\anaconda\envs\AI_1\lib\site-packages\sklearn\utils\validation.py:72: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  return f(**kwargs)<jupyter_text>Test<jupyter_code>pred_svc = nolinear_svc.predict(X_test)
metrics.accuracy_score(Y_test,pred_svc)<jupyter_output><empty_output><jupyter_text>Hyper-Parameters tuning Significant HPs for SVM:**C** :    C parameter adds a penalty for each misclassified data point.Normal setting: 0.1 < c < 100**gamma(RBF)**:  Controls the distance of influence of a single training point. --> Kernel trickKernelized SVM compute decision boundaries in terms of similarity measures in a high-dimensional feature space without actually doing a transformation.(For the fact that it's a costly operation to actually transform data points to a high-dimensional feature space)Normal setting: 0.0001 < gamma < 10 How Hyper-Parameters affect the decision boundary and classification results**C**:If c is small, the penalty for misclassified points is low so a decision boundary with a large margin is chosen at the expense of a greater number of misclassifications. If c is large, SVM tries to minimize the number of misclassified examples due to high penalty which results in a decision boundary with a smaller margin.**gamma**:Low values of gamma indicates a large similarity radius which results in more points being grouped together. For high values of gamma, the points need to be very close to each other in order to be considered in the same group (or class). Therefore, models with very large gamma values tend to overfit.  Note: These parameters need to optimized simultaneously.E.g: If gamma is large, the effect of c becomes negligible. If gamma is small, c affects the model just like how it affects a linear model.<jupyter_code>C= np.logspace(-1,1,10)
gamma = ['scale','auto']
parameters = {'C':C,'gamma':gamma}
print(C,'\n',gamma)
grid_svm = model_selection.GridSearchCV(estimator = svm.SVC(max_iter=10000),param_grid =parameters,scoring='balanced_accuracy'
                                        ,cv=4,verbose =3, n_jobs=-1)  
# max_iter: I don't want the estimator to run it out completely, just determine the best candidate will be fine  
# scoring: This is very important, it decides how CrossValidation selects the best candidate. For the fact that the classes in the training set is not balanced, so we choose balanced_accuracy
# CV:  CrossValidation in training set
# n_jobs: use all the possible processor to speed-up the CV process
grid_svm.fit(X_train,Y_train)
print(grid_svm.best_params_, grid_svm.best_score_)    # From the terminal outputs we can see that the best hyperparameters is : C=1.29,gamma=scale<jupyter_output>Fitting 4 folds for each of 20 candidates, totalling 80 fits<jupyter_text>Get the best model<jupyter_code># Default model
Default_nolinear_svc = svm.SVC(kernel='rbf')   # radial basis function
Default_nolinear_svc.fit(X_train,Y_train)
pred_svc = Default_nolinear_svc.predict(X_test)
metrics.accuracy_score(Y_test,pred_svc)
# Best model we get from the CrossValidation
Best_nolinear_svc = svm.SVC(kernel='rbf',C=1.29,gamma='scale')   # radial basis function
Best_nolinear_svc.fit(X_train,Y_train)
pred_svc = Best_nolinear_svc.predict(X_test)
metrics.accuracy_score(Y_test,pred_svc)<jupyter_output>D:\anaconda\envs\AI_1\lib\site-packages\sklearn\utils\validation.py:72: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  return f(**kwargs)<jupyter_text>Reduce the dimentionality  Methods:1. Feature Selection  -->  Random Forest2. Dementionality Reduction        2.1 Factors Based  --> PCA        2.2 Projection Based  --> UMAP, t-SNE  1st: Random Forest**This is one of the most commonly used techniques which tells us the importance of each feature present in the dataset. We can find the importance of each feature and keep the top most features, resulting in dimensionality reduction**<jupyter_code>from sklearn.ensemble import RandomForestRegressor
from sklearn.feature_selection import SelectFromModel
random_forest_model = RandomForestRegressor(random_state=1234, max_depth=5,n_estimators=500)
# random_forest_model.fit(X_features,Y_labels)
# get a small sample of the dataset to fit into the model, due to the fact that training Random_Forest needs a lot of time
S_train_x,S_test_x,S_train_y,S_test_y=model_selection.train_test_split(X_train,Y_train, test_size=0.1,random_state=1234,stratify=Y_train) 
print(S_test_x.shape,S_test_y.shape)
random_forest_model.fit(S_test_x,S_test_y)  
original_features = X_features.columns
importances = random_forest_model.feature_importances_
indices = np.argsort(importances)[-20:]  #  we only show the top 10 features
plt.figure(figsize=[20,10])
plt.title('Feature Importances',fontdict={'size':18})
plt.barh(range(len(indices)), importances[indices], color='b', align='center')
plt.yticks(range(len(indices)), [original_features[i] for i in indices],fontsize=18)
plt.xticks(fontsize=18)
plt.xlabel('Relative Importance',fontdict={'size':18})
plt.show()
arg = np.argsort(importances)
accumulate_sum=[]
for i in range(50,1000,50):
    accumulate_sum.append(np.sum(importances[arg[-i:]]))
    print("Dimension:",i,"  Accumulate sum:",accumulate_sum[-1])
    
plt.figure(figsize=[18,10])
plt.plot(range(50,1000,50),accumulate_sum )
plt.xlabel("Number of sorted columns",fontdict={'size':18})
plt.ylabel("Cumulative importance sum",fontdict={'size':18})
plt.xticks(fontsize=18)
plt.yticks(fontsize=18)

# plt.title("Column-wise and Cumulative importance sum")
acc_list=[]
for dimension in range(500,800,20):
    indices=np.argsort(importances)[-dimension:]
    np.sum(importances[indices])
    RF_reduced_features=X_features.iloc[:,indices]
    RF_reduced_x_train,RF_reduced_x_test,RF_reduced_y_train,RF_reduced_y_test=model_selection.train_test_split(RF_reduced_features,Y_labels, test_size=0.4,random_state=1234,stratify=Y_labels)
    RF_linear_svc = svm.LinearSVC()
    RF_linear_svc.fit(RF_reduced_x_train,RF_reduced_y_train)
    RF_pred_linear_svc = RF_linear_svc.predict(RF_reduced_x_test)
    acc=metrics.accuracy_score(RF_reduced_y_test, RF_pred_linear_svc)
    acc_list.append(acc)
print(acc_list)
plt.plot(np.arange(500, 800, 20), acc_list)
plt.xlabel("Dimension")
plt.ylabel("Accuracy")
# plt.title("Random Forest")
plt.show()
plt.figure(figsize=[20,10])
plt.plot(np.arange(500, 800, 20), acc_list)
plt.xlabel("Dimension",fontdict={'size':18})
plt.ylabel("Accuracy",fontdict={'size':18})
plt.xticks(fontsize=18)
plt.yticks(fontsize=18)
# plt.title("Random Forest")
plt.show()
count=0
for dimension in range(500,800,20):
    print("Dimension: ",dimension,", Accuracy: ",acc_list[count])
    count+=1
RF_acc_list=acc_list<jupyter_output>Dimension:  500 , Accuracy:  0.888874003617121
Dimension:  520 , Accuracy:  0.8894098734007636
Dimension:  540 , Accuracy:  0.8927590595485297
Dimension:  560 , Accuracy:  0.8923571572107978
Dimension:  580 , Accuracy:  0.8924911246567084
Dimension:  600 , Accuracy:  0.8942327014535468
Dimension:  620 , Accuracy:  0.8938977828387702
Dimension:  640 , Accuracy:  0.8947685712371893
Dimension:  660 , Accuracy:  0.897179985263581
Dimension:  680 , Accuracy:  0.8991225132292853
Dimension:  700 , Accuracy:  0.8996583830129279
Dimension:  720 , Accuracy:  0.9005961551343024
Dimension:  740 , Accuracy:  0.9009980574720343
Dimension:  760 , Accuracy:  0.9014669435327215
Dimension:  780 , Accuracy:  0.9014669435327215<jupyter_text>Note:We use the SelectFromModel() of sklearn to select the features based on the importance of their weights.The above process is to show the graphic resultsRF_method = SelectFromModel(random_forest_model)RF_reduced_features = RF_method.fit_transform(X_features, Y_labels)  2nd: PCA, LDA**Principal Component Analysis: This is one of the most widely used techniques for dealing with linear data. It divides the data into a set of components which try to explain as much variance as possible**Note: One of the most advantages for PCA is that it can run in a short time. So when we get into a high dimension inputs and largenumber of samples, we prefer to use PCA* Drawbacks: In the context of predictive modeling, dimensionality reduction is quite dangerousE.g.:* In finance, stock returns have about 15-25% annual standard deviation. Changes in bond yields are historically much lower standard deviation. If you perform PCA on the covariance matrix of stock returns and changes in bond yields, then the top PCs will all reflect variance of the stocks and the smallest ones will reflect the variances of the bonds. If you throw away the PCs that explain the bonds, then you might be in for some trouble. For instance, the bonds might have very different distributional characteristics than stocks (thinner tails, different time-varying variance properties, different mean reversion, cointegration, etc). These might be very important to model, depending on the circumstances.    If you perform PCA on the correlation matrix, then you might see more of the PCs explaining bonds near the top.* Sometimes we actually prefer the low variability features for anomaly detection, since a significant shift in a low variability dimension is a strong indicator of anomalous behavior. The motivating example they provide is as follows:    Assume a user always logs in from a Mac. The "operating system" dimension of their activity would be very low variance. But if we saw a login event from that same user where the "operating system" was Windows, that would be very interesting, and something we'd like to catch.<jupyter_code>from sklearn.decomposition import PCA
pca_method = PCA(n_components=300)
PCA_reduced_features = pca_method.fit_transform(X_features)
plt.plot(range(300), pca_method.explained_variance_ratio_)
plt.plot(range(300), np.cumsum(pca_method.explained_variance_ratio_))
plt.title("Component-wise and Cumulative Explained Variance")<jupyter_output><empty_output><jupyter_text>Note: From the graph above we can see that PCA components=**300** explains around **85%** variance in the dataset. We've try different n_components values (10-1000), and find that 300 is the **most appropriate value**  LDA* PCA: component axis that maximize the variance* LDA: component axis that maximize the class separation<jupyter_code>from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
lda_method = LinearDiscriminantAnalysis(n_components=49)
LDA_reduced_features = lda_method.fit_transform(X_features,Y_labels)
LDA_reduced_features.shape
def LDA_fit(dimension=49):
    lda_method = LinearDiscriminantAnalysis(n_components=dimension)
    lda_method.fit(X_train, Y_train)
    return lda_method


def LinearSVM_LDA(LDA_reduced_x_train, LDA_reduced_y_train, LDA_reduced_x_test, LDA_reduced_y_test):
    LDA_linear_svc = svm.LinearSVC()
    LDA_linear_svc.fit(LDA_reduced_x_train, LDA_reduced_y_train)
    LDA_pred_linear_svc = LDA_linear_svc.predict(LDA_reduced_x_test)
    return metrics.accuracy_score(LDA_reduced_y_test, LDA_pred_linear_svc)


def LDA_plotting(acc_list):
    # LDA
    for dimension in range(4, 50, 5):
        LDA_method = LDA_fit(dimension)
        LDA_reduced_features = LDA_method.transform(X_features)
        LDA_reduced_x_train, LDA_reduced_x_test, LDA_reduced_y_train, LDA_reduced_y_test = model_selection.train_test_split(
            LDA_reduced_features, Y_labels, test_size=0.4, random_state=1234, stratify=Y_labels)
        acc = LinearSVM_LDA(LDA_reduced_x_train, LDA_reduced_y_train, LDA_reduced_x_test, LDA_reduced_y_test)
        acc_list.append(acc)
        print("Dimension:", dimension,", Accuracy:",acc)
    print(acc_list)
    plt.figure(figsize=[20, 10])
    plt.plot(np.arange(4, 50, 5), acc_list)
    plt.xlabel("Dimension",fontdict={'size':18})
    plt.ylabel("Accuracy",fontdict={'size':18})
    plt.xticks(fontsize=18)
    plt.yticks(fontsize=18)
#     plt.title("Linear Discriminant Analysis")
    plt.show()
    return acc_list
LDA_acc_list=[]
LDA_acc_list=LDA_plotting(LDA_acc_list)
print(LDA_acc_list)
count=0
for dimension in range(4, 50, 5):
    print("Dimension: ",dimension,", Accuracy: ",LDA_acc_list[count])
    count+=1<jupyter_output>Dimension:  4 , Accuracy:  0.3694822158215554
Dimension:  9 , Accuracy:  0.5797441221783106
Dimension:  14 , Accuracy:  0.7231562730256548
Dimension:  19 , Accuracy:  0.7697099604796035
Dimension:  24 , Accuracy:  0.8336794159019358
Dimension:  29 , Accuracy:  0.871793154263514
Dimension:  34 , Accuracy:  0.8859267198070869
Dimension:  39 , Accuracy:  0.9039453412820685
Dimension:  44 , Accuracy:  0.9170071672583562
Dimension:  49 , Accuracy:  0.9241074418916203<jupyter_text>3rd: UMAP **t-SNE:This technique works well when the data is strongly non-linear. It works extremely well for visualizations as well*** ShortComes: loss of large-scale information, slow computation time, and inability to meaningfully represent very large datasets.**UMAP:This technique works well for high dimensional data. Its run-time is shorter as compared to t-SNE. So due to the same reason, we prefer UMAP. Here are the details:**Key advantages of UMAP :* It can handle large datasets and high dimensional data without too much difficulty* It combines the power of visualization with the ability to reduce the dimensions of the data* Along with preserving the local structure, it also preserves the global structure of the data. UMAP maps nearby points on the manifold to nearby points in the low dimensional representation, and does the same for far away points**How does UMAP work:**This method uses the concept of k-nearest neighbor and optimizes the results using stochastic gradient descent. It first calculates the distance between the points in high dimensional space, projects them onto the low dimensional space, and calculates the distance between points in this low dimensional space. It then uses Stochastic Gradient Descent to minimize the difference between these distances.<jupyter_code>import umap   # conda install -c conda-forge umap-learn  
import umap.plot
"""
# Check code for using umap.plot
import pandas as pd
import datashader as ds
import datashader.transfer_functions as tf
import datashader.bundling as bd
import matplotlib.pyplot as plt
import colorcet
import matplotlib.colors
import matplotlib.cm
import bokeh.plotting as bpl
import bokeh.transform as btr
import holoviews as hv
import holoviews.operation.datashader as hd
"""
# Originally, we need to set the random_state to ensure reproducibility. But according to UMAP's official document, 
# it will shut down any of the multi-threading that does not support explicit reproducibility
UMAP_reduced_features = umap.UMAP(n_neighbors=30, min_dist=0.82, n_components=50).fit_transform(X_features)<jupyter_output><empty_output><jupyter_text>This part is just for training<jupyter_code>UMAP_reduced_features = umap.UMAP(n_neighbors=30, min_dist=0.82, n_components=50,random_state=3).fit_transform(X_features) 
UMAP_reduced_x_train,UMAP_reduced_x_test,UMAP_reduced_y_train,UMAP_reduced_y_test=model_selection.train_test_split(UMAP_reduced_features,Y_labels, test_size=0.4,random_state=1234,stratify=Y_labels)
UMAP_linear_svc = svm.LinearSVC()
UMAP_linear_svc.fit(UMAP_reduced_x_train,UMAP_reduced_y_train)
UMAP_pred_linear_svc = UMAP_linear_svc.predict(UMAP_reduced_x_test)
metrics.accuracy_score(UMAP_reduced_y_test, UMAP_pred_linear_svc)<jupyter_output>D:\anaconda\envs\AI_1\lib\site-packages\sklearn\utils\validation.py:63: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  return f(*args, **kwargs)
D:\anaconda\envs\AI_1\lib\site-packages\sklearn\svm\_base.py:986: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
  "the number of iterations.", ConvergenceWarning)<jupyter_text>Plotting the features<jupyter_code>plot_features=umap.UMAP(n_neighbors=3, min_dist=0.3,n_components=2).fit(X_features)
umap.plot.points(plot_features)<jupyter_output>*c* argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with *x* & *y*.  Please use the *color* keyword-argument or provide a 2-D array with a single row if you intend to specify the same RGB or RGBA value for all points.<jupyter_text>TSNENote: TSNE is an unsupervised method for dimesionality reduction/visualization, so it does not really work with a TRAIN and TEST. So we  simply take all of the data and use fit_transform to have the transformation and plot it.<jupyter_code>from sklearn.manifold import TSNE
pca_method=PCA(n_components=50)
pca_method.fit(X_train)
pca_reduced_features=pca_method.transform(X_features)
# reduced_features=PCA(n_components=50).fit_transform(X_features)
dim_list=[2,3]
perplexity_list=[20,25,30,40,50]
tsne_acc_list=[]
for dim in dim_list:
    dim_acc_list=[]
    for perplexity in perplexity_list:
        tsne_method=TSNE(n_components=dim,perplexity=perplexity,n_iter=300,init='pca',method='barnes_hut')
        TSNE_reduced_features=tsne_method.fit_transform(pca_reduced_features)
        TSNE_reduced_x_train,TSNE_reduced_x_test,TSNE_reduced_y_train,TSNE_reduced_y_test=model_selection.train_test_split(TSNE_reduced_features,Y_labels, test_size=0.4,random_state=1234,stratify=Y_labels)
        TSNE_linear_svc = svm.LinearSVC()
        TSNE_linear_svc.fit(TSNE_reduced_x_train,reduced_y_train)
        TSNE_pred_linear_svc = TSNE_linear_svc.predict(TSNE_reduced_x_test)
        acc=metrics.accuracy_score(reduced_y_test, TSNE_pred_linear_svc)
        dim_acc_list.append(acc)
        print("Reduced Dimension:", dim, "Perplexity:",perplexity,"Acc:",acc)
    tsne_acc_list.append(dim_acc_list)
print("Perplexity in each dim:",perplexity_list)
print("Dim=2, Acc list:",tsne_acc_list[0])
print("Dim=3, Acc list:",tsne_acc_list[1])

iter_list=[300,400,500,600,800]
lr_list=[50,100,150,200,250,300]
tsne_acc_list_2=[]
lr_acc_list=[]
for it in iter_list:
    lr_acc_list=[]
    for lr in lr_list:
        tsne_method=TSNE(n_components=3,perplexity=30,n_iter=it,learning_rate=lr,init='pca',method='barnes_hut')
        TSNE_reduced_features=tsne_method.fit_transform(pca_reduced_features)
        TSNE_reduced_x_train,TSNE_reduced_x_test,TSNE_reduced_y_train,TSNE_reduced_y_test=model_selection.train_test_split(TSNE_reduced_features,Y_labels, test_size=0.4,random_state=1234,stratify=Y_labels)
        TSNE_linear_svc = svm.LinearSVC()
        TSNE_linear_svc.fit(TSNE_reduced_x_train,reduced_y_train)
        TSNE_pred_linear_svc = TSNE_linear_svc.predict(TSNE_reduced_x_test)
        acc=metrics.accuracy_score(reduced_y_test, TSNE_pred_linear_svc)
        lr_acc_list.append(acc)
        print("Iteration:",it," learning rate:",lr,"Acc:",acc)
    tsne_acc_list_2.append(lr_acc_list)
len(tsne_acc_list_2[0])
tsne_acc_list_2[0]
print("Learning rate at each iteration:",lr_list)
for i in range(5):
    print("Iteration:",iter_list[i],"  Acc:",tsne_acc_list_2[0][i*6:(i+1)*6])
# If n_components = 2 
plt.figure(figsize=(12,8))
plt.title('t-SNE components')
plt.scatter(TSNE_reduced_features[:,0], TSNE_reduced_features[:,1])
# If n_components = 3
plt.figure(figsize=(12,8))
plt.title('t-SNE components')
plt.scatter(TSNE_reduced_features[:,0], TSNE_reduced_features[:,1])
plt.scatter(TSNE_reduced_features[:,1], TSNE_reduced_features[:,2])
plt.scatter(TSNE_reduced_features[:,2], TSNE_reduced_features[:,0])<jupyter_output><empty_output><jupyter_text>4th VarianceLow Variance filter: We apply this approach to identify and drop constant variables from the dataset. The target variable is not unduly affected by variables with low variance, and hence these variables can be safely dropped<jupyter_code>variance = X_train.var(axis=0)
thredholds=[0.5,0.7,1,1.5,2]
li=[]
for thred in thredholds:
    index=(variance > thred)
    li.append(np.sum(index))
    
li
thredholds=[0.5,0.7,1,1.5,2]
V_acc_list=[]
for thred in thredholds:
    index=(variance > thred)
    print("The reduced dimension is:",np.sum(index)," The variance thredhold is:",thred)
    LV_reduced_features=X_features.loc[:,index]
    LV_reduced_x_train,LV_reduced_x_test,LV_reduced_y_train,LV_reduced_y_test=model_selection.train_test_split(LV_reduced_features,Y_labels, test_size=0.4,random_state=1234,stratify=Y_labels)
    LV_linear_svc = svm.LinearSVC()
    LV_linear_svc.fit(LV_reduced_x_train,LV_reduced_y_train)
    LV_pred_linear_svc = LV_linear_svc.predict(LV_reduced_x_test)
    acc=metrics.accuracy_score(LV_reduced_y_test, LV_pred_linear_svc)
    V_acc_list.append(acc)
print(V_acc_list)
plt.figure(figsize=[20, 10])
plt.plot(thredholds, V_acc_list)
plt.xlabel("Dimension",fontdict={'size':18})
plt.ylabel("Accuracy",fontdict={'size':18})
plt.xticks(fontsize=18)
plt.yticks(fontsize=18)
# plt.title("Low Variance Filter")
plt.show()
count=0
for thred in thredholds:
    index=(variance > thred)
    print(" The variance thredhold is: ",thred," The reduced dimension is: ",np.sum(index)," The accuracy is: ",V_acc_list[count])<jupyter_output>The variance thredhold is:  0.5  The reduced dimension is:  618  The accuracy is:  0.8965101480340277
 The variance thredhold is:  0.7  The reduced dimension is:  399  The accuracy is:  0.8965101480340277
 The variance thredhold is:  1  The reduced dimension is:  230  The accuracy is:  0.8965101480340277
 The variance thredhold is:  1.5  The reduced dimension is:  110  The accuracy is:  0.8965101480340277
 The variance thredhold is:  2  The reduced dimension is:  58  The accuracy is:  0.8965101480340277<jupyter_text>All Methods (Including the ones didn't show above )Missing Value Ratio: If the dataset has too many missing values, we use this approach to reduce the number of variables. We can drop the variables having a large number of missing values in themHigh Correlation filter: A pair of variables having high correlation increases multicollinearity in the dataset. So, we can use this technique to find highly correlated features and drop them accordinglyBoth Backward Feature Elimination and Forward Feature Selection techniques take a lot of computational time and are thus generally used on smaller datasets    Forward:* We start with a single feature. Essentially, we train the model n number of times using each feature separately* The variable giving the best performance is selected as the starting variable* Then we repeat this process and add one variable at a time. The variable that produces the highest increase in performance is retained* We repeat this process until no significant improvement is seen in the model’s performanceBackward:* We first take all the n variables present in our dataset and train the model using them* We then calculate the performance of the model* Now, we compute the performance of the model after eliminating each variable (n times), i.e., we drop one variable every time and train the model on the remaining n-1 variables* We identify the variable whose removal has produced the smallest (or no) change in the performance of the model, and then drop that variable* Repeat this process until no variable can be droppedFactor Analysis: This technique is best suited for situations where we have highly correlated set of variables. It divides the variables based on their correlation into different groups, and represents each group with a factorIndependent Component Analysis: We can use ICA to transform the data into independent components which describe the data using less number of componentsISOMAP: We use this technique when the data is strongly non-lineart-SNE: This technique also works well when the data is strongly non-linear. It works extremely well for visualizations as wellUMAP: This technique works well for high dimensional data. Its run-time is shorter as compared to t-SNE Note: metrics to decide which dimensionality reduction method to use* original dimension is high or low: t-SNE,UMAP* dataset is large or small:  t-SNE||random forest|| forward/backward feature selection, UMAP||PCA* data is linear or strongly non-linear: PCA, t-SNE* data is highly correlated or independent: FA||High correlation filter, Low-variance Filter  Check Results<jupyter_code>RF_reduced_features.shape
PCA_reduced_features.shape
UMAP_reduced_features.shape
TSNE_reduced_features.shape
LDA_reduced_features.shape
RF_reduced_x_train,RF_reduced_x_test,RF_reduced_y_train,RF_reduced_y_test=model_selection.train_test_split(RF_reduced_features,Y_labels, test_size=0.4,random_state=1234,stratify=Y_labels)
PCA_reduced_x_train,PCA_reduced_x_test,PCA_reduced_y_train,PCA_reduced_y_test=model_selection.train_test_split(PCA_reduced_features,Y_labels, test_size=0.4,random_state=1234,stratify=Y_labels)
UMAP_reduced_x_train,UMAP_reduced_x_test,UMAP_reduced_y_train,UMAP_reduced_y_test=model_selection.train_test_split(UMAP_reduced_features,Y_labels, test_size=0.4,random_state=1234,stratify=Y_labels)
TSNE_reduced_x_train,TSNE_reduced_x_test,TSNE_reduced_y_train,TSNE_reduced_y_test=model_selection.train_test_split(TSNE_reduced_features,Y_labels, test_size=0.4,random_state=1234,stratify=Y_labels)
UMAP_reduced_x_train,UMAP_reduced_x_test,UMAP_reduced_y_train,UMAP_reduced_y_test=model_selection.train_test_split(UMAP_reduced_features,Y_labels, test_size=0.4,random_state=1234,stratify=Y_labels)
TSNE_reduced_x_train,TSNE_reduced_x_test,TSNE_reduced_y_train,TSNE_reduced_y_test=model_selection.train_test_split(TSNE_reduced_features,Y_labels, test_size=0.4,random_state=1234,stratify=Y_labels)
LDA_reduced_x_train,LDA_reduced_x_test,LDA_reduced_y_train,LDA_reduced_y_test=model_selection.train_test_split(LDA_reduced_features,Y_labels, test_size=0.4,random_state=1234,stratify=Y_labels)
RF_linear_svc = svm.LinearSVC()
RF_linear_svc.fit(RF_reduced_x_train,RF_reduced_y_train)
RF_pred_linear_svc = RF_linear_svc.predict(RF_reduced_x_test)
metrics.accuracy_score(RF_reduced_y_test, RF_pred_linear_svc)
PCA_linear_svc = svm.LinearSVC()
PCA_linear_svc.fit(PCA_reduced_x_train,PCA_reduced_y_train)
PCA_pred_linear_svc = PCA_linear_svc.predict(PCA_reduced_x_test)
metrics.accuracy_score(PCA_reduced_y_test, PCA_pred_linear_svc)
LDA_linear_svc = svm.LinearSVC()
LDA_linear_svc.fit(LDA_reduced_x_train,LDA_reduced_y_train)
LDA_pred_linear_svc = LDA_linear_svc.predict(LDA_reduced_x_test)
metrics.accuracy_score(LDA_reduced_y_test, LDA_pred_linear_svc)
UMAP_linear_svc = svm.LinearSVC()
UMAP_linear_svc.fit(UMAP_reduced_x_train,UMAP_reduced_y_train)
UMAP_pred_linear_svc = UMAP_linear_svc.predict(UMAP_reduced_x_test)
metrics.accuracy_score(UMAP_reduced_y_test, UMAP_pred_linear_svc)
TSNE_linear_svc = svm.LinearSVC()
TSNE_linear_svc.fit(TSNE_reduced_x_train,TSNE_reduced_y_train)
TSNE_pred_linear_svc = TSNE_linear_svc.predict(TSNE_reduced_x_test)
metrics.accuracy_score(TSNE_reduced_y_test, TSNE_pred_linear_svc)<jupyter_output>D:\anaconda\envs\AI_1\lib\site-packages\sklearn\utils\validation.py:63: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  return f(*args, **kwargs)
D:\anaconda\envs\AI_1\lib\site-packages\sklearn\svm\_base.py:986: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
  "the number of iterations.", ConvergenceWarning)
{'fraction_non_alphanumeric': 0.08807345319598182, 'fraction_numerical': 0.04085354149337769, 'mean_word_length': 6.377625678546141, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 67, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 25, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '432721', 'license': 'MIT', 'n_tokens_mistral': 11394, 'n_tokens_neox': 10229, 'n_words': 2812, 'path': 'classical_methods.ipynb', 'repo_name': 'BrianWU-S/Dimentionality_Reduction', 'size': '31258'}