<jupyter_start><jupyter_text>Feature Selections on Santander's Dataset:<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.model_selection import train_test_split

df = pd.read_csv('train.csv')
df<jupyter_output><empty_output><jupyter_text>DATA PREPROCESSINGData preprocessing is a data mining technique which is used to transform the raw data in a useful and efficient format.<jupyter_code># getting the shape
df.shape
# reading the head of the data
df.head(10)
# describing the data
df.describe()
# removing duplicate rows
df.drop_duplicates(inplace=True)
df.shape
# creating column name's list
col = df.columns.to_list()

# printing number of unique values in each column.
df.nunique()
# checking the null or missing values in dataframe
plt.figure(figsize=(10,10))
sns.heatmap(data=df.isna(),yticklabels=False,cbar=False,cmap='Set2')
plt.show()
# columns information
df.info()
# Check the presence of null data
[col for col in df.columns if df[col].isnull().sum() > 0]<jupyter_output><empty_output><jupyter_text>1. Removal of Constant Features<jupyter_code>from sklearn.feature_selection import VarianceThreshold
sel = VarianceThreshold(threshold=0)
sel.fit(df)
sum(sel.get_support())
print(len([
        x for x in df.columns
        if x not in df.columns[sel.get_support()]
    ]))

constant_features=[x for x in df.columns if x not in df.columns[sel.get_support()]]<jupyter_output>34<jupyter_text>So there are 34 constant feature columns with same value in all the data out of 377 columns. So in a single step we are able to identify 38 features with zero variance which does not provide any predictive information of output variable. So we can straight away delete these columns. In the below statement we delete all the 34 constant feature columns.<jupyter_code>constant_features
X_train = sel.transform(df)
df.shape<jupyter_output><empty_output><jupyter_text>2. Quasi - Constant FeaturesThese are the feature columns where most of the data has the same value. So in this section we consider the threshold as 99% ie., a column is quasi-constant feature column if more than 99% of the data has same value. We exclude all the quasi constant features as part of the pre-processing step. This is an important step when we have data with thousands of features and in these cases there is a high chance of having lot of constant features and quasi-constant features. It is ideal to test and remove these as they will have almost zero predictability of output variabe (Y).The quasi-constant features and once identified delete those columns from original data.<jupyter_code>df = pd.read_csv('train.csv')
sel = VarianceThreshold(
    threshold=0.01)

sel.fit(df)
sum(sel.get_support())
len(df.columns[sel.get_support()])
print(
    len([
        x for x in df.columns
        if x not in df.columns[sel.get_support()]
    ]))

q_constant=[x for x in df.columns if x not in df.columns[sel.get_support()]]
q_constant<jupyter_output><empty_output><jupyter_text>As we observe there are 97 features out of 274 total feature columns with more than 99.9% same value in all the data. So this simple test tells us that around 40% of the feature columns are not needed in building the machine learning model.<jupyter_code>df['ind_var31'].value_counts() / np.float(len(df))
X_train = sel.transform(df)
X_train.shape<jupyter_output><empty_output><jupyter_text>3. Duplicated FeaturesSometimes there is a chance that duplicate features exist in the data. Usually this can happen when we are one-hot encoding the data. So it is always preferable to delete the duplicate features in the data. This may not happen always but it is ideal to test this and if any duplicate features exist then drop them. Below is the code to identify the deuplicate features. As we see below that there are duplicate features in the data.<jupyter_code>df = pd.read_csv('train.csv')
data_t = df.T
data_t.head()
data_t.duplicated().sum()
data_t[data_t.duplicated()]
duplicated_features = data_t[data_t.duplicated()].index.values
duplicated_features
data_unique = data_t.drop_duplicates(keep='first').T
data_unique.shape
duplicated_features = [col for col in data_t.columns if col not in data_unique.columns]
duplicated_features<jupyter_output><empty_output><jupyter_text>As we observe here there are 62 duplicate columns in the data. We can drop all these columns as our pre-processing step. Below is the code to drop all the duplicated columns contained in duplicated_features.  4.Pearson Correlation:-In statistics, the Pearson correlation coefficient, also referred to as Pearson's r, the Pearson product-moment correlation coefficient, or the bivariate correlation, is a statistic that measures linear correlation between two variables X and Y. It has a value between +1 and −1. ![image.png](attachment:image.png)<jupyter_code>df = pd.read_csv('train.csv')
from sklearn.preprocessing import LabelEncoder
le=LabelEncoder()
df.head()
df['TARGET']=le.fit_transform(df['TARGET'])
col=df.columns
col=col.drop('TARGET')
from scipy.stats import pearsonr
cor=[]
for data1 in col:
    corr, _ = pearsonr(df[data1], df['TARGET'])
    cor.append([data1,corr])
cor.sort(key=lambda x: x[1],reverse=True)
cor<jupyter_output>C:\Users\hp\anaconda3\lib\site-packages\scipy\stats\stats.py:3845: PearsonRConstantInputWarning: An input array is constant; the correlation coefficent is not defined.
  warnings.warn(PearsonRConstantInputWarning())<jupyter_text>5. Fishers Score:-Fisher scoring is a hill-climbing algorithm for getting results - it maximizes the likelihood by getting successively closer and closer to the maximum by taking another step ( an iteration).  It knows when it has reached the top of the hill in that taking another step does not increase the likelihood.  It is known to be an efficient procedure - not many steps are usually needed - and generally converges to an answer. When this is the case you do not need to be concerned about it - accept what  you have got. Changing metaphors,  you can drive the care without knowing how the internal combustion engine works.<jupyter_code>df = pd.read_csv('train.csv')
data = df.groupby(df.TARGET)
data1 = data.get_group(0)
data2 = data.get_group(1)
def fisher(x,y):
    return abs(((np.mean(data1) - np.mean(data2))**2)/(np.var(data1) + np.var(data2)))
fisher(data1,data2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08206734534064213, 'fraction_numerical': 0.010649960845732186, 'mean_word_length': 5.347912524850894, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '372503', 'license': 'MIT', 'n_tokens_mistral': 1918, 'n_tokens_neox': 1849, 'n_words': 742, 'path': 'FeatureSelection.ipynb', 'repo_name': 'anujgoyal1234/ML-Projects', 'size': '6385'}