<jupyter_start><jupyter_text>Filter based Feature Selection methods- Filter methods select features from a dataset independently for any machine learning algorithm. - These methods rely only on the characteristics of these variables, so features are filtered out of the data before learning begins.- **Univariate filter methods** evaluate each feature individually. - These methods consist of providing a score to each feature, often based on statistical tests.   Video Explanation of Variance Threshold : https://youtu.be/Z141lNzivXU **Filter Methods: Advantages**- They are computationally inexpensive, you can process thousands of features in a matter of seconds.- Filter methods are very good for eliminating irrelevant, redundant, constant, duplicated, and correlated features. **Filter Method Types**1. Basic Statistical Filter Methods    - VarianceThreshod (Remove the Constant Feature and Quasi-Constant Features)    - Remove Duplicate Features2. Correlation & Ranking based statistical Filter Methods    - Pearson’s correlation coefficient    - Spearman’s rank coefficient    - Kendall’s rank coefficient3. Statistical Test based Methods    - Anova or F-Test    - Mutual Information    - Chi Square      Variance Threshold**Removing Numerical features with low variance**- We simply compute the variance of each features, and we select the subset of features based on a user-specified threshold.- We assume that features with a higher variance may contain more useful information.- **This feature selection algorithm looks only at the features (X), not the desired outputs (y), and can thus be used for unsupervised learning.**- As we are not taking the relationship between features variables or feature and target variables into account, which is one of the drawbacks of Variance Threshold filter method.- It is applicable only on Numerical features.<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
# Lets create a dataframe
col1 = np.ones(100)
col2 = np.zeros(100)
col2[:1] = 1
col3 = np.random.randint(15,30,size = 100)
col4 = np.ones(100)
col4[:40] = 0
col5 = np.zeros(100)
col5[:20] = 1
col6 = np.random.randint(30,40,size = 100)
col7 = np.ones(100)
col8 = np.zeros(100)
col8[:4] = 1
df = pd.DataFrame({"A":col1,"B":col2,"C":col3,"D":col4,"E":col5,"F":col6,"G":col7})
df.head(8)
df_var = df.var()
df_var = df.var().to_frame() #Convert Series to DataFrame
df_var
df_var.shape
df_var = df_var.rename(columns={0:"Variance"})
df_var
# select those columns where variance is geater than 1 , variancethreshold = 1%
df1 = df.loc[:, df.var() > 0.01]
df1<jupyter_output><empty_output><jupyter_text>**Feature Selection - Variance Threshold : sklearn.feature_selection.VarianceThreshold**<jupyter_code>from sklearn.feature_selection import VarianceThreshold
# Lets use the sklearn
selector = VarianceThreshold(threshold=0.01) # variance 1%
# fit the object to the data
selector.fit(df)
# Get features which have the variance greater than the set threshold value = 0.01
selector.get_support(indices=False)
df.columns
df.columns[selector.get_support()]
#Get features which have the variance less than the set threshold value using list comprehension
selected_cols = [column for column in df.columns if column not in df.columns[selector.get_support()]]
#features which have the variance less than the set threshold value
selected_cols
# lets drop those columns from the dataset where variance is < 1
df2 = df.drop(labels=selected_cols,axis=1)
df2.head()
df_op = selector.transform(df)
df_op<jupyter_output><empty_output><jupyter_text>Feature Selection Method : Variance Threshold to remove the Constant Features- Those features which contain constant values (only one value for all the outputs or target values) in the dataset.- These features provide no information that allows ML models to predict the target.threshold = 0<jupyter_code>#https://www.kaggle.com/iabhishekofficial/mobile-price-classification --Load the dataset
#df_m = pd.read_csv("https://raw.githubusercontent.com/atulpatelDS/Data_Files/master/Mobile_Price/train.csv")#,
                     #encoding = "ISO-8859-1")
# https://www.kaggle.com/c/santander-customer-satisfaction/data --Load the dataset
df_s = pd.read_csv("https://raw.githubusercontent.com/atulpatelDS/Data_Files/master/Santander_CS/train.csv")#,
                     #encoding = "ISO-8859-1")
#df_m.head()
df_s.head()
X_s = df_s.iloc[:,0:-1]
y_s = df_s["TARGET"]
#X_m = df_m.iloc[:,0:-1]
#y_m = df_m["price_range"]
#y_m = df_m["color"]
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test = train_test_split(X_s,y_s,test_size = 0.3,random_state = 42)
#X_train,X_test,y_train,y_test = train_test_split(X_m,y_m,test_size = 0.3,random_state = 42)
X_train.shape,X_test.shape
# Lets apply variancethreshold
constant_selector= VarianceThreshold(threshold=0)
constant_selector.fit(X_train)
# Get features which have the variance greater than the set threshold value = 0.0
sum(constant_selector.get_support())
# Above we can get how many features are not constant
# find the number of constant features with the help of the following script:
constant_selected_cols = [column for column in X_train.columns if column not in X_train.columns[constant_selector.get_support()]]
# Constant Columns
len(constant_selected_cols)
# lets drop those columns from the dataset where variance is = 0
X_train_op = X_train.drop(labels=constant_selected_cols,axis=1)
X_train_op.head()
# Other method ---lets drop those columns from the dataset where variance is = 0
X_train_ac = constant_selector.transform(X_train)
X_test_ac = constant_selector.transform(X_test)
X_train_ac.shape, X_test_ac.shape<jupyter_output><empty_output><jupyter_text>Feature Selection Method : Variance Threshold to remove the Quasi-Constant Features- Features that are almost constant. In other words, these features have the same values for a very large subset of the outputs. - Such features are not very useful for making predictions. There is no rule as to what should be the threshold for the variance of quasi-constant features. However, as a rule of thumb, remove those quasi-constant features that have more than 99% similar values for the output observations.- We will create a quasi-constant filter with the help of VarianceThreshold function. However, instead of passing 0 as the value for the threshold parameter, we will pass 0.01, which means that if the variance of the values in a column is less than 0.01, remove that column. In other words, remove feature column where approximately 99% of the values are similar.threshold = 0.01<jupyter_code># Lets create threshold = 0.01 to remove the quasi-constant columns
quasi_constant_selector = VarianceThreshold(threshold=0.01)
quasi_constant_selector.fit(X_train)
X_s.shape,y_s.shape
#X_m.shape,y_m.shape
# Get features which have the variance greater than the set threshold value = 0.01
sum(quasi_constant_selector.get_support())
# Filter out the Quasi-constant features
quasi_features = [cols for cols in X_train.columns if cols not in X_train.columns[quasi_constant_selector.get_support()]]
# Check the Quasi-constant columns count
len(quasi_features)
print("Total Features :",len(X_train.columns))
print("Total constant features :",len(constant_selected_cols))
print("Total non-quasi constant features :",len(X_train.columns)- len(quasi_features))
frame = X_train.var().to_frame()
#frame = X_train.var().to_frame()
frame = frame.rename(columns={0:"Variance"})
print("Total Features :",sum(frame.Variance.values>=0))
print("Total constant features :",sum(frame.Variance.values==0))
print("Total non constant and non-quast constant features :",sum(frame.Variance.values > 0.01))
# lets transfor the train nad test data using fit selector
X_train_aq = quasi_constant_selector.transform(X_train)
X_test_aq = quasi_constant_selector.transform(X_test)
X_train_aq.shape, X_test_aq.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08884973571608357, 'fraction_numerical': 0.01824817518248175, 'mean_word_length': 5.600498338870432, 'pattern_counts': {'":': 7, '<': 18, '<?xml version=': 0, '>': 20, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '177380', 'license': 'MIT', 'n_tokens_mistral': 2407, 'n_tokens_neox': 2296, 'n_words': 824, 'path': 'Feature_Engineering/Feature Selection using VarianceThreshold to remove Constant and Quasi Constant Features -Tutorial 2.ipynb', 'repo_name': 'atulpatelDS/Youtube', 'size': '7946'}