<jupyter_start><jupyter_text>Feature Selection for Regression Problem using ANOVA(Analysis of Variance) or F-Test - A univariate test,Linear model for testing the individual effect of each of features with target.- ANOVA assumes a linear relationship between the features and the target, and also that the variables are normally distributed.- It’s well-suited for continuous variables and requires a binary target, but sklearn extends it to regression problems, also. Youtube Video Explanation : https://youtu.be/wElwOM88xJQ<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from scipy import stats
# Load The Dataset
# https://data.world/nrippner/ols-regression-challenge
import pandas as pd
df_cancer = pd.read_csv("https://raw.githubusercontent.com/atulpatelDS/Data_Files/master/Cancer/cancer_reg.csv",
                     encoding = "ISO-8859-1")
df_cancer.head()
df_Tar = df_cancer["TARGET_deathRate"]
df_cancer.drop(labels=["Geography","binnedInc","TARGET_deathRate"],axis=1,inplace=True)
df_cancer = pd.concat([df_cancer,df_Tar],axis=1)
df_cancer.head()
df_cancer.isnull().sum()
df_cancer = df_cancer.apply(lambda x: x.fillna(x.mean()),axis = 0)
#df.apply(lambda x: x.fillna(x.mean()),axis=0)
df_cancer.isnull().sum()
X = df_cancer.drop(labels=["TARGET_deathRate"],axis=1) # Features
y = df_cancer["TARGET_deathRate"] # Target
print(type(X))
print(type(y))
print(X.shape)
print(y.shape)
# Lets perform Feature selection using Anova or F-Test Linear Regression sklearn 
from sklearn.feature_selection import f_regression, SelectKBest
"""
f_regression()
Univariate linear regression tests.

Linear model for testing the individual effect of each of many regressors.
This is a scoring function to be used in a feature selection procedure, not
a free standing feature selection procedure.

This is done in 2 steps:

1. The correlation between each regressor and the target is computed,
   that is, ((X[:, i] - mean(X[:, i])) * (y - mean_y)) / (std(X[:, i]) *
   std(y)).
2. It is converted to an F score then to a p-value.
"""
# apply SelectKBest class to extract top 30 best features
fs = SelectKBest(score_func=f_regression,k=30) ## Select k as per your business understaing
# Apply feature selection
fit = fs.fit(X,y)
print(np.round(fit.pvalues_,4))
print(np.round(fit.scores_,3))
features_score = pd.DataFrame(fit.scores_)
features_pvalue = pd.DataFrame(np.round(fit.pvalues_,4))
features = pd.DataFrame(X.columns)
feature_score = pd.concat([features,features_score,features_pvalue],axis=1)
# Assign the column name
feature_score.columns = ["Input_Features","F_Score","P_Value"]
print(feature_score.nlargest(30,columns="F_Score"))<jupyter_output>Input_Features     F_Score  P_Value
17        PctBachDeg25_Over  938.976311   0.0000
2             incidenceRate  770.736162   0.0000
24   PctPublicCoverageAlone  770.417642   0.0000
5            povertyPercent  688.332859   0.0000
3                 medIncome  685.295384   0.0000
16             PctHS25_Over  596.004470   0.0000
23        PctPublicCoverage  595.943105   0.0000
18       PctEmployed16_Over  571.370931   0.0000
20       PctPrivateCoverage  533.339204   0.0000
19     PctUnemployed16_Over  508.904879   0.0000
21  PctPrivateCoverageAlone  362.259196   0.0000
29     PctMarriedHouseholds  286.654761   0.0000
15          PctBachDeg18_24  275.027280   0.0000
22       PctEmpPrivCoverage  234.491705   0.0000
11           PercentMarried  233.399629   0.0000
13               PctHS18_24  224.382260   0.0000
26                 PctBlack  215.384630   0.0000
28             PctOtherRace  113.908891   0.0000
27                 PctAsian  109.522761   0.0000
25                 P[...]<jupyter_text>Feature Selection for Classification Problem using ANOVA(Analysis of Variance) or F-Testhttps://datascience.stackexchange.com/questions/74465/how-to-understand-anova-f-for-feature-selection-in-python-sklearn-selectkbest-w<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from scipy import stats
import math
#Load the dataset #https://www.kaggle.com/burak3ergun/loan-data-set
df_loan = pd.read_csv("https://raw.githubusercontent.com/atulpatelDS/Data_Files/master/Loan_Dataset/loan_data_set.csv")
df_loan
df_loan.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 614 entries, 0 to 613
Data columns (total 13 columns):
 #   Column             Non-Null Count  Dtype  
---  ------             --------------  -----  
 0   Loan_ID            614 non-null    object 
 1   Gender             601 non-null    object 
 2   Married            611 non-null    object 
 3   Dependents         599 non-null    object 
 4   Education          614 non-null    object 
 5   Self_Employed      582 non-null    object 
 6   ApplicantIncome    614 non-null    int64  
 7   CoapplicantIncome  614 non-null    float64
 8   LoanAmount         592 non-null    float64
 9   Loan_Amount_Term   600 non-null    float64
 10  Credit_History     564 non-null    float64
 11  Property_Area      614 non-null    object 
 12  Loan_Status        614 non-null    object 
dtypes: float64(4), int64(1), object(8)
memory usage: 62.5+ KB<jupyter_text>**If the categorical variables have highly distinct values we could try binning before one hot encoding and/or reducing the poorly represented levels into one "other" category.**<jupyter_code># Remove all null value
df_loan.dropna(inplace=True)
# drop the column("Loan_ID")
df_loan.drop(labels=["Loan_ID"],axis=1,inplace=True)
df_loan.info()
df_loan.head()
df_loan.reset_index(drop=True,inplace=True)
'''
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
cols = df_loan.columns.tolist()
for column in cols:
    if df_loan[column].dtype == 'object':
        df_loan[column] = le.fit_transform(df_loan[column])
'''
# encode the Categorical Variable
from sklearn.preprocessing import LabelEncoder 
class MultiColumnLabelEncoder:
    def __init__(self,columns = None):
        self.columns = columns # array of column names to encode

    def fit(self,X,y=None):
        return self # not relevant here

    def transform(self,X):
        '''
        Transforms columns of X specified in self.columns using
        LabelEncoder(). If no columns specified, transforms all
        columns in X.
        '''
        output = X.copy()
        if self.columns is not None:
            for col in self.columns:
                output[col] = LabelEncoder().fit_transform(output[col])
        else:
            for colname,col in output.iteritems():
                output[colname] = LabelEncoder().fit_transform(col)
        return output

    def fit_transform(self,X,y=None):
        return self.fit(X,y).transform(X)
cat_cols = df_loan.select_dtypes(include=["object"]).columns
cat_cols
df_loan = MultiColumnLabelEncoder(columns = cat_cols).fit_transform(df_loan)
df_loan.info()
X = df_loan.iloc[:,0:-1]
X.head(1)
y = df_loan["Loan_Status"]
X.head()
# Define feature selection
from sklearn.feature_selection import f_classif,SelectKBest 
fs = SelectKBest(score_func=f_classif,k=5)
# Apply feature selection
fs.fit(X,y)
# True = Top 5 columns as we mentioned k =5
# False = Allthe remaining columns excpet k =5
fs.get_support()
fs.get_support().sum()
def list_ceil(x):
    return[math.ceil(i) for i in x]
print("P_Values :")
print(np.round(fs.pvalues_,4))
print("F Values :")
print(fs.scores_)
list_ceil(fs.scores_)
features_score = pd.DataFrame(fs.scores_)
features_pvalue = pd.DataFrame(np.round(fs.pvalues_,4))
features = pd.DataFrame(X.columns)
feature_score = pd.concat([features,features_score,features_pvalue],axis=1)
# Assign the column name
feature_score.columns = ["Input_Features","Score","P_Value"]
print(feature_score.nlargest(5,columns="Score"))<jupyter_output>Input_Features       Score  P_Value
9  Credit_History  186.122727   0.0000
1         Married    6.107491   0.0138
7      LoanAmount    2.473684   0.1164
3       Education    2.249320   0.1343
0          Gender    1.997174   0.1582
{'fraction_non_alphanumeric': 0.09925187032418953, 'fraction_numerical': 0.06396508728179551, 'mean_word_length': 3.541902604756512, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '128219', 'license': 'MIT', 'n_tokens_mistral': 2981, 'n_tokens_neox': 2594, 'n_words': 688, 'path': 'Feature_Engineering/Feature Selection using ANOVA Test for Classification and Regression - Tutorial 4.ipynb', 'repo_name': 'atulpatelDS/Youtube', 'size': '8020'}