<jupyter_start><jupyter_text>Data Mining and Machine Learning Feature Selection in Supervised Classification Edgar Acuna September 2021 Modules required: feature_selection from scikit-learn, feature selection from Arizona State University , skrebate, Orange and mRMR  Datasets: Diabetes, Bupa, Breast-Cancer-Wisconsin, vehicle, Landsat, Loan and Heart-Cleveland<jupyter_code>import warnings
warnings.filterwarnings("ignore")
import pandas as pd
import numpy as np
import math as m
from sklearn.model_selection import train_test_split
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2, SelectPercentile, f_classif, mutual_info_classif
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Example 1: Diabetes dataset<jupyter_code>url= "http://academic.uprm.edu/eacuna/diabetes.dat"
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
data = pd.read_table(url, names=names,header=None)
y=data['class']
X=data.iloc[:,0:8]<jupyter_output><empty_output><jupyter_text>I. Feature Selection by Visualization using boxplots<jupyter_code>print(data.shape)
data.head()
#normalizando los datos
data1=data.drop('class',axis=1)
data1_norm=(data1 - data1.min()) / (data1.max() - data1.min())
data1_norm['class']=data['class']
plt.figure(figsize=(10,50))
df_long = pd.melt(data1_norm, "class", var_name="a", value_name="c")
sns.factorplot("a", hue="class", y="c", data=df_long, kind="box")<jupyter_output><empty_output><jupyter_text>Comment: From the boxplots we can see  that features: plas, age,mass, preg,and pedi have boxplot not so overlaped  II-Feature selection using statistical measures  II-1 Using the Chi-Square test  It is desirable to have discretized attributes and two classes. I do not recommend to use this method because the Chi-Square test is approximated and it is not robust.<jupyter_code>y=data['class']
X=data.iloc[:,0:8]
# Funcion auxiliar para discretizar cualquier columna de un dataframe
def disc_col_ew(df,str,k,out):
    df1=df[str]
    bins=np.linspace(df1.min(), df1.max(),k)
    if out=="num":
        df1=pd.cut(df1,bins=bins,include_lowest=True, right=True,labels=False)
    else:
        bins[0]=float('-inf')
        bins[k-1]=float('inf')
        df1=pd.cut(df1,bins=bins,include_lowest=True, right=True)  
    return df1
# funcion auxiliar para determinar el numero optimo de intervalos segun la formula de scott
def nclass_scott(x):
    h=3.5*(np.var(x,ddof=1)**.5)*len(x)**(-.3333)
    intervals=m.ceil((max(x)-min(x))/h)
    return int(intervals)
#Funcion para discretizar todas las colunmnas de un dataframe
def disc_ew(df,out):
    name=df.columns.tolist()
    disc=pd.DataFrame()
    for name in df.columns.tolist():
        k=nclass_scott(df[name])
        disc[name]=disc_col_ew(df,name,k,out)
    return disc
#discretizando las columnas de la matriz predictora X de diabetes
diab_disc=disc_ew(X,out="num")
#Choosing the best three features with Chi-square and scikit-learn
y1=y.to_numpy()
X1=diab_disc.to_numpy()
test = SelectKBest(score_func=chi2, k=3)
fit = test.fit(X1, y1)
# summarize scores
np.set_printoptions(precision=3)
print(fit.scores_)
#Ranking the features according to their scores 
idx = np.argsort(fit.scores_)
print(X.columns[idx[::-1]])
#Printing the subset containing the best three fetaures
features= fit.transform(X1)
print(features)<jupyter_output>[[ 4 11  6]
 [ 0  6  2]
 [ 6 14  2]
 ...
 [ 3  9  1]
 [ 0 10  5]
 [ 0  7  0]]<jupyter_text>Comentario: Las tres mejores variables con la prueba de Chi-square son plas, age y preg por tener el Chi-Square mas alto.  II-2 Using the F test to find the best features  Either  scikit-learn or the ASU library can be used<jupyter_code># Selecion  de Features usando los p-values de la F-test como score
# Aqui usamos scikit-learn y se selecciona el 30% de deatures con el mas alto score
selector = SelectPercentile(f_classif, percentile=30)
fit=selector.fit(X, y)
scores = -np.log10(selector.pvalues_)
scores /= scores.max()
print(scores)
features= fit.transform(X)
# Imprime los datos de las tres mejores features
print(features)<jupyter_output>[0.221 1.    0.027 0.034 0.084 0.378 0.14  0.253]
[[148.   33.6  50. ]
 [ 85.   26.6  31. ]
 [183.   23.3  32. ]
 ...
 [121.   26.2  30. ]
 [126.   30.1  47. ]
 [ 93.   30.4  23. ]]<jupyter_text>Comentario: Las tres, mejores variables con la prueba de F son plas,mass y age<jupyter_code>import numpy as np
from sklearn.feature_selection import f_classif


def f_score(X, y):
    """
    This function implements the anova f_value feature selection (existing method for classification in scikit-learn),
    where f_score = sum((ni/(c-1))*(mean_i - mean)^2)/((1/(n - c))*sum((ni-1)*std_i^2))
    Input
    -----
    X: {numpy array}, shape (n_samples, n_features)
        input data
    y : {numpy array},shape (n_samples,)
        input class labels
    Output
    ------
    F: {numpy array}, shape (n_features,)
        f-score for each feature
    """

    F, pval = f_classif(X, y)
    return F
def feature_ranking(F):
    """
    Rank features in descending order according to f-score, the higher the f-score, the more important the feature is
    """
    idx = np.argsort(F)
    return idx[::-1]
scoref =f_score(X, y)
idx = feature_ranking(scoref)
print(scoref,idx)<jupyter_output>[ 39.67  213.162   3.257   4.304  13.281  71.772  23.871  46.141] [1 5 7 0 6 4 3 2]<jupyter_text>Comentario: las tres variables mas importantes con la prueba de F son: plas,mass y age  III Uaing Information measures  Usando Mutual Information (Entropia)Using the  scikit-learn module<jupyter_code># Univariate feature selection with Mutual Information
scores = mutual_info_classif(X,y)
scores /= scores.max()
print(scores)
#Ranking the features according to their scores 
idx = np.argsort(scores)
print(X.columns[idx[::-1]])<jupyter_output>[0.38  1.    0.    0.223 0.295 0.665 0.104 0.731]
Index(['plas', 'age', 'mass', 'preg', 'test', 'skin', 'pedi', 'pres'], dtype='object')<jupyter_text>Comentario: Las tres variables usando el criterio de Mutual Information son: plas, mass y age IV-Using ReliefF to select the best features  IV-1 Usando el modulo skrebate<jupyter_code>from skrebate import ReliefF

df=data.drop('class',axis=1)
#Normalizando las predictoras
df_norm=(df - df.min()) / (df.max() - df.min())
features, labels = df_norm.values, data['class'].values
features
fs = ReliefF(n_neighbors=10)
fs.fit(features, labels)
for feature_name, feature_score in zip(data.columns,fs.feature_importances_):
    print (feature_name, feature_score)
yscore=fs.feature_importances_
oyscore=-np.sort(-yscore)
print(oyscore)
import matplotlib.pyplot as plt
x=np.arange(0,8)
plt.plot(x,oyscore,'k-',lw=3)<jupyter_output><empty_output><jupyter_text>Comentario: las tres variables mas importantes con el ReliefF son: plas,mass y skin  IV-2 Usando el Relief de Orange<jupyter_code>import Orange
df = Orange.data.Table("https://academic.uprm.edu/eacuna/diabetes.tab")
#ReliefF usando 10 vecinos mas cercanos y  una muestrd m=100 para updating de los pesos
print('Feature scores for best  features (scored individually):')
meas =Orange.preprocess.score.ReliefF(n_iterations=400,k_nearest=10)
scores = meas(df)
for attr, score in zip(df.domain.attributes, scores):
    print('%.3f' % score, attr.name)
#Ranking the features according to their scores 
idx = np.argsort(scores)
print(X.columns[idx[::-1]])<jupyter_output>Index(['plas', 'mass', 'preg', 'skin', 'age', 'pedi', 'pres', 'test'], dtype='object')<jupyter_text>The best features are: plas, mass and skin.  Example 2:  Bupa dataset<jupyter_code>#load data
url= "http://academic.uprm.edu/eacuna/bupa.dat"
names = ['mcv', 'alkphos', 'sgpt', 'aspar', 'gammagt', 'drinks', 'class']
data = pd.read_table(url, names=names,header=None)
y=data['class']
X=data.iloc[:,0:6]
y1=y.to_numpy()
X1=X.to_numpy()
features, labels = X.values, data['class'].values<jupyter_output><empty_output><jupyter_text>Using skrebate<jupyter_code>df=data.drop('class',axis=1)
#Normalizando las predictoras
df_norm=(df - df.min()) / (df.max() - df.min())
features, labels = df_norm.values, data['class'].values
df.info()
fs = ReliefF(n_neighbors=10)
fs.fit(features, labels)
for feature_name, feature_score in zip(data.columns,fs.feature_importances_):
    print (feature_name, feature_score)<jupyter_output>mcv 0.0028375286041189907
alkphos -0.0006200378071833658
sgpt 0.005445820136289467
aspar 0.0033427442123094295
gammagt 0.004875918205280923
drinks 0.00485507246376812<jupyter_text>El relielf de skrebate recomienda gammagt, aspar and sgpt   Using the ReliefF from Orange<jupyter_code>import Orange
df = Orange.data.Table("https://datasets.biolab.si/core/bupa.tab")
#ReliefF usando 10 vecinos mas cercanos y  una muestra de n=300 para updating de los pesos
print('Feature scores for best ten features (scored individually):')
meas =Orange.preprocess.score.ReliefF(n_iterations=300,k_nearest=10)
scores = meas(df)
for attr, score in zip(df.domain.attributes, scores):
    print('%.4f' % score, attr.name)<jupyter_output>Feature scores for best ten features (scored individually):
0.0020 mcv
0.0003 alkphos
0.0059 sgpt
0.0026 sgot
0.0051 gammagt
0.0065 drinks<jupyter_text>Example 3: Breast-cancer-Wisconsin<jupyter_code>names = [
    'ID','Clump', 'Unif_Cell_Size', 'Unif_Cell_Shape', 'Marginal_Adh', 'Single_Cell_Size', 'Bare_Nuclei', 'Bland_Chromatine', 'Normal_Nucleoi', 'Mitoses','class']
data = pd.read_table("https://academic.uprm.edu/eacuna/breast-cancer-wisconsin.data", names=names, header=None,sep=",",na_values="?")
data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 699 entries, 0 to 698
Data columns (total 11 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   ID                699 non-null    int64  
 1   Clump             699 non-null    int64  
 2   Unif_Cell_Size    699 non-null    int64  
 3   Unif_Cell_Shape   699 non-null    int64  
 4   Marginal_Adh      699 non-null    int64  
 5   Single_Cell_Size  699 non-null    int64  
 6   Bare_Nuclei       683 non-null    float64
 7   Bland_Chromatine  699 non-null    int64  
 8   Normal_Nucleoi    699 non-null    int64  
 9   Mitoses           699 non-null    int64  
 10  class             699 non-null    int64  
dtypes: float64(1), int64(10)
memory usage: 60.2 KB<jupyter_text>Using skrebate<jupyter_code>#Dropping the ID column
df=data.drop('ID',axis=1)
#Dropping the class column
df1=df.drop('class',axis=1)
#Normalizando las predictoras
df_norm=(df1 - df1.min()) / (df1.max() - df1.min())
features, labels = df_norm.values, df['class'].values
fs = ReliefF(n_neighbors=10)
fs.fit(features, labels)
for feature_name, feature_score in zip(df1.columns,fs.feature_importances_):
    print (feature_name, feature_score)
yscore=fs.feature_importances_
oyscore=-np.sort(-yscore)
print(oyscore)
x=np.arange(0,9)
plt.plot(x,oyscore,'k-',lw=3)<jupyter_output>[0.556 0.537 0.522 0.477 0.472 0.302 0.282 0.209 0.068]<jupyter_text>Using Relief from Orange<jupyter_code>import Orange
df = Orange.data.Table("https://academic.uprm.edu/eacuna/breast-cancer-wisconsin.tab")
#ReliefF usando 10 vecinos mas cercanos y  una muestrd m=400 para updating de los pesos
print('Feature scores for best ten features (scored individually):')
meas =Orange.preprocess.score.ReliefF(n_iterations=500,k_nearest=1)
scores = meas(df)
for attr, score in zip(df.domain.attributes, scores):
    print('%.3f' % score, attr.name)<jupyter_output>Feature scores for best ten features (scored individually):
0.576 Clump
0.210 Unif_Cell_Size
0.212 Unif_Cell_Shape
0.128 Marginal_Adh
0.178 Single_Cell_Size
0.634 Bare_Nuclei
0.524 Bland_Chromatine
0.398 Normal_Nucleoi
0.038 Mitoses<jupyter_text>Comment:The best attributes are:6(Bare Nuclei),1 (Clump Thickness),7 (Bland Chromatin), 8 (Normal. nucleoli)  V. Relief for multiclass problem  Example 4: The vehicle dataset<jupyter_code>df1=pd.read_csv("http://academic.uprm.edu/eacuna/vehicle.csv")
df1.info()
#Convirtiendo en matriz la tabla de predictoras y la columna de clases
y=df1['Class']
X=df1.iloc[:,0:18]
y1=y.values
X1=X.values
#Normalizando las predictoras
X_norm=(X - X.min()) / (X.max() - X.min())
features, labels = X_norm.values, y1<jupyter_output><empty_output><jupyter_text>Using skrebate<jupyter_code>fs = ReliefF(n_neighbors=10)
fs.fit(features, labels)
for feature_name, feature_score in zip(X.columns,fs.feature_importances_):
    print (feature_name, feature_score)
idx = np.argsort(fs.feature_importances_)
print(X.columns[idx[::-1]])
yscore=fs.feature_importances_
oyscore=-np.sort(-yscore)
print(oyscore)
x=np.arange(0,18)
plt.plot(x,oyscore,'k-',lw=3)<jupyter_output>[0.063 0.056 0.056 0.055 0.055 0.05  0.049 0.048 0.04  0.032 0.032 0.029
 0.026 0.022 0.021 0.021 0.018 0.016]<jupyter_text>The best features are: 'ELONGATEDNESS', 'SCATTER_RATIO', 'SCALED_VARIANCE_MINOR', 'HOLLOWS_RATIO', 'PR.AXIS_RECTANGULARITY', 'DISTANCE_CIRCULARITY', 'MAX.LENGTH_RECTANGULARITY', 'SCALED_VARIANCE_MAJOR'  Example 5: Landsat Dataset<jupyter_code>names=['a1','a2','a3','a4','a5','a6','a7','a8','a9','a10','a11','a12','a13','a14','a15','a16','a17','a18','a19','a20','a21','a22','a23','a24','a25','a26','a27','a38','a29','a30','a31','a32','a33','a34','a35','a36','class']
url='http://academic.uprm.edu/eacuna/landsat.txt'
data = pd.read_csv(url,names=names, header=None,delim_whitespace=True)
data.head()
y=data['class']
X=data.iloc[:,0:36]
y1=y.to_numpy()
X1=X.to_numpy()
features, labels = X.values, data['class'].values
df=data.drop('class',axis=1)
#Normalizando las predictoras
df_norm=(df - df.min()) / (df.max() - df.min())
features, labels = df_norm.values, data['class'].values
df.shape
fs = ReliefF(n_neighbors=10)
fs.fit(features, labels)
for feature_name, feature_score in zip(data.columns,fs.feature_importances_):
    print (feature_name, feature_score)
idx = np.argsort(fs.feature_importances_)
print(df.columns[idx[::-1]])<jupyter_output>Index(['a13', 'a25', 'a21', 'a17', 'a33', 'a29', 'a9', 'a5', 'a1', 'a30',
       'a14', 'a22', 'a26', 'a18', 'a34', 'a10', 'a2', 'a6', 'a3', 'a16',
       'a27', 'a19', 'a38', 'a12', 'a15', 'a4', 'a24', 'a36', 'a20', 'a32',
       'a35', 'a31', 'a8', 'a23', 'a11', 'a7'],
      dtype='object')<jupyter_text>Comment:The best attributes are:a13,a25,a21,a17,a33,a29,a9,a5,a1,a30  Using Relief from Orange<jupyter_code>#Usando el Relief de Orange
import Orange
df = Orange.data.Table("https://academic.uprm.edu/eacuna/landsat.tab")
print('Feature scores  (scored individually):')
#ReliefF usando 10 vecinos mas cercanos y  una muestrd m=3000 para updating de los pesos
meas =Orange.preprocess.score.ReliefF(n_iterations=3000,k_nearest=10)
scores = meas(df)
for attr, score in zip(df.domain.attributes, scores):
    print('%.3f' % score, attr.name)

idx = np.argsort(scores)
print(idx[::-1])<jupyter_output>[12 16 24 20 32  4 28  0  8 13 17 29 25 21 33  1  9 15 26  5  2 18 27 14
 23 19 35 11  3 31 30 34 22  7 10  6]<jupyter_text>Comment:The best attributes are:a13,a25,a17,a21,a33,a29,a1,a5,a9  VI.  Applying Relief to a dataset with mixed type of attributes  VI-1 The Loan Dataset<jupyter_code>import Orange
df = Orange.data.Table("https://academic.uprm.edu/eacuna/loan.tab")
df
meas =Orange.preprocess.score.ReliefF(n_iterations=2000,k_nearest=10)
scores = meas(df)
for attr, score in zip(df.domain.attributes, scores):
    print('%.3f' % score, attr.name)<jupyter_output>0.077 Sexo
-0.012 Familia
0.008 CasPropia
0.016 AnosEmpleo
0.056 Sueldo
0.127 StatustMarital<jupyter_text>VI-2 The Heart-Cleveland Dataset<jupyter_code>#Usando el Relief de Orange
import Orange
df = Orange.data.Table("https://academic.uprm.edu/eacuna/heart_disease.tab")
print('Feature scores (scored individually):')
#ReliefF usando 10 vecinos mas cercanos y  una muestrd m=200 ara updating de los pesos
meas =Orange.preprocess.score.ReliefF(n_iterations=200,k_nearest=10)
scores = meas(df)
for attr, score in zip(df.domain.attributes, scores):
    print('%.3f' % score, attr.name)

idx = np.argsort(scores)
print(idx[::-1])<jupyter_output>[ 2 12  1 11  6 10  8  9  5  7  3  0  4]<jupyter_text>The best features are: chest pain, thal, major vessels colored, gender  VII. Feature selection using Minumum Redudancy and Maximum Relevancy (mRMR)  mRMR applied to landsat<jupyter_code>import pandas as pd
from mrmr import mrmr_classif

# Chhosing teh top ten best features 
mrmr = mrmr_classif(X,y, K = 36)
print(mrmr)<jupyter_output>['a18', 'a20', 'a17', 'a16', 'a22', 'a21', 'a24', 'a14', 'a13', 'a8', 'a30', 'a32', 'a5', 'a6', 'a38', 'a29', 'a34', 'a12', 'a33', 'a4', 'a9', 'a2', 'a36', 'a10', 'a25', 'a1', 'a26', 'a19', 'a15', 'a23', 'a27', 'a7', 'a31', 'a3', 'a11', 'a35']
{'fraction_non_alphanumeric': 0.14284861509073543, 'fraction_numerical': 0.06327602674307545, 'mean_word_length': 5.067729083665339, 'pattern_counts': {'":': 1, '<': 74, '<?xml version=': 0, '>': 74, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '39856', 'license': 'MIT', 'n_tokens_mistral': 7069, 'n_tokens_neox': 6354, 'n_words': 1541, 'path': 'notebooks/fselection12021.ipynb', 'repo_name': 'eacunafer/Data-Mining-Machine-Learning-subgraduado-', 'size': '16752'}