<jupyter_start><jupyter_text>Univariate feature selection Univariate feature selection removes features by calculating a statistical score for individual features only select features with high-ranking scores. Scores for each feature is calculated independently, make this method very efficient and can be performed in parallel. Univariate feature selection is useful when the number of features is very large, compared to the number samples.In univariate feature selection, scores are typically evaluated between input features and class labels/target variables. For classification problems, continous features can be scored using various statistical test, including Mann-Whitney U tests, t-tests, ANOVA models, area under the ROC curve (AUROC). ANOVA models can accomodate multi-class labels. For discrete features, chi-square test, mutual information or gini index are useful criteria.<jupyter_code>from sklearn.feature_selection import SelectKBest, f_classif
labels = split['fraud']
split_feature = split.drop(columns = 'fraud')
split_feature.head()
# 5 best feature
n=5

# Create and fit selector
selector = SelectKBest(f_classif, k=n)
selector.fit(split_feature, labels)
# Get columns to keep
cols = selector.get_support(indices=True)
# Create new dataframe with only desired columns, or overwrite existing
# features_df_new = split_feature.iloc[:, cols]
# features_df_new.sample(3)

need_column_keys = list(split_feature.iloc[:, cols].columns)

need_column_keys
# sampling data
trainset = split[['feature_22', 'feature_38', 'feature_158'
       , 'feature_162', 'feature_188', 'fraud']].sample(frac=0.5)
testset = split[['feature_22', 'feature_38', 'feature_158'
       , 'feature_162', 'feature_188', 'fraud']].sample(frac=0.5)

testset.shape, trainset.shape<jupyter_output><empty_output><jupyter_text>Train dataset information<jupyter_code>train_qtd = trainset.shape[0]
train_fraud_qtd = len(list(filter(lambda x: x == 1, trainset['fraud'])))
train_nonfraud_qtd = len(list(filter(lambda x: x == 0, trainset['fraud'])))
train_proportion = train_fraud_qtd / train_qtd

print('train_total:', train_qtd)
print('train_fraud:', train_fraud_qtd)
print('train_nonfraud', train_nonfraud_qtd)
print('train_fraud proportion', train_proportion)
trainset.head()<jupyter_output>train_total: 47290
train_fraud: 2013
train_nonfraud 45277
train_fraud proportion 0.04256713893000635<jupyter_text>Test dataset information<jupyter_code>test_qtd = testset.shape[0]
test_fraud_qtd = len(list(filter(lambda x: x == 1, testset['fraud'])))
test_nonfraud_qtd = len(list(filter(lambda x: x == 0, testset['fraud'])))
test_proportion = test_fraud_qtd / test_qtd

print('test_total:', test_qtd)
print('test_fraud:', test_fraud_qtd)
print('test_nonfraud', test_nonfraud_qtd)
print('test_fraud proportion', test_proportion)
testset.head()<jupyter_output>test_total: 47290
test_fraud: 2016
test_nonfraud 45274
test_fraud proportion 0.04263057728906745<jupyter_text>Learning<jupyter_code>trainlabel = trainset['fraud']
testlabel = testset['fraud']
clf = tree.DecisionTreeClassifier()
clf = clf.fit(trainset, trainlabel)<jupyter_output><empty_output><jupyter_text>Evaluate<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(trainset, trainlabel
                                                   # , test_size=0.20
                                                   )
y_pred = clf.predict(X_test)
# import classification report and confusion matrix
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix

classification_report = classification_report(y_test, y_pred)
confusion_matrix = confusion_matrix(y_test, y_pred)

print("CLASSIFICATION REPORT")
print(classification_report)
print("CONFUSION MATRIX") 
print(confusion_matrix)<jupyter_output>CLASSIFICATION REPORT
              precision    recall  f1-score   support

           0       1.00      1.00      1.00     11340
           1       1.00      1.00      1.00       483

    accuracy                           1.00     11823
   macro avg       1.00      1.00      1.00     11823
weighted avg       1.00      1.00      1.00     11823

CONFUSION MATRIX
[[11340     0]
 [    0   483]]
{'fraction_non_alphanumeric': 0.09045702107506512, 'fraction_numerical': 0.04286052569263557, 'mean_word_length': 4.739130434782608, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '448343', 'license': 'MIT', 'n_tokens_mistral': 1481, 'n_tokens_neox': 1311, 'n_words': 335, 'path': 'sol/Classified_frod.ipynb', 'repo_name': 'nshovkun/Analytics', 'size': '4223'}