<jupyter_start><jupyter_text>Apply Decision Tree  This notebook contains application of 'Decision Tree' in section-III(C) from the paper<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeClassifier

df = pd.read_csv('https://raw.githubusercontent.com/ferdouszislam/Android-Malware-Detection-ML/main/datasets/Feature-Selected_Dataset/Main_Dataset-Weka_Feature_Selected.csv?token=AKGHTOZCFCA62MER45KW3HLAUEPP4')
df.info()
X = df.drop('class', axis = 1)
y = df['class']<jupyter_output><empty_output><jupyter_text>Trying to find out the hyperparameter for which the decision tree gives out highest combination of accuracy, f1 score and AUC<jupyter_code>from sklearn.model_selection import StratifiedKFold,cross_val_score

cv = StratifiedKFold(n_splits=10, random_state=42, shuffle= True)

max_depth = [x for x in range(1,30)]

accuracies = []
f1s = []
aucs = []
for depth in max_depth:
    model = DecisionTreeClassifier(criterion='gini',max_depth = depth)
    accuracie_segments = cross_val_score(model, X, y, scoring='accuracy',cv=cv, n_jobs=1)
    f1_segments = cross_val_score(model, X, y, scoring='f1',cv=cv, n_jobs=1)
    auc_segments = cross_val_score(model, X, y, scoring='roc_auc',cv=cv, n_jobs=1)
    accuracies.append(np.mean(accuracie_segments))
    f1s.append(np.mean(f1_segments))
    aucs.append(np.mean(auc_segments))
plt.figure(figsize =(15,9))
plt.title('Accuracy,F1 Score, and Area Under ROC-curve VS max depth')
plt.plot(max_depth, accuracies, 'ro-',  max_depth, f1s ,'bv-', max_depth, aucs,'yo-')
plt.axvline(x=17, color='k', linestyle='--')
plt.legend(['Accuracies','F1 Scores','AUC', 'selected value (max_depth=17)'])
plt.xlabel('Max depths', fontsize=18)
plt.ylabel('Accuracy,F1 Score,Area Under ROC-curve', fontsize=18)
plt.show()<jupyter_output><empty_output><jupyter_text>Hyperparameter(max_depth, criterion) selection using GridSearchCV<jupyter_code>'''
courtesy-
    <https://towardsdatascience.com/building-a-k-nearest-neighbors-k-nn-model-with-scikit-learn-51209555453a>
    <https://towardsdatascience.com/gridsearchcv-for-beginners-db48a90114ee>
'''  
from sklearn.model_selection import GridSearchCV

# create decision tree model
decisionTree_model_gscv = DecisionTreeClassifier(random_state=42)

# create a dictionary of all parameter values we want to exhaustively search
param_grid = {'max_depth': np.arange(1, 30), 'criterion': ['gini', 'entropy']}
# use gridsearch to check all values in param_grid
decisionTree_gscv = GridSearchCV(decisionTree_model_gscv, param_grid, scoring=['accuracy', 'f1', 'roc_auc'], refit='accuracy', cv=cv)
# fit model to data
decisionTree_gscv.fit(X, y)
#check top performing parameter values
decisionTree_gscv.best_params_
decisionTree_gscv.best_score_<jupyter_output><empty_output><jupyter_text>So we can confirm that max_depth=17 and criterion='gini' gives best outcome.  Evaluate the model on experimenting/training set<jupyter_code>y_predict = decisionTree_gscv.predict(X)
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

print('Evaluation on experimenting/test dataset')
print('Accuracy:', round(accuracy_score(y, y_predict), 3))
print('Precision:', round(precision_score(y, y_predict), 3))
print('Recall:', round(recall_score(y, y_predict), 3))
print('F1-score:', round(f1_score(y, y_predict), 3))<jupyter_output>Evaluation on experimenting/test dataset
Accuracy: 0.965
Precision: 0.983
Recall: 0.92
F1-score: 0.951<jupyter_text>ROC curve<jupyter_code>from sklearn.metrics import roc_curve, roc_auc_score

# calculate the fpr and tpr for all thresholds of the classification
y_probs = decisionTree_gscv.predict_proba(X)
y_probs = y_probs[:,1]
fpr, tpr, threshold = roc_curve(y, y_probs)
roc_auc = roc_auc_score(y, y_probs)

plt.title('Decision Tree Train-set ROC-Curve')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.3f' % roc_auc)
plt.legend(loc = 'lower right')
plt.plot([0, 1], [0, 1],'r--')
plt.ylabel('True Positive Rate', fontsize=18)
plt.xlabel('False Positive Rate', fontsize=18)
plt.show()<jupyter_output><empty_output><jupyter_text>Train set evaluation  accuracy = 0.965, precision = 0.983, recall = 0.92, f1 = 0.951, auc = 0.994  Evaluate the model on holdout/test set<jupyter_code>holdout_df = pd.read_csv('https://raw.githubusercontent.com/ferdouszislam/Android-Malware-Detection-ML/main/datasets/Feature-Selected_Dataset/Holdout_Dataset-Weka_Feature_Selected.csv?token=<KEY>')
holdout_df.shape
holdout_df.info()
X_holdout = holdout_df.drop('class', axis = 1)
y_holdout = holdout_df['class']
y_holdout_predict = decisionTree_gscv.predict(X_holdout)
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

print('Evaluation on holdout dataset')
print('Accuracy:', round(accuracy_score(y_holdout, y_holdout_predict), 3))
print('Precision:', round(precision_score(y_holdout, y_holdout_predict), 3))
print('Recall:', round(recall_score(y_holdout, y_holdout_predict), 3))
print('F1-score:', round(f1_score(y_holdout, y_holdout_predict), 3))<jupyter_output>Evaluation on holdout dataset
Accuracy: 0.961
Precision: 0.961
Recall: 0.93
F1-score: 0.945<jupyter_text>ROC Curve<jupyter_code>from sklearn.metrics import roc_curve, roc_auc_score

# calculate the fpr and tpr for all thresholds of the classification
y_holdout_probs = decisionTree_gscv.predict_proba(X_holdout)
y_holdout_probs = y_holdout_probs[:,1]
fpr, tpr, threshold = roc_curve(y_holdout, y_holdout_probs)
roc_auc = roc_auc_score(y_holdout, y_holdout_probs)

plt.title('Decision Tree test-set ROC-Curve')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.3f' % roc_auc)
plt.legend(loc = 'lower right')
plt.plot([0, 1], [0, 1],'r--')
plt.ylabel('True Positive Rate', fontsize=18)
plt.xlabel('False Positive Rate', fontsize=18)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12868908716540836, 'fraction_numerical': 0.026252573781743307, 'mean_word_length': 7.423410404624278, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '407793', 'license': 'MIT', 'n_tokens_mistral': 2325, 'n_tokens_neox': 2154, 'n_words': 437, 'path': 'jupyter_notebooks/3A. Apply Decision Tree.ipynb', 'repo_name': 'AshfaqJamil/Android-Malware-Detection-ML', 'size': '5852'}