<jupyter_start><jupyter_text>Working with 10s time windows<jupyter_code>data = pd.read_csv('../Parser/human-bot-10000-data.csv', index_col=[0])
len(data[data['label'] == 1]), len(data[data['label'] == 0])<jupyter_output><empty_output><jupyter_text>Very balanced data<jupyter_code>X = data.drop('label', axis=1)
y = data['label']
X.describe()
classifier(X, y, methods=methods)<jupyter_output>Logistic regression
Confusion matrix:
[[42 12]
 [10 51]]
Accuracy: 0.808695652173913
AUC score: 0.8069216757741348

	Naive Bayes
Confusion matrix:
[[34 20]
 [ 5 56]]
Accuracy: 0.782608695652174
AUC score: 0.7738312082574379

	K-NN
Confusion matrix:
[[47  7]
 [ 8 53]]
Accuracy: 0.8695652173913043
AUC score: 0.8696114146933819

	Decision Tree
Confusion matrix:
[[45  9]
 [ 6 55]]
Accuracy: 0.8695652173913043
AUC score: 0.8674863387978142

	Random Forest
Confusion matrix:
[[50  4]
 [ 7 54]]
Accuracy: 0.9043478260869565
AUC score: 0.9055859137826352

	Extra Trees
Confusion matrix:
[[50  4]
 [ 8 53]]
Accuracy: 0.8956521739130435
AUC score: 0.8973891924711596

	Vote
Confusion matrix:
[[44 10]
 [ 6 55]]
Accuracy: 0.8608695652173913
AUC score: 0.8582270795385549<jupyter_text>As we can see, the accuracy is not high enough. Lets try 20s time windows<jupyter_code>data = pd.read_csv('../Parser/human-bot-20000-data.csv', index_col=[0])
len(data[data['label'] == 1]), len(data[data['label'] == 0])<jupyter_output><empty_output><jupyter_text>Still very balanced<jupyter_code>X = data.drop('label', axis=1)
y = data['label']
X.describe()
classifier(X, y, methods=methods)<jupyter_output>Logistic regression
Confusion matrix:
[[25  6]
 [ 3 24]]
Accuracy: 0.8448275862068966
AUC score: 0.8476702508960573

	Naive Bayes
Confusion matrix:
[[18 13]
 [ 1 26]]
Accuracy: 0.7586206896551724
AUC score: 0.7718040621266427

	K-NN
Confusion matrix:
[[25  6]
 [ 2 25]]
Accuracy: 0.8620689655172413
AUC score: 0.8661887694145758

	Decision Tree
Confusion matrix:
[[30  1]
 [ 3 24]]
Accuracy: 0.9310344827586207
AUC score: 0.9283154121863799

	Random Forest
Confusion matrix:
[[29  2]
 [ 1 26]]
Accuracy: 0.9482758620689655
AUC score: 0.9492234169653525

	Extra Trees
Confusion matrix:
[[27  4]
 [ 1 26]]
Accuracy: 0.9137931034482759
AUC score: 0.9169653524492233

	Vote
Confusion matrix:
[[28  3]
 [ 1 26]]
Accuracy: 0.9310344827586207
AUC score: 0.9330943847072878<jupyter_text>Accuracy raised pretty good  Boosting We will work with 20s time windows as they have shown better accuracy<jupyter_code>data = pd.read_csv('../Parser/human-bot-20000-data.csv', index_col=[0])
X = data.drop('label', axis=1)
y = data['label']<jupyter_output><empty_output><jupyter_text>Bagging<jupyter_code>boosted_classifier(X, y, BaggingClassifier, methods)<jupyter_output>BaggingClassifier + Logistic regression
Confusion matrix:
[[20  5]
 [ 5 28]]
Accuracy: 0.8275862068965517
AUC score: 0.8242424242424243

	BaggingClassifier + Naive Bayes
Confusion matrix:
[[17  8]
 [ 1 32]]
Accuracy: 0.8448275862068966
AUC score: 0.8248484848484848

	BaggingClassifier + K-NN
Confusion matrix:
[[21  4]
 [ 3 30]]
Accuracy: 0.8793103448275862
AUC score: 0.8745454545454546

	BaggingClassifier + Decision Tree
Confusion matrix:
[[23  2]
 [ 3 30]]
Accuracy: 0.9137931034482759
AUC score: 0.9145454545454547

	BaggingClassifier + Random Forest
Confusion matrix:
[[24  1]
 [ 2 31]]
Accuracy: 0.9482758620689655
AUC score: 0.9496969696969697

	BaggingClassifier + Extra Trees
Confusion matrix:
[[24  1]
 [ 3 30]]
Accuracy: 0.9310344827586207
AUC score: 0.9345454545454546

	BaggingClassifier + Vote
Confusion matrix:
[[24  1]
 [ 1 32]]
Accuracy: 0.9655172413793104
AUC score: 0.9648484848484848<jupyter_text>Ada boost<jupyter_code>boosted_classifier(X, y, AdaBoostClassifier, methods)<jupyter_output>AdaBoostClassifier + Logistic regression
Confusion matrix:
[[20  3]
 [ 5 30]]
Accuracy: 0.8620689655172413
AUC score: 0.8633540372670807

	AdaBoostClassifier + Naive Bayes
Confusion matrix:
[[14  9]
 [ 0 35]]
Accuracy: 0.8448275862068966
AUC score: 0.8043478260869565

	AdaBoostClassifier + Decision Tree
Confusion matrix:
[[21  2]
 [ 2 33]]
Accuracy: 0.9310344827586207
AUC score: 0.9279503105590063

	AdaBoostClassifier + Random Forest
Confusion matrix:
[[21  2]
 [ 4 31]]
Accuracy: 0.896551724137931
AUC score: 0.8993788819875776

	AdaBoostClassifier + Extra Trees
Confusion matrix:
[[21  2]
 [ 4 31]]
Accuracy: 0.896551724137931
AUC score: 0.8993788819875776

	AdaBoostClassifier + Vote
Confusion matrix:
[[20  3]
 [ 2 33]]
Accuracy: 0.9137931034482759
AUC score: 0.9062111801242235<jupyter_text>Gradient Boosting<jupyter_code>classifier(X, y, {'GradientBoosting': GradientBoostingClassifier()})<jupyter_output>GradientBoosting
Confusion matrix:
[[26  3]
 [ 2 27]]
Accuracy: 0.9137931034482759
AUC score: 0.9137931034482758<jupyter_text>Small research / feature selection<jupyter_code>data = pd.read_csv('../Parser/human-bot-20000-data.csv', index_col=[0])
X = data.drop('label', axis=1)
y = data['label']
X_train, X_test, y_train, y_test = train_test_split(X, y)
X.info()
import eli5
from eli5.sklearn import PermutationImportance<jupyter_output><empty_output><jupyter_text>eli5 provides a way to compute feature importances for any black-box estimator by measuring how score decreases when a feature is not available; the method is also known as “permutation importance” or “Mean Decrease Accuracy (MDA)”.<jupyter_code>RF_clf = RandomForestClassifier().fit(X_train, y_train)
DT_clf = DecisionTreeClassifier().fit(X_train, y_train)
V_clf = VotingClassifier(estimators=[('lr', LogisticRegressionCV(cv=5, max_iter=1000)),
                                                  ('nb', GaussianNB()),
                                                  ('dt', DecisionTreeClassifier()),
                                                  ('et', ExtraTreesClassifier(n_estimators=100, bootstrap=True)),
                                                  ('rf', RandomForestClassifier(n_estimators=150))],
                                      voting='soft')<jupyter_output><empty_output><jupyter_text>As RandomForest and DecisionTree showed best accuracy before, we'll use them to estimate feature importances<jupyter_code>perm_rf = PermutationImportance(RF_clf).fit(X_test, y_test)
perm_dt = PermutationImportance(DT_clf).fit(X_test, y_test)
eli5.show_weights(perm_rf)
eli5.show_weights(perm_dt)<jupyter_output><empty_output><jupyter_text>Making a conclusion, that we should use features ['gpm', 'tbc', 'epm', 'aht'] ('dev'?)<jupyter_code>X_red = X[['gpm', 'tbc', 'epm', 'aht']]
classifier(X_red, y, methods)
boosted_classifier(X_red, y, BaggingClassifier, methods)
boosted_classifier(X_red, y, AdaBoostClassifier, methods)
classifier(X_red, y, {'GradientBoosting': GradientBoostingClassifier()})<jupyter_output>GradientBoosting
Confusion matrix:
[[26  5]
 [ 2 25]]
Accuracy: 0.8793103448275862
AUC score: 0.8823178016726405<jupyter_text>Let's try add 'dev'<jupyter_code>X_red_wdev = X[['tbc', 'aht', 'tln']]
classifier(X_red_wdev, y, methods)
boosted_classifier(X_red, y, BaggingClassifier, methods)
boosted_classifier(X_red, y, AdaBoostClassifier, methods)<jupyter_output>AdaBoostClassifier + Logistic regression
Confusion matrix:
[[ 8 27]
 [ 1 22]]
Accuracy: 0.5172413793103449
AUC score: 0.5925465838509316

	AdaBoostClassifier + Naive Bayes
Confusion matrix:
[[25 10]
 [ 1 22]]
Accuracy: 0.8103448275862069
AUC score: 0.8354037267080746

	AdaBoostClassifier + Decision Tree
Confusion matrix:
[[33  2]
 [ 2 21]]
Accuracy: 0.9310344827586207
AUC score: 0.9279503105590062

	AdaBoostClassifier + Random Forest
Confusion matrix:
[[34  1]
 [ 4 19]]
Accuracy: 0.9137931034482759
AUC score: 0.8987577639751554

	AdaBoostClassifier + Extra Trees
Confusion matrix:
[[35  0]
 [ 1 22]]
Accuracy: 0.9827586206896551
AUC score: 0.9782608695652174

	AdaBoostClassifier + Vote
Confusion matrix:
[[34  1]
 [ 2 21]]
Accuracy: 0.9482758620689655
AUC score: 0.9422360248447206
{'fraction_non_alphanumeric': 0.11491194729507158, 'fraction_numerical': 0.18446724946154822, 'mean_word_length': 5.295055821371611, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '499618', 'license': 'MIT', 'n_tokens_mistral': 4035, 'n_tokens_neox': 3011, 'n_words': 689, 'path': 'Classifier/classifier.ipynb', 'repo_name': 'roma-vinn/GameBot-detection', 'size': '7893'}