<jupyter_start><jupyter_text>XGBoost with CICIDS This notebook provides comparsion stats.  The CICIDS2017 version used is [from the University of New Brunswick, Canada](http://www.unb.ca/cic/datasets/ids-2017.html).<jupyter_code>import json
import os
import numpy as np
import pandas as pd
pd.set_option('display.max_columns', None)<jupyter_output><empty_output><jupyter_text>Data loading and prep<jupyter_code>def load_df(filename):
    filepath = os.path.join('CICIDS2017', filename+'.pkl')
    return pd.read_pickle(filepath)
cic_train_data = load_df('cic_train_data')
cic_test_data = load_df('cic_test_data')
cic_train_labels = load_df('cic_train_labels')
cic_test_labels = load_df('cic_test_labels')<jupyter_output><empty_output><jupyter_text>We only need 6 features, so we filter accordingly.   The mapping is as follows:  | NSL-KDD field | CICIDS2017 field ||---------------|---------------------|| duration | flow_duration || protocol_type | protocol || src_bytes | total_fwd_packets || dst_bytes | total_backward_packets || count | flow_packets_per_s || srv_count | destination_port |<jupyter_code>fields = ['flow_duration', 'protocol', 'total_fwd_packets', 'total_backward_packets','flow_packets_per_s','destination_port']

cic_train_data = cic_train_data.filter(fields, axis=1) 
cic_test_data = cic_test_data.filter(fields, axis=1)<jupyter_output><empty_output><jupyter_text>Label Translation As we are doing binary classification, we only need to know if the entry is normal/benign (*0*) or malicious (*1*).  For this, we load the Tokenizer word index to get the benign label representation and then define a function to transfor our labels.<jupyter_code>with open(os.path.join('CICIDS2017','cic_label_wordindex.json')) as json_in:
    data = json.load(json_in)
    print(data)
    normal_index = data['benign']
def f(x):
    return 0 if x == normal_index else 1
f = np.vectorize(f)
cic_train_labels.head()
cic_train_labels = f(cic_train_labels['label_encoded'].values)
cic_test_labels = f(cic_test_labels['label_encoded'].values)
cic_train_labels[:5]<jupyter_output><empty_output><jupyter_text>Building and training the model<jupyter_code>print("Training Set Size:",len(cic_train_data))
print("Training Label Size:",len(cic_train_labels))
print("Test Set Size:",len(cic_test_data))
print("Test Label Size:",len(cic_test_labels))
from xgboost import XGBClassifier

xgb = XGBClassifier(
    n_estimators=100,
    n_jobs=-1,
    random_state=0
)

xgb.fit(cic_train_data, cic_train_labels)
from sklearn.metrics import accuracy_score

cic_test_predictions = xgb.predict(cic_test_data)
predicted = [round(value) for value in cic_test_predictions]

accuracy = accuracy_score(cic_test_labels, predicted)
print("Mean accuarcy score: {0:1.3f}".format(accuracy))<jupyter_output>Mean accuarcy score: 0.795
{'fraction_non_alphanumeric': 0.12081254454739843, 'fraction_numerical': 0.014255167498218105, 'mean_word_length': 6.754143646408839, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '156206', 'license': 'Apache-2.0', 'n_tokens_mistral': 1011, 'n_tokens_neox': 973, 'n_words': 207, 'path': 'CICIDS-2017-XGBoost.ipynb', 'repo_name': 'twseptian/Master_IPython', 'size': '2806'}