<jupyter_start><jupyter_text>InternetFirewallSupervisedLearning Author: <NAME>Predicting the Internet Firewall action based on log info. Supervised Learning Assignment for Master DSLS. Data availability:- [Direct link](https://archive.ics.uci.edu/ml/machine-learning-databases/00542/log2.csv)- [Archive link with some info](https://archive.ics.uci.edu/ml/datasets/Internet+Firewall+Data)- [<NAME> and <NAME>, "Classification of firewall log files with multiclass support vector machine," 2018 6th International Symposium on Digital Forensic and Security (ISDFS), 2018, pp. 1-4, doi: 10.1109/ISDFS.2018.8355382.](https://doi.org/10.1109/ISDFS.2018.8355382) Table 1. Features and Description. Adapted from Ertam & Kaya, 2018.|        Feature       |                  Description                 ||:--------------------:|:--------------------------------------------:|| Source Port          | Client Source Port                           || Destination Port     | Client Destination Port                      || NAT Source Port      | Network Address Translation Source Port      || NAT Destination Port | Network Address Translation Destination Port || Elapsed Time (sec)   | Elapsed Time for flow                        || Bytes                | Total Bytes                                  || Bytes Sent           | Bytes Sent                                   || Bytes Received       | Bytes Received                               || Packets              | Total Packets                                || pkts_sent            | Packets Sent                                 || pkts_received        | Packets Received                             || Action               | Class (allow, deny, drop, reset-both)        | Table 2. Classes to predict (Actions). Adapted from Ertam & Kaya, 2018.|   Action   |                                                               Description                                                              ||:----------:|:--------------------------------------------------------------------------------------------------------------------------------------:|| Allow      | Allows the internet traffic.                                                                                                           || Deny       | Blocks traffic and enforces the default Deny Action defined for the application that is being denied.                                  || Drop       | Silently drops the traffic; for an application, it overrides the default deny action. A TCP reset is not sent to the host/application. || Reset-Both | Sends a TCP reset to both the client-side and server-sidedevices.                                                                      |<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
df = pd.read_csv("log2.csv")
df
df.info()
allow, deny, drop, reset_both = df['Action'].value_counts()
print('Number of allowed logs: ', allow)  
print('Number of denied logs: ', deny)
print('Number of dropped logs: ', drop)  
print('Number of reset-both logs: ', reset_both) 

print('\n')
print('% of allowed logs', round(allow / len(df) * 100, 1), '%')
print('% of denied logs', round(deny / len(df) * 100, 1), '%')
print('% of dropped logs', round(drop / len(df) * 100, 1), '%')
print('% of reset-both logs', round(reset_both / len(df) * 100, 1), '%')
df['Action'].value_counts().plot(kind='barh')<jupyter_output><empty_output><jupyter_text>Pretty unbalanced classes to predict unfortunately. Might run in some problems with the reset-both class later on. SMOTE could be used, but in this case I won't attempt it (out of the scope of this assignment).<jupyter_code># missing data
df.isnull().sum() 
# no missing data, no imputation needed
description = df.groupby(['Action']).describe()
c = df.corr().abs()
sns.heatmap(c, cmap=sns.color_palette("Blues", as_cmap=True))<jupyter_output><empty_output><jupyter_text>'Bytes Sent', 'Bytes Received', 'pkts_sent' and 'pkts_received' can be discarded as Bytes and Packets are the total of the two pairs respectively.I am also going to discard packets for bytes as it is highly correlated. I'll keep Bytes since it is more detailed than packets (1 packet consists of multiple bytes) All of the port variables should not be seen as continous, but probably are interesting to see the range<jupyter_code>sns.displot(df, x="Source Port", hue="Action")
plt.show()
description['Source Port']<jupyter_output><empty_output><jupyter_text>All Actions of drop seem to happen in high source ports (minimum 49156). Reset-both Source Port minimum is 1024<jupyter_code>sns.displot(df, x="Destination Port", hue="Action")
plt.show()
description['Destination Port']<jupyter_output><empty_output><jupyter_text>Most actions seem to have a very low destination port. All drop actions are done on Destination Port 445<jupyter_code>sns.displot(df, x="NAT Source Port", hue="Action")
plt.show()
description['NAT Source Port']<jupyter_output><empty_output><jupyter_text>Allowed actions seme to be uniformly distributed over NAT Source Ports. All dropped NAT Source Ports are equal to 0. Most deny and reset-both actions have NAT Source Ports of 0.<jupyter_code>sns.displot(df, x="NAT Destination Port", hue="Action")
plt.show()
description['NAT Destination Port']<jupyter_output><empty_output><jupyter_text>Allowed actions seem to be distributed over all NAT Destination Ports, but mostly under 443. All dropped NAT Destination Ports are equal to 0. Most deny and reset-both actions have NAT Destination Ports of 0.<jupyter_code>cols = ['Source Port', 
        'Destination Port', 
        'NAT Source Port', 
        'NAT Destination Port', 
        'Bytes', 
        'Elapsed Time (sec)']
df_features = df[cols].rename(columns={'Source Port':'source_port',
                                       'Destination Port':'destination_port', 
                                       'NAT Source Port':'nat_source_port', 
                                       'NAT Destination Port':'nat_destination_port',
                                       'Bytes':'bytes',
                                       'Elapsed Time (sec)':'elapsed_time'})
c = df_features.corr().abs()
sns.heatmap(c, cmap=sns.color_palette("Blues", as_cmap=True))
y = np.array(df['Action'].replace({'allow':0,'deny':1,'drop':2, 'reset-both':3}))
X = np.array(df_features)
print(y.shape)
print(X.shape)<jupyter_output>(65532,)
(65532, 6)<jupyter_text>Let's normalize because the variance differs greatly per feature.<jupyter_code>from sklearn.preprocessing import StandardScaler

def normalize(X):
    scalar = StandardScaler()
    scalar = scalar.fit(X)
    X = scalar.transform(X)
    return X

X = normalize(X)
from sklearn.model_selection import train_test_split, ShuffleSplit

#split
test_size = 0.4
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = test_size)

#cross validation
cv = ShuffleSplit(n_splits=100, test_size=test_size, random_state=42)
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix

def evaluate(y_test, y_pred, X_test):
    print(confusion_matrix(y_test, y_pred))
    print(classification_report(y_test, y_pred))
# Based on code from https://github.com/fenna/student_BFVM19DATASC3
from sklearn.metrics import mean_squared_error

def plot_learning_curves(model, X_train, y_train, X_val, y_val, training_sizes=range(999, len(X_train), 1000)):
    MSE_train, MSE_val = calculate_MSE_over_training_sizes(model, X_train, y_train, 
                                                           X_val, y_val, training_sizes)

    plt.plot(training_sizes, MSE_train,
             "r-o", linewidth=2, label="trainingsdata")
    plt.plot(training_sizes, MSE_val, 
             "b-*", linewidth=3, label="validationdata")
    plt.legend(loc="best", fontsize=14)   
    plt.xlabel("Training set size", fontsize=14) 
    plt.ylabel("RMSE", fontsize=14) 

    
def calculate_MSE_over_training_sizes(model, X_train, y_train, X_val, y_val, training_sizes):
    train_errors, val_errors = [], []
    for m in training_sizes:
        model.fit(X_train[:m], y_train[:m])
        y_train_predict = model.predict(X_train[:m])
        y_val_predict = model.predict(X_val)
        train_errors.append(mean_squared_error(y_train_predict, y_train[:m]))
        val_errors.append(mean_squared_error(y_val_predict, y_val))
        
    return np.sqrt(train_errors), np.sqrt(val_errors)<jupyter_output><empty_output><jupyter_text>Most basic model; A dummy classifier<jupyter_code>from sklearn.dummy import DummyClassifier
dm = DummyClassifier(strategy='stratified') 
# stratified generates predictions by respecting the training set’s class distribution.
dm.fit(X_train, y_train)
y_pred = dm.predict(X_test)
evaluate(y_test, y_pred, X_test)
plot_learning_curves(dm, X_train, y_train, X_test, y_test, range(1999, len(X_train), 2000))<jupyter_output><empty_output><jupyter_text>Based on this learning curve you can not say if the model is underfitted or overfitted as it actually doesn't learn over iterations since it is a dummy classifier. Let's move on from this dummy model and attempt a basic classifier; Logistic regression.  Logistic Regression<jupyter_code># train
from sklearn.linear_model import LogisticRegression

lg = LogisticRegression(max_iter=1000, n_jobs=-1) # n_jobs=-1 to use all processors because of large dataset
lg.fit(X_train, y_train)
# evaluation
y_pred = lg.predict(X_test)
evaluate(y_test, y_pred, X_test)<jupyter_output>[[14888    44    67     0]
 [    6  5826   234     0]
 [    0     0  5120     0]
 [    6    22     0     0]]
              precision    recall  f1-score   support

           0       1.00      0.99      1.00     14999
           1       0.99      0.96      0.97      6066
           2       0.94      1.00      0.97      5120
           3       0.00      0.00      0.00        28

    accuracy                           0.99     26213
   macro avg       0.73      0.74      0.74     26213
weighted avg       0.99      0.99      0.99     26213<jupyter_text>This warning basically means that there are not enough instances of the fourth class to have in both the training and test set, so it cannot evaluate it's prediction.<jupyter_code>plot_learning_curves(lg, X_train, y_train, X_test, y_test, range(1999, len(X_train), 2000))<jupyter_output><empty_output><jupyter_text>Based on this learning curve I would say this model is maybe a tiny bit overfitted but overall fitted well, as the validation data and training data error are close together. As the fourth class could not be evaluated, let's make a decision tree which will at least attempt it albeit the low support.  Decision Tree<jupyter_code>from sklearn.tree import DecisionTreeClassifier
dt = DecisionTreeClassifier()
dt.fit(X_train, y_train)
y_pred = dt.predict(X_test)
evaluate(y_test, y_pred, X_test)
plot_learning_curves(dt, X_train, y_train, X_test, y_test)<jupyter_output><empty_output><jupyter_text>Based on this learning curve I would say this model is underfitted as there is still a large gap between training error and validation error. Although mait looks like more data would not fix this problem. Let's now get to the real deal, comparing SVM methods to Ertam & Kaya, 2018.  SVM   Activation function: linear<jupyter_code>from sklearn.svm import SVC
svm_lin = SVC(kernel='linear')
svm_lin.fit(X_train, y_train)
y_pred = svm_lin.predict(X_test)
evaluate(y_test, y_pred, X_test)<jupyter_output>[[14960    34     5     0]
 [    6  5842   218     0]
 [    0     0  5120     0]
 [    6    22     0     0]]
              precision    recall  f1-score   support

           0       1.00      1.00      1.00     14999
           1       0.99      0.96      0.98      6066
           2       0.96      1.00      0.98      5120
           3       0.00      0.00      0.00        28

    accuracy                           0.99     26213
   macro avg       0.74      0.74      0.74     26213
weighted avg       0.99      0.99      0.99     26213<jupyter_text>This warning basically means that there are not enough instances of the fourth class to have in both the training and test set, so it cannot evaluate it's prediction.<jupyter_code>plot_learning_curves(svm_lin, X_train, y_train, X_test, y_test, range(4999, len(X_train), 5000))<jupyter_output><empty_output><jupyter_text>Based on this learning curve I would say this model is roughly fitted good. No large gap in error between training and validation data.  Activation function: poly<jupyter_code>from sklearn.svm import SVC
svm_poly = SVC(kernel='poly')
svm_poly.fit(X_train, y_train)
y_pred = svm_poly.predict(X_test)
evaluate(y_test, y_pred, X_test)<jupyter_output>[[14673    65   260     1]
 [    6  5794   266     0]
 [    0     0  5120     0]
 [    6    22     0     0]]
              precision    recall  f1-score   support

           0       1.00      0.98      0.99     14999
           1       0.99      0.96      0.97      6066
           2       0.91      1.00      0.95      5120
           3       0.00      0.00      0.00        28

    accuracy                           0.98     26213
   macro avg       0.72      0.73      0.73     26213
weighted avg       0.98      0.98      0.98     26213<jupyter_text>This warning basically means that there are not enough instances of the fourth class to have in both the training and test set, so it cannot evaluate it's prediction.<jupyter_code>plot_learning_curves(svm_poly, X_train, y_train, X_test, y_test, range(4999, len(X_train), 5000))<jupyter_output><empty_output><jupyter_text>Based on this learning curve I would say this model is overfitted as the error increases again after 25000 instances.  Activation function: RBF<jupyter_code>from sklearn.svm import SVC
svm_rbf = SVC(kernel='rbf')
svm_rbf.fit(X_train, y_train)
y_pred = svm_rbf.predict(X_test)
evaluate(y_test, y_pred, X_test)<jupyter_output>[[14941    53     5     0]
 [    6  5849   211     0]
 [    0     0  5120     0]
 [    6    22     0     0]]
              precision    recall  f1-score   support

           0       1.00      1.00      1.00     14999
           1       0.99      0.96      0.98      6066
           2       0.96      1.00      0.98      5120
           3       0.00      0.00      0.00        28

    accuracy                           0.99     26213
   macro avg       0.74      0.74      0.74     26213
weighted avg       0.99      0.99      0.99     26213<jupyter_text>This warning basically means that there are not enough instances of the fourth class to have in both the training and test set, so it cannot evaluate it's prediction.<jupyter_code>plot_learning_curves(svm_rbf, X_train, y_train, X_test, y_test, range(4999, len(X_train), 5000))<jupyter_output><empty_output><jupyter_text>Based on this learning curve I would say this model is roughly fitted good. No large gap in error between training and validation data.  Activation function: sigmoid<jupyter_code>from sklearn.svm import SVC
svm_sig = SVC(kernel='sigmoid')
svm_sig.fit(X_train, y_train)
y_pred = svm_sig.predict(X_test)
evaluate(y_test, y_pred, X_test)<jupyter_output>[[13972   321   706     0]
 [  519  5016   531     0]
 [  837     0  4283     0]
 [    8    20     0     0]]
              precision    recall  f1-score   support

           0       0.91      0.93      0.92     14999
           1       0.94      0.83      0.88      6066
           2       0.78      0.84      0.81      5120
           3       0.00      0.00      0.00        28

    accuracy                           0.89     26213
   macro avg       0.66      0.65      0.65     26213
weighted avg       0.89      0.89      0.89     26213<jupyter_text>This warning basically means that there are not enough instances of the fourth class to have in both the training and test set, so it cannot evaluate it's prediction.<jupyter_code>plot_learning_curves(svm_sig, X_train, y_train, X_test, y_test, range(4999, len(X_train), 5000))<jupyter_output><empty_output><jupyter_text>Based on this learning curve I would say this model is overfitted (kind of a weird learning curve actually), stopping at 30000 instances would be advised.  Ensemble learning (Voting Classifier)<jupyter_code>from sklearn.ensemble import VotingClassifier
estimators = [('svm_lin', svm_lin), ('svm_poly', svm_poly), ('svm_rbf', svm_rbf)]
vclf = VotingClassifier(estimators=estimators, n_jobs=-1)  # n_jobs=-1 to use all processors because of large dataset
vclf.fit(X_train, y_train)
y_pred = vclf.predict(X_test)
evaluate(y_test, y_pred, X_test)<jupyter_output>[[14950    44     5     0]
 [    6  5836   224     0]
 [    0     0  5120     0]
 [    6    22     0     0]]
              precision    recall  f1-score   support

           0       1.00      1.00      1.00     14999
           1       0.99      0.96      0.98      6066
           2       0.96      1.00      0.98      5120
           3       0.00      0.00      0.00        28

    accuracy                           0.99     26213
   macro avg       0.74      0.74      0.74     26213
weighted avg       0.99      0.99      0.99     26213<jupyter_text>This warning basically means that there are not enough instances of the fourth class to have in both the training and test set, so it cannot evaluate it's prediction.<jupyter_code>plot_learning_curves(vclf, X_train, y_train, X_test, y_test, range(4999, len(X_train), 5000))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09612955093602848, 'fraction_numerical': 0.05501320776386815, 'mean_word_length': 2.3665184612410592, 'pattern_counts': {'":': 0, '<': 85, '<?xml version=': 0, '>': 85, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 53, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '265699', 'license': 'MIT', 'n_tokens_mistral': 5850, 'n_tokens_neox': 5244, 'n_words': 1646, 'path': 'InternetFirewallSupervisedLearning.ipynb', 'repo_name': 'Magiduck/InternetFirewallSupervisedLearning', 'size': '17428'}