<jupyter_start><jupyter_text>Exploratory Data Analysis NotebookLet's begin by importing the necessary libraries and modules.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from sklearn.ensemble import RandomForestRegressor
import os<jupyter_output><empty_output><jupyter_text>We set a few figure parameters for the matplotlib plots.<jupyter_code>plt.rcParams.update({'figure.figsize': (10.0, 30.0)})
plt.rcParams.update({'font.size': 14})<jupyter_output><empty_output><jupyter_text>We download the necessary files from Google Drive.The local directory is not utilised since this notebook was created on Google Colaboratory and hence, having the data on the clould and downloading them whenever required was the better option.<jupyter_code>!gdown https://drive.google.com/uc?id=1oCIVZbIWmg2RvcCHcxko5alyZh19nGoF
!gdown https://drive.google.com/uc?id=1vCS6gDbsS3mVE5WLHM0nu-ww0GcI1ybN
!gdown https://drive.google.com/uc?id=10_pO2fGiAKSdNjO-p3d1dFo3VuY1juL5
!gdown https://drive.google.com/uc?id=1kSH1dteCOdFT5eFFM-1DCdbl7TZ4gXzS
!gdown https://drive.google.com/uc?id=1nRKTC6Pez1fRAMYkFIHQNLfB0Hhj5fok
!gdown https://drive.google.com/uc?id=1lA19-bvYJ_btc1WSXpKqU9L6nD3dpUjR
!gdown https://drive.google.com/uc?id=1Av_X9N5BB38edPLK0vtYIBQ9toUdDB1e
!gdown https://drive.google.com/uc?id=11u-MTRj8qiqC0rsB23F37lc4MjNh3Fhk<jupyter_output>Downloading...
From: https://drive.google.com/uc?id=1oCIVZbIWmg2RvcCHcxko5alyZh19nGoF
To: /content/Monday-WorkingHours.pcap_ISCX.csv
177MB [00:02, 85.4MB/s]
Downloading...
From: https://drive.google.com/uc?id=1vCS6gDbsS3mVE5WLHM0nu-ww0GcI1ybN
To: /content/Tuesday-WorkingHours.pcap_ISCX.csv
135MB [00:01, 90.8MB/s]
Downloading...
From: https://drive.google.com/uc?id=10_pO2fGiAKSdNjO-p3d1dFo3VuY1juL5
To: /content/Wednesday-workingHours.pcap_ISCX.csv
225MB [00:02, 109MB/s] 
Downloading...
From: https://drive.google.com/uc?id=1kSH1dteCOdFT5eFFM-1DCdbl7TZ4gXzS
To: /content/Thursday-WorkingHours-Morning-WebAttacks.pcap_ISCX.csv
52.0MB [00:00, 101MB/s] 
Downloading...
From: https://drive.google.com/uc?id=1nRKTC6Pez1fRAMYkFIHQNLfB0Hhj5fok
To: /content/Thursday-WorkingHours-Afternoon-Infilteration.pcap_ISCX.csv
83.1MB [00:00, 89.4MB/s]
Downloading...
From: https://drive.google.com/uc?id=1lA19-bvYJ_btc1WSXpKqU9L6nD3dpUjR
To: /content/Friday-WorkingHours-Morning.pcap_ISCX.csv
58.3MB [00:00, 103MB/[...]<jupyter_text>We now have all the datasets (Monday to Friday) in our pressent working directory.Since this is the initial stage of the EDA process, let us examine a random dataset.<jupyter_code>df = pd.read_csv("Thursday-WorkingHours-Afternoon-Infilteration.pcap_ISCX.csv")
df.head()<jupyter_output><empty_output><jupyter_text>Let's get into the schema of the data.<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 288602 entries, 0 to 288601
Data columns (total 79 columns):
 #   Column                        Non-Null Count   Dtype  
---  ------                        --------------   -----  
 0    Destination Port             288602 non-null  int64  
 1    Flow Duration                288602 non-null  int64  
 2    Total Fwd Packets            288602 non-null  int64  
 3    Total Backward Packets       288602 non-null  int64  
 4   Total Length of Fwd Packets   288602 non-null  int64  
 5    Total Length of Bwd Packets  288602 non-null  int64  
 6    Fwd Packet Length Max        288602 non-null  int64  
 7    Fwd Packet Length Min        288602 non-null  int64  
 8    Fwd Packet Length Mean       288602 non-null  float64
 9    Fwd Packet Length Std        288602 non-null  float64
 10  Bwd Packet Length Max         288602 non-null  int64  
 11   Bwd Packet Length Min        288602 non-null  int64  
 12   Bwd Packet Length Mean       288602 non-nul[...]<jupyter_text>We notice that for every few columns, the column names have an extra " " in the beginning.Thus, let's add it to our checklist that we should rectify this.Let's now move on with the statistics of the data.<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>We notice that two columns (Flow Bytes/s and Flow Packets/s) have NULL and INF values in them.Let's add this to our checklist as well.Hence, our checklist contains of two tasks:- Rectify column names- Remove invalid dataLet's start with them<jupyter_code>cols = list(df.columns)

for col in cols:
  if col[0] == ' ':
    df = df.rename(columns={col: col[1:]})
cols = list(df.columns)<jupyter_output><empty_output><jupyter_text>Since the number of NULL and INF records are very few, we can go ahead and delete the records since they may not amount to much of a difference.<jupyter_code>df.replace([np.inf, -np.inf], np.nan, inplace=True)
df.dropna(inplace=True)
df = df.reset_index(drop=True)<jupyter_output><empty_output><jupyter_text>Now that we have a fairly clean dataframe, we can go ahead with further analytics.Let's start with understanding by what magnitude each feature is relevant to the infiltration attacks.There are several ways to do this:- Correlation Matrix- Gini Importance- Permutation based Feature Importance- SHAP (SHapley Additive exPlanations)Let's go ahead with the classic Gini Importance using Random Forest Regression.<jupyter_code>X = df[cols[:-1]]
y = df['Label']<jupyter_output><empty_output><jupyter_text>We use Label Encoder to convert our String labels to integers to enable training the Regressor. (Machines only understand numbers)<jupyter_code>le = LabelEncoder()
y = le.fit_transform(y)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=12)
rf = RandomForestRegressor(n_estimators=100)
rf.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Now that our Regressor is ready, let's plot the importance bar graph in descending order of importance.<jupyter_code>feature_names = cols[: -1]

sorted_idx = rf.feature_importances_.argsort()
plt.barh(np.array(feature_names)[sorted_idx], rf.feature_importances_[sorted_idx])
plt.xlabel("Random Forest Feature Importance")
plt.ylabel("Features")
plt.title("Feature Importances plot")<jupyter_output><empty_output><jupyter_text>With this plot, we obtain insights about how each column affects the malware detection process.The few most important features are:- Destination Port- Subflow Fwd Bytes- Total Length of Fwd Packets These results are extremely similar to the ones produced by the researchers in the research article.Similarly, this process can be utilized to obtain the important features for individual malware categories (the results of which are stated in the article).Let's move ahead by simply merging all the datasets to create one master dataframe.<jupyter_code>datasets = os.listdir()
datasets.remove('.config')
datasets.remove('sample_data')
datasets
df = pd.read_csv(datasets[0])

for i in range(1, len(datasets)):
  df_inter = pd.read_csv(datasets[i])
  df = pd.concat([df, df_inter], axis=0)
  
cols = list(df.columns)
df = df.sample(frac = 1)<jupyter_output><empty_output><jupyter_text>Let's perform similar analysis as done earlier.<jupyter_code>df.head()
df.info()
cols = list(df.columns)

for col in cols:
  if col[0] == ' ':
    df = df.rename(columns={col: col[1:]})
cols = list(df.columns)
df.describe()<jupyter_output><empty_output><jupyter_text>Let's now look at the number of unique values in each column.<jupyter_code>print(df.nunique())
print()
for col in df:
    if(df[col].nunique()<20):
        print(col, ": ",df[col].unique())<jupyter_output>Destination Port                 53805
Flow Duration                  1050899
Total Fwd Packets                 1432
Total Backward Packets            1747
Total Length of Fwd Packets      17928
                                ...   
Idle Mean                       222016
Idle Std                        197616
Idle Max                        149737
Idle Min                        223888
Label                               13
Length: 79, dtype: int64

Fwd PSH Flags :  [0 1]
Bwd PSH Flags :  [0]
Fwd URG Flags :  [0 1]
Bwd URG Flags :  [0]
FIN Flag Count :  [0 1]
SYN Flag Count :  [0 1]
RST Flag Count :  [0 1]
PSH Flag Count :  [0 1]
ACK Flag Count :  [1 0]
URG Flag Count :  [0 1]
CWE Flag Count :  [0 1]
ECE Flag Count :  [0 1]
Fwd Avg Bytes/Bulk :  [0]
Fwd Avg Packets/Bulk :  [0]
Fwd Avg Bulk Rate :  [0]
Bwd Avg Bytes/Bulk :  [0]
Bwd Avg Packets/Bulk :  [0]
Bwd Avg Bulk Rate :  [0]
Label :  ['BENIGN' 'DDoS' 'DoS Hulk' 'PortScan' 'DoS Slowhttptest' 'DoS GoldenEye'
 'Web' 'DoS slowloris' '[...]<jupyter_text>We notice that the Label column various Web attacks stated.But, they work in similar ways (according to the dataset) and hence, grouping them as 'Web' would be a good idea.<jupyter_code>df.iloc[:,-1] = df.iloc[:, -1].apply(lambda x : 'Web' if x[:3] == 'Web' or x[:3] == 'Sql' else x)
df.iloc[:, -1].value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10105379922351636, 'fraction_numerical': 0.04470327232390461, 'mean_word_length': 4.1285551763367465, 'pattern_counts': {'":': 1, '<': 60, '<?xml version=': 0, '>': 59, 'https://': 14, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '512981', 'license': 'MIT', 'n_tokens_mistral': 3471, 'n_tokens_neox': 3159, 'n_words': 873, 'path': 'EDA.ipynb', 'repo_name': 'soundarzozm/network-intrusion-detection', 'size': '9015'}