<jupyter_start><jupyter_text>Step 2 : Experiment II ---  1. Imports<jupyter_code>import warnings
warnings.filterwarnings('ignore') 
import math
import numpy as np #operaciones matriciales y con vectores
import pandas as pd #tratamiento de datos
import random
import matplotlib.pyplot as plt #gráficos
import seaborn as sns
from sklearn.model_selection import cross_validate
from sklearn import preprocessing
import joblib
from sklearn.ensemble import  RandomForestClassifier<jupyter_output><empty_output><jupyter_text>---  2. Load the datasets Let's load the **benign dataset** :<jupyter_code>b = pd.read_csv(r"C:\Users\Usuario\Documents\Github\PDG\PDG-2\Datasets\Time Window\Originals\Benigns\BeningTimeWindowsStratosphere.csv", delimiter = ",")
b.shape<jupyter_output><empty_output><jupyter_text>Let's load the **malign train** :<jupyter_code>m = pd.read_csv(r"C:\Users\Usuario\Documents\Github\PDG\PDG-2\Datasets\Time Window\Originals\Maligns\MalignTimeWindowsStratosphere.csv", delimiter = ",")
m.shape<jupyter_output><empty_output><jupyter_text>Let's create a **sample**:<jupyter_code>m = m.sample(n=800)
m.shape<jupyter_output><empty_output><jupyter_text>Let's join both datasets:<jupyter_code>frames = [b, m]
dataset = pd.concat(frames)
dataset.head()
dataset.shape
dataset = dataset.sample(frac=1)<jupyter_output><empty_output><jupyter_text>---  3. Let's see the dataset...<jupyter_code>dataset.info()
dataset.describe(include="all").T
null_counts = dataset.isnull().sum()
print("Number of null values in each column:\n{}".format(null_counts))
fig, axs = plt.subplots(1,2,figsize=(20,10))
sns.countplot(x='Type',data=dataset,ax=axs[0])
axs[0].set_title("Frequency of each Type")
dataset.Type.value_counts().plot(x=None,y=None, kind='pie', ax=axs[1],autopct='%1.2f%%')
axs[1].set_title("Percentage of each Type")
plt.show()<jupyter_output><empty_output><jupyter_text>---  4. EDA  4.1 Remove NaNs<jupyter_code>dataset.columns
dataset['First_Protocol'] = dataset['First_Protocol'].replace(np.nan,"None",regex=True)
dataset['Second_Protocol'] = dataset['Second_Protocol'].replace(np.nan,"None",regex=True)
dataset['Third_Protocol'] = dataset['Third_Protocol'].replace(np.nan,"None",regex=True)
dataset['second_sp'] = dataset['second_sp'].replace(np.nan,-1,regex=True)
dataset['third_sp'] = dataset['third_sp'].replace(np.nan,-1,regex=True)

dataset['second_sp'] = dataset['second_sp'].replace(np.nan,-1,regex=True)
dataset['third_sp'] = dataset['third_sp'].replace(np.nan,-1,regex=True)
dataset['second_dp'] = dataset['second_dp'].replace(np.nan,-1,regex=True)
dataset['third_dp'] = dataset['third_dp'].replace(np.nan,-1,regex=True)

dataset['second_dp'] = dataset['second_dp'].replace(np.nan,-1,regex=True)
dataset['third_dp'] = dataset['third_dp'].replace(np.nan,-1,regex=True)
null_counts = dataset.isnull().sum()
print("Number of null values in each column:\n{}".format(null_counts))<jupyter_output>Number of null values in each column:
Name               0
Netflows           0
First_Protocol     0
Second_Protocol    0
Third_Protocol     0
p1_d               0
p2_d               0
p3_d               0
duration           0
max_d              0
min_d              0
packets            0
Avg_bps            0
Avg_pps            0
Avg_bpp            0
Bytes              0
number_sp          0
number_dp          0
first_sp           0
second_sp          0
third_sp           0
first_dp           0
second_dp          0
third_dp           0
p1_ip              0
p2_ip              0
p3_ip              0
p1_ib              0
p2_ib              0
p3_ib              0
Type               0
dtype: int64<jupyter_text>4.2 Changing Dtypes Columns like :    1. second_sp   2. third_sp   3. second_dp   4. third_dp Have to be **INTEGER** because they are **PORTS**<jupyter_code>dataset['second_sp'] = dataset['second_sp'].astype(np.int64)
dataset['third_sp'] = dataset['third_sp'].astype(np.int64)
dataset['second_dp'] = dataset['second_dp'].astype(np.int64)
dataset['third_dp'] = dataset['third_dp'].astype(np.int64)
dataset.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1319 entries, 284 to 231862
Data columns (total 31 columns):
Name               1319 non-null object
Netflows           1319 non-null int64
First_Protocol     1319 non-null object
Second_Protocol    1319 non-null object
Third_Protocol     1319 non-null object
p1_d               1319 non-null float64
p2_d               1319 non-null float64
p3_d               1319 non-null float64
duration           1319 non-null float64
max_d              1319 non-null float64
min_d              1319 non-null float64
packets            1319 non-null int64
Avg_bps            1319 non-null int64
Avg_pps            1319 non-null int64
Avg_bpp            1319 non-null int64
Bytes              1319 non-null int64
number_sp          1319 non-null int64
number_dp          1319 non-null int64
first_sp           1319 non-null int64
second_sp          1319 non-null int64
third_sp           1319 non-null int64
first_dp           1319 non-null int64
second_dp      [...]<jupyter_text>4.3  Drop Innnecesary Columns... Let's verify if there are columns that have less than 4 values in the column's values<jupyter_code>for col in dataset.columns:
    if (len(dataset[col].unique()) < 4):
        print(dataset[col].value_counts())
        print()<jupyter_output>Malign    800
Benign    519
Name: Type, dtype: int64<jupyter_text>Obviously, the column **Type** can't be delete because it's the **TARGET** ---  4.4. Let's create the Dictionary<jupyter_code>le = preprocessing.LabelEncoder()<jupyter_output><empty_output><jupyter_text>4.4.1 First_Protocol var<jupyter_code>dataset.First_Protocol.unique()
first_protocol_column_codified = le.fit_transform(dataset.First_Protocol)
le_name_mapping_first_protocol = dict(zip(le.classes_, le.transform(le.classes_)))
print(le_name_mapping_first_protocol)
dataset.First_Protocol = le.fit_transform(dataset.First_Protocol)
joblib.dump(le, './Tools/label_encoder_first_protocol_exp1.encoder')<jupyter_output><empty_output><jupyter_text>4.4.2 Second_Protocol var<jupyter_code>dataset.Second_Protocol.unique()
second_protocol_column_codified = le.fit_transform(dataset.Second_Protocol)
le_name_mapping_second_protocol = dict(zip(le.classes_, le.transform(le.classes_)))
print(le_name_mapping_second_protocol)
dataset.Second_Protocol = le.fit_transform(dataset.Second_Protocol)
joblib.dump(le, './Tools/label_encoder_second_protocol_exp1.encoder')<jupyter_output><empty_output><jupyter_text>4.4.3 Third_Protocol var<jupyter_code>dataset.Third_Protocol.unique()
third_protocol_column_codified = le.fit_transform(dataset.Third_Protocol)
le_name_mapping_third_protocol = dict(zip(le.classes_, le.transform(le.classes_)))
print(le_name_mapping_third_protocol)
dataset.Third_Protocol = le.fit_transform(dataset.Third_Protocol)
joblib.dump(le, './Tools/label_encoder_third_protocol_exp1.encoder')<jupyter_output><empty_output><jupyter_text>4.4.4 Type<jupyter_code>dataset.Type.unique()
Type_protocol_column_codified = le.fit_transform(dataset.Type)
le_name_mapping_type = dict(zip(le.classes_, le.transform(le.classes_)))
print(le_name_mapping_type)
dataset.Type = le.fit_transform(dataset.Type)
joblib.dump(le, './Tools/label_encoder_type_exp1.encoder')<jupyter_output><empty_output><jupyter_text>---  4.5 Let's see the outliers...<jupyter_code>dataset.columns
invalid = ["Name","First_Protocol","Second_Protocol","Third_Protocol","Type"]
for column in dataset.columns:  
    if column not in invalid:
        fig = plt.figure()
        ax = plt.subplot()
        ax.set_title("Boxplot of %s" %column)
        ax.boxplot(dataset[column])<jupyter_output><empty_output><jupyter_text>--- --- --- --- ---  5. Let's Standardize the dataset...<jupyter_code>names = dataset.Name
dataset = dataset.drop(['Name'], axis=1)
scaler = preprocessing.RobustScaler()
dataset.columns
dataset 
dataset[['Netflows', 
       'p1_d', 'p2_d', 'p3_d', 'duration', 'max_d', 'min_d', 'packets',
       'Avg_bps', 'Avg_pps', 'Avg_bpp', 'Bytes', 'number_sp', 'number_dp',
        'p1_ip', 'p2_ip', 'p3_ip', 'p1_ib', 'p2_ib', 'p3_ib']] = scaler.fit_transform(dataset[['Netflows', 
       'p1_d', 'p2_d', 'p3_d', 'duration', 'max_d', 'min_d', 'packets',
       'Avg_bps', 'Avg_pps', 'Avg_bpp', 'Bytes', 'number_sp', 'number_dp',
        'p1_ip', 'p2_ip', 'p3_ip', 'p1_ib', 'p2_ib', 'p3_ib'
]])
dataset.shape
dataset.head()<jupyter_output><empty_output><jupyter_text>---  6.  ::::::: FEATURE IMPORTANCE ::::::::<jupyter_code>clf=RandomForestClassifier(n_estimators =10, random_state = 1234, class_weight="balanced")
X = dataset.iloc[:,:-1]
y = dataset.Type
output = cross_validate(clf, X, y, cv=20, scoring = 'accuracy', return_estimator =True)
first_sp = []
avg_bps = []
number_sp = []
Bytes = []
number_dp = []
duration = []
p1_ib = []
First_Protocol = []
Avg_pps = []
p2_ib = []
p2_d = []
Netflows = []
p3_d = []
p3_ib = []
max_d = []
p3_ip = []
Avg_bpp = []
packets = []
second_sp = []
first_dp = []
third_sp = []
Second_Protocol = []
second_dp = []
p1_d = []
Third_Protocol = []
third_dp = []
p2_ip = []
p1_ip = []
min_d = []
for idx,estimator in enumerate(output['estimator']):
#     print("Features sorted by their score for estimator {}:".format(idx))    
    feature_importances = pd.DataFrame(estimator.feature_importances_,
                                       index = X.columns,
                                       columns=['importance']).sort_values('importance', ascending=False)
#     print(feature_importances)
    for i in range(0,len(feature_importances.index)):
        if feature_importances.index[i] == "first_sp":
            first_sp.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "Avg_bps":
            avg_bps.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "number_sp":
            number_sp.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "Bytes":
            Bytes.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "number_dp":
            number_dp.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "duration":
            duration.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "p1_ib":
            p1_ib.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "First_Protocol":
            First_Protocol.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "Avg_pps":
            Avg_pps.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "p2_ib":
            p2_ib.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "p2_d":
            p2_d.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "Netflows":
            Netflows.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "p3_d":
            p3_d.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "p3_ib":
            p3_ib.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "max_d":
            max_d.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "p3_ip":
            p3_ip.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "Avg_bpp":
            Avg_bpp.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "packets":
            packets.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "second_sp":
            second_sp.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "first_dp":
            first_dp.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "third_sp":
            third_sp.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "Second_Protocol":
            Second_Protocol.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "second_dp":
            second_dp.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "p1_d":
            p1_d.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "Third_Protocol":
            Third_Protocol.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "third_dp":
            third_dp.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "p2_ip":
            p2_ip.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "p1_ip":
            p1_ip.append(feature_importances.iloc[i,0])
        if feature_importances.index[i] == "min_d":
            min_d.append(feature_importances.iloc[i,0])
        
        
first_sp = pd.Series(first_sp)
avg_bps = pd.Series(avg_bps)
number_sp = pd.Series(number_sp)
Bytes = pd.Series(Bytes)
number_dp = pd.Series(number_dp)
duration = pd.Series(duration)
p1_ib = pd.Series(p1_ib)
First_Protocol = pd.Series(First_Protocol)
Avg_pps = pd.Series(Avg_pps)
p2_ib = pd.Series(p2_ib)
p2_d = pd.Series(p2_d)
Netflows = pd.Series(Netflows)
p3_d = pd.Series(p3_d)
p3_ib = pd.Series(p3_ib)
max_d = pd.Series(max_d)
p3_ip = pd.Series(p3_ip)
Avg_bpp = pd.Series(Avg_bpp)
packets = pd.Series(packets)
second_sp = pd.Series(second_sp)
first_dp = pd.Series(first_dp)
third_sp = pd.Series(third_sp)
Second_Protocol = pd.Series(Second_Protocol)
second_dp = pd.Series(second_dp)
p1_d = pd.Series(p1_d)
Third_Protocol = pd.Series(Third_Protocol)
third_dp = pd.Series(third_dp)
p2_ip = pd.Series(p2_ip)
p1_ip = pd.Series(p1_ip)
min_d = pd.Series(min_d)
var_pd = pd.DataFrame(columns=['Var',"Value"])
var_pd= var_pd.append({'Var' : 'First_sp', 'Value' : first_sp.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'Avg_bps', 'Value' : avg_bps.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'Number_sp', 'Value' : number_sp.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'Bytes', 'Value' : Bytes.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'Number_dp', 'Value' : number_dp.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'Duration', 'Value' : duration.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'P1_ib', 'Value' : p1_ib.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'First_Protocol', 'Value' : First_Protocol.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'Avg_pps', 'Value' : Avg_pps.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'P2_ib', 'Value' : p2_ib.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'P2_d', 'Value' : p2_d.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'Netflows', 'Value' : Netflows.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'P3_d', 'Value' : p3_d.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'P3_ib', 'Value' : p3_ib.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'Max_d', 'Value' : max_d.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'P3_ib', 'Value' : p3_ip.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'Avg_bpp', 'Value' : Avg_bpp.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'Packets', 'Value' : packets.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'Second_sp', 'Value' : second_sp.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'First_dp', 'Value' : first_dp.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'Third_sp', 'Value' : third_sp.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'Second_protocol', 'Value' : Second_Protocol.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'Second_dp', 'Value' : second_dp.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'P1_d', 'Value' : p1_d.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'Third_Protocol', 'Value' : Third_Protocol.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'Third_dp', 'Value' : third_dp.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'P2_ip', 'Value' : p2_ip.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'P1_ip', 'Value' : p1_ip.mean()}, ignore_index=True)
var_pd=var_pd.append({'Var' : 'Min_d', 'Value' : min_d.mean()}, ignore_index=True)
var_pd.sort_values(by=['Value'], ascending=False)<jupyter_output><empty_output><jupyter_text>---  Important In this **step**, we have to decide what columns we will use in the training. In the **previous dataframe**, we can see the values of the **mean of the importance stimator of each column, order in Descending order.**.  We decide to choose the vars that have in the column of "value", a **value > 0.01**. In this case, the following columns have been selected: 1. First_sp	2. Avg_bps	3. P1_ib	4. Duration	5. Number_dp	6. Bytes	7. Number_sp	8. First_Protocol	9. P2_ib	10. First_dp	11. P3_ib	12. Netflows	13. P3_d	14.  Second_protocol<jupyter_code>dataset.columns
dataset = dataset[['first_sp','Avg_bps','p1_ib','duration','number_dp','Bytes'
                                          ,'number_sp','First_Protocol','p2_ib'
                                          ,'first_dp','p3_ib','Netflows','p3_d','Second_Protocol','Type']]
dataset.columns
columns_chosen = dataset.columns
columns_chosen
dataset.to_csv(r"C:\Users\Usuario\Documents\Github\PDG\PDG-2\Datasets\Time Window\Standardized\SDatasetExp1.csv",sep=',',index=False)<jupyter_output><empty_output><jupyter_text>---  8. Let's save the Robust Scaler...<jupyter_code>scaler_filename = "./Tools/scalerExp1.save"
joblib.dump(scaler, scaler_filename)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14780587368182332, 'fraction_numerical': 0.023925615149109878, 'mean_word_length': 4.361398176291793, 'pattern_counts': {'":': 29, '<': 76, '<?xml version=': 0, '>': 76, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '315996', 'license': 'MIT', 'n_tokens_mistral': 6597, 'n_tokens_neox': 6415, 'n_words': 1042, 'path': 'Botnets/Phases/Phase 3/Experiments/New Step/.ipynb_checkpoints/Step 2  Experiment -I EDA-checkpoint.ipynb', 'repo_name': 'i2tResearch/Ciberseguridad_web', 'size': '17638'}