<jupyter_start><jupyter_text>I Step: Clean both datasets (Benign and Malign) ---  1. Imports<jupyter_code>import warnings
warnings.filterwarnings('ignore') 
import math
import numpy as np #operaciones matriciales y con vectores
import pandas as pd #tratamiento de datos
import random
import matplotlib.pyplot as plt #gráficos
import seaborn as sns<jupyter_output><empty_output><jupyter_text>---  2. Dataset Nomenclatures 1. **Name**: Reference of the time window2. **Netflows**: Number of netflows in the t.w3. **First_protocol**: Top 1 of the protocols in the t.w4. **Second_protocol**: Top 2 of the protocols in the t.w5. **Third_protocol**: Top 3 of the protocols in the t.w6. **p1_d**: percentil 25% of the duration7. **p2_d**: percentil 50% of the duration8. **p3_d**: percentil 75% of the duration9. **duration** : sum of all durations in the t.w10. **max_d**: max number of all duration values in the t.w11. **min_d**: min number of all duration values in the t.w12. **packets**: number of packets in the t.w13. **Avg_bps**: Average of bits per second14. **Avg_pps**: Average of packets per second15. **Avg_bpp**: Average of bytes per package16. **Bytes**: Number of bytes in the t.w17. **sp**: total of source ports in the t.w18. **dp**: total of destination ports in the t.w19. **first_sp**: top 1 of source ports in the t.w20. **second_sp**: top 2 of source ports in the t.w21. **third_sp**: top 3 of source ports in the t.w22. **first_dp**: top 1 of destination ports in the t.w23. **second_dp**: top 2 of destination ports in the t.w24. **third_dp**: top 3 of destination ports in the t.w25. **p1_ip**: percentil 25% of input packets26. **p2_ip**: percentil 50% of input packets27. **p3_ip**: percentil 75% of input packets28. **p1_ib**: percentil 25% of input bytes29. **p2_ib**: percentil 50% of input bytes30. **p3_ib**: percentil 75% of input bytes31. **type**: Benign/Malign ---  3. Stratosphere's Benign Dataset  3.1 Let's Load...<jupyter_code>stratosphere_benign_dataset = pd.read_csv(r"C:\Users\Usuario\Documents\Github\PDG\PDG-2\Datasets\Time Window\Originals\Benigns v2\BeningTimeWindowsStratosphere.csv", delimiter = ",")
stratosphere_benign_dataset.shape
stratosphere_benign_dataset.head()<jupyter_output><empty_output><jupyter_text>3.2 Let's transform the NaN Values...<jupyter_code>stratosphere_benign_dataset['First_Protocol'] = stratosphere_benign_dataset['First_Protocol'].replace(np.nan,"None",regex=True)
stratosphere_benign_dataset['Second_Protocol'] = stratosphere_benign_dataset['Second_Protocol'].replace(np.nan,"None",regex=True)
stratosphere_benign_dataset['Third_Protocol'] = stratosphere_benign_dataset['Third_Protocol'].replace(np.nan,"None",regex=True)
stratosphere_benign_dataset['second_sp'] = stratosphere_benign_dataset['second_sp'].replace(np.nan,-1,regex=True)
stratosphere_benign_dataset['third_sp'] = stratosphere_benign_dataset['third_sp'].replace(np.nan,-1,regex=True)
stratosphere_benign_dataset['second_dp'] = stratosphere_benign_dataset['second_dp'].replace(np.nan,-1,regex=True)
stratosphere_benign_dataset['third_dp'] = stratosphere_benign_dataset['third_dp'].replace(np.nan,-1,regex=True)<jupyter_output><empty_output><jupyter_text>3.3 Let's change dtypes of some columns... Columns like :     1. second_sp    2. third_sp    3. second_dp    4. third_dp Have to be **INTEGER** because they are **PORTS**<jupyter_code>stratosphere_benign_dataset['second_sp'] = stratosphere_benign_dataset['second_sp'].astype(np.int64)
stratosphere_benign_dataset['third_sp'] = stratosphere_benign_dataset['third_sp'].astype(np.int64)
stratosphere_benign_dataset['second_dp'] = stratosphere_benign_dataset['second_dp'].astype(np.int64)
stratosphere_benign_dataset['third_dp'] = stratosphere_benign_dataset['third_dp'].astype(np.int64)<jupyter_output><empty_output><jupyter_text>3.4 Let's see the dataset...<jupyter_code>stratosphere_benign_dataset.info()
stratosphere_benign_dataset.describe(include="all").T<jupyter_output><empty_output><jupyter_text>3.5 Let's plot the dataset<jupyter_code>fig = plt.figure(figsize=(25,20))
ax = fig.gca()
stratosphere_benign_dataset.hist(ax = ax)
invalid = ["First_Protocol","Second_Protocol","Third_Protocol","Type","Name"]
invalid
for column in stratosphere_benign_dataset.columns:  
    if column not in invalid:
        fig = plt.figure()
        ax = plt.subplot()
        ax.set_title("Boxplot of %s" %column)
        ax.boxplot(stratosphere_benign_dataset[column])<jupyter_output><empty_output><jupyter_text>3.6 Let's drop some outliers...  p1_d<jupyter_code>sns.boxplot(y=stratosphere_benign_dataset["p1_d"])
values = stratosphere_benign_dataset.iloc[stratosphere_benign_dataset['p1_d'].argsort()[-4:]]
values = values[['p1_d','Name']]
print(values)
stratosphere_benign_dataset=stratosphere_benign_dataset[stratosphere_benign_dataset.p1_d<0.017]
sns.boxplot(y=stratosphere_benign_dataset["p1_d"])<jupyter_output><empty_output><jupyter_text>first_dp<jupyter_code>sns.boxplot(y=stratosphere_benign_dataset["first_dp"])
values = stratosphere_benign_dataset.iloc[stratosphere_benign_dataset['first_dp'].argsort()[-4:]]
values = values[['first_dp','Name']]
print(values)
stratosphere_benign_dataset=stratosphere_benign_dataset[stratosphere_benign_dataset.first_dp<59370]
sns.boxplot(y=stratosphere_benign_dataset["first_dp"])<jupyter_output><empty_output><jupyter_text>3.7 Let's save the cleaned Stratosphere's Benign Dataset<jupyter_code>stratosphere_benign_dataset.to_csv(r"C:\Users\Usuario\Documents\Github\PDG\PDG-2\Datasets\Time Window\Cleaned\CleanedStratosphersBenignDataset.csv",sep=',',index=False)<jupyter_output><empty_output><jupyter_text>--- ---  4. Stratosphere's Malign Dataset  4.1 Let's load the dataset...<jupyter_code>stratosphere_malign_dataset = pd.read_csv(r"C:\Users\Usuario\Documents\Github\PDG\PDG-2\Datasets\Time Window\Originals\Maligns v2\MalignTimeWindowsStratosphere.csv",
                                          delimiter = ",")
stratosphere_malign_dataset.shape
stratosphere_malign_dataset.head()<jupyter_output><empty_output><jupyter_text>4.2 Let's transform the NaN values...<jupyter_code>stratosphere_malign_dataset['First_Protocol'] = stratosphere_malign_dataset['First_Protocol'].replace(np.nan,"None",regex=True)
stratosphere_malign_dataset['Second_Protocol'] = stratosphere_malign_dataset['Second_Protocol'].replace(np.nan,"None",regex=True)
stratosphere_malign_dataset['Third_Protocol'] = stratosphere_malign_dataset['Third_Protocol'].replace(np.nan,"None",regex=True)
stratosphere_malign_dataset['second_sp'] = stratosphere_malign_dataset['second_sp'].replace(np.nan,-1,regex=True)
stratosphere_malign_dataset['third_sp'] = stratosphere_malign_dataset['third_sp'].replace(np.nan,-1,regex=True)
stratosphere_malign_dataset['second_dp'] = stratosphere_malign_dataset['second_dp'].replace(np.nan,-1,regex=True)
stratosphere_malign_dataset['third_dp'] = stratosphere_malign_dataset['third_dp'].replace(np.nan,-1,regex=True)<jupyter_output><empty_output><jupyter_text>4.3 Let's change dtypes of some columns Columns like :     1. second_sp    2. third_sp    3. second_dp    4. third_dp Have to be **INTEGER** because they are **PORTS**<jupyter_code>stratosphere_malign_dataset['second_sp'] = stratosphere_malign_dataset['second_sp'].astype(np.int64)
stratosphere_malign_dataset['third_sp'] = stratosphere_malign_dataset['third_sp'].astype(np.int64)
stratosphere_malign_dataset['second_dp'] = stratosphere_malign_dataset['second_dp'].astype(np.int64)
stratosphere_malign_dataset['third_dp'] = stratosphere_malign_dataset['third_dp'].astype(np.int64)<jupyter_output><empty_output><jupyter_text>4.4 Let's see the dataset...<jupyter_code>stratosphere_malign_dataset.info()
stratosphere_malign_dataset.describe(include="all").T<jupyter_output><empty_output><jupyter_text>4.5 Let's plot the dataset...<jupyter_code>fig = plt.figure(figsize=(25,20))
ax = fig.gca()
stratosphere_malign_dataset.hist(ax = ax)
for column in stratosphere_malign_dataset.columns:  
    if column not in invalid:
        fig = plt.figure()
        ax = plt.subplot()
        ax.set_title("Boxplot of %s" %column)
        ax.boxplot(stratosphere_malign_dataset[column])<jupyter_output><empty_output><jupyter_text>4.6 Let's drop some outliers...  p1_ib<jupyter_code>sns.boxplot(y=stratosphere_malign_dataset["p1_ib"])
values = stratosphere_malign_dataset.iloc[stratosphere_malign_dataset['p1_ib'].argsort()[-4:]]
values = values[['p1_ib','Name']]
print(values)
stratosphere_malign_dataset=stratosphere_malign_dataset[stratosphere_malign_dataset.p1_ib<153547]
sns.boxplot(y=stratosphere_malign_dataset["p1_ib"])<jupyter_output><empty_output><jupyter_text>4.7 Let's save the cleaned Stratosphere's Malign Dataset<jupyter_code>stratosphere_malign_dataset.to_csv(r"C:\Users\Usuario\Documents\Github\PDG\PDG-2\Datasets\Time Window\Cleaned\CleanedStratosphersMalignDataset.csv",sep=',',index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13228523301516001, 'fraction_numerical': 0.022122403144300955, 'mean_word_length': 8.01417004048583, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 65, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '631966', 'license': 'MIT', 'n_tokens_mistral': 3260, 'n_tokens_neox': 3122, 'n_words': 511, 'path': 'Botnets/Phases/Phase 3/Experiments/.ipynb_checkpoints/Step_1 EDA of the originals Datasets-checkpoint.ipynb', 'repo_name': 'i2tResearch/Ciberseguridad_web', 'size': '8905'}