<jupyter_start><jupyter_text><NAME> & <NAME>Malicious Data detection based on Neural NetworkSR2I Projet du filière, supervised by: <NAME>17 Juin 2019  Content 1. [Analyzing the dataset](paragraph1) 2. [Extracting the feature vectors](paragraphe2) 3. [Training and testing the Neural Network](paragraphe3) 4. [Comparison with other models](paragraphe4) 5. [Testing the model](paragraphe5)Note that all the functions used in this scripts are stored in the 'facilities file', check the file if needed.   1. Analyzing the dataset<jupyter_code>import pandas as pd 
import matplotlib.pyplot as plt 
import numpy as np

from keras.models import Sequential
from keras.layers import Dense,Activation
from sklearn.model_selection import train_test_split

from ipynb.fs.defs.facilities import *
data_file = ['attack1withlabels', 'attack2withlabels', 'attack4withlabels',
             'attack8withlabels','attack16withlabels']
data_atk1 = read_raw_data(data_file[0])
data_atk2 = read_raw_data(data_file[1])
data_atk4 = read_raw_data(data_file[2])
data_atk8 = read_raw_data(data_file[3])
data_atk16 = read_raw_data(data_file[4])

data_atk2['Label'] = data_atk2['Label']/2
data_atk4['Label'] = data_atk4['Label']/4
data_atk8['Label'] = data_atk8['Label']/8
data_atk16['Label'] = data_atk16['Label']/16

raw_data = [data_atk1, data_atk2, data_atk4, data_atk8, data_atk16]
atk_type = [1,2,4,8,16]<jupyter_output><empty_output><jupyter_text>2.1 All the BSM message in which the distance is bigger than range of line-of-sight (800m for this project) is malicious<jupyter_code>for i in range(5):
    data = raw_data[i]
    # We first get the index of the BSM in which the distance is above 800.
    Plus800_index = check_range(data, 800)
    # We get the all the labels of all the BSM message.
    data_Plus800 = data.iloc[Plus800_index]
    label_this = np.unique(np.array(data_Plus800.iloc[:,11]))
    print('For all the data for attack type', atk_type[i], ', The labels are ', label_this)<jupyter_output>For all the data for attack type 1 , The labels are  [1]
For all the data for attack type 2 , The labels are  [1]
For all the data for attack type 3 , The labels are  [1]
For all the data for attack type 4 , The labels are  [1]
For all the data for attack type 5 , The labels are  [1]<jupyter_text>2.2 For a communication session between a sender and a receiver, all the message are either malicious or normal<jupyter_code>for i in range(5):
    data = raw_data[i]
    print('************For the attack type',atk_type[i], '************' )
    # we show the statistics of the dataset. 
    statistics = check_session(data)<jupyter_output>************For the attack type 1 ************
There are  387516 rows in the dataset
267305 rows are normal, i.e.,  0.689790873151044 percent of rows in the dataset
267305 rows are malicious, i.e.,  0.689790873151044 percent of rows in the dataset
There are  30588 sessions in the dataset
20973 session are normal, i.e.,  0.6856610435464888 percent of sessions in the dataset 
9615 session are malicious, i.e.,  0.3143389564535112 percent of sessions in the dataset 
For all the session, there are only 1 kind of label, in other word, a session is whether attack or normal
************For the attack type 2 ************
There are  387516 rows in the dataset
267305 rows are normal, i.e.,  0.689790873151044 percent of rows in the dataset
387516 rows are malicious, i.e.,  1.0 percent of rows in the dataset
There are  30588 sessions in the dataset
20973 session are normal, i.e.,  0.6856610435464888 percent of sessions in the dataset 
9615 session are malicious, i.e.,  0.3143389564535112 percent of[...]<jupyter_text>2. Extracting the feature vectors  We extract 6 features from the paper,<jupyter_code>for i in range(5):
    data = raw_data[i]
    data_with_features = add_feature_vectors(data)
    put_csv(data_with_features, atk_type[i])<jupyter_output><empty_output><jupyter_text>We save the vectors in csv file for further processing.  3. Training and testing the Neural Network<jupyter_code>vector_file = ['attack1with7FeatureVector', 'attack2with7FeatureVector', 'attack4with7FeatureVector',
             'attack8with7FeatureVector','attack16with7FeatureVector']

data_vector1 = read_vector_data(vector_file[0])
data_vector2 = read_vector_data(vector_file[1])
data_vector4 = read_vector_data(vector_file[2])
data_vector8 = read_vector_data(vector_file[3])
data_vector16 = read_vector_data(vector_file[4])

data_vector2['Label'] = data_vector2['Label']/2
data_vector4['Label'] = data_vector4['Label']/4
data_vector8['Label'] = data_vector8['Label']/8
data_vector16['Label'] = data_vector16['Label']/16<jupyter_output><empty_output><jupyter_text>3.1 Check<jupyter_code>Hyper_parameter = ['sigmoid', 'mean_absolute_error', 'RMSprop', 'binary_accuracy']
NN_structure = [64, 16, 1]
model1 = NN_model(data_vector1, Hyper_parameter, NN_structure)
model2 = NN_model(data_vector2, Hyper_parameter, NN_structure)
model4 = NN_model(data_vector4, Hyper_parameter, NN_structure)
model8 = NN_model(data_vector8, Hyper_parameter, NN_structure)
model16 = NN_model(data_vector16, Hyper_parameter, NN_structure)
data_overall = data_vector1.append([data_vector2, data_vector4, data_vector8, data_vector16])
model_all = NN_model(data_overall, Hyper_parameter, NN_structure)
model_trained = [model1, model2, model4, model8, model16]<jupyter_output><empty_output><jupyter_text>ClassficationThe first step is identify whether the session is normal or malicious. Now we train a second model to classify the five types of attack.<jupyter_code>malicicous_1 = data_vector1.loc[(data_vector1['Label']==1)]
malicicous_2 = data_vector2.loc[(data_vector2['Label']==1)]
malicicous_4 = data_vector4.loc[(data_vector4['Label']==1)]
malicicous_8 = data_vector8.loc[(data_vector8['Label']==1)]
malicicous_16 = data_vector16.loc[(data_vector16['Label']==1)]

malicious_data_overall = malicicous_1.append([malicicous_2, malicicous_4, malicicous_8, malicicous_16])
labels_classfication = to_labels(malicicous_1, malicicous_2, malicicous_4, malicicous_8, malicicous_16)
Hyper_parameter = ['sigmoid', 'categorical_crossentropy', 'RMSprop', 'categorical_accuracy']
NN_structure = [128, 32, 5]

model_classification = Classification_model(malicious_data_overall, labels_classfication, Hyper_parameter, NN_structure)<jupyter_output>Epoch 1/20
37337/37337 [==============================] - 2s 41us/step - loss: 1.1565 - categorical_accuracy: 0.5686
Epoch 2/20
37337/37337 [==============================] - 1s 15us/step - loss: 0.6428 - categorical_accuracy: 0.7673
Epoch 3/20
37337/37337 [==============================] - 1s 15us/step - loss: 0.4872 - categorical_accuracy: 0.8019
Epoch 4/20
37337/37337 [==============================] - 1s 15us/step - loss: 0.4369 - categorical_accuracy: 0.8132
Epoch 5/20
37337/37337 [==============================] - 1s 15us/step - loss: 0.4033 - categorical_accuracy: 0.8216
Epoch 6/20
37337/37337 [==============================] - 1s 15us/step - loss: 0.3846 - categorical_accuracy: 0.8259
Epoch 7/20
37337/37337 [==============================] - 1s 15us/step - loss: 0.3732 - categorical_accuracy: 0.8310
Epoch 8/20
37337/37337 [==============================] - 1s 15us/step - loss: 0.3619 - categorical_accuracy: 0.8352
Epoch 9/20
37337/37337 [==============================] - 1s 15u[...]<jupyter_text>4. Comparison with other models  This part is done by Liang and show in separate file.  5. Testing the model  We randomly choose a session for validating the models.<jupyter_code>input_type = 4 # Which type of attack our malicious is chosen from, choose 0~4 please1
mode = 1 # Whether the data that we choose is attack or normal, choose 0 or 1 please!
validation(input_type, mode)<jupyter_output>The session we randomly choose from the dataset 16  is shown as follows
{'fraction_non_alphanumeric': 0.1366547131147541, 'fraction_numerical': 0.08119877049180328, 'mean_word_length': 5.98479427549195, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '278862', 'license': 'MIT', 'n_tokens_mistral': 2992, 'n_tokens_neox': 2592, 'n_words': 745, 'path': 'main.ipynb', 'repo_name': 'liangwangENST/SR2I208-Detection-Malicious-data', 'size': '7834'}