<jupyter_start><jupyter_text>Pickle NSL-KDD This notebooks intended use is to load the CSV data into a Pandas dataframe, normalize and scale the data, then write the DataFrame into a pickle to save these steps for every ML framework run.  The output are four pickle files: kdd_train_data, kdd_train_labels, kdd_test_data and kdd_test_labels.  These pickles can be restored as dataframes by calling [pandas.read_pickle()](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_pickle.html).<jupyter_code>import os
import json
import pandas as pd
import numpy as np
from keras.preprocessing.text import Tokenizer
from sklearn import preprocessing
pd.set_option('display.max_columns', None)<jupyter_output>Using TensorFlow backend.<jupyter_text>Data Loading and Prep Fist of all, load the header, so Pandas is able to identify the columns correctly.  As the header info for the last two columns is missing, I am adding these by hand to the DataFrame.  They are the type of traffic, *label*, and the difficulty of detection, *difficulty_level*.  These columns are removed from the data-pickled but for completeness' sake, I'm including these.<jupyter_code>print('Loading header info from "Field Names.csv"')
header_col = pd.read_csv(os.path.join('NSL_KDD', 'Field Names.csv'), header=None)
header_col = header_col.append(pd.DataFrame([['label','symbolic'],['difficulty_level','continuous']]))

header_names = header_col[0].values
print('loaded {} header names: {}'.format(len(header_names), header_names))<jupyter_output>Loading header info from "Field Names.csv"
loaded 43 header names: ['duration' 'protocol_type' 'service' 'flag' 'src_bytes' 'dst_bytes'
 'land' 'wrong_fragment' 'urgent' 'hot' 'num_failed_logins' 'logged_in'
 'num_compromised' 'root_shell' 'su_attempted' 'num_root'
 'num_file_creations' 'num_shells' 'num_access_files' 'num_outbound_cmds'
 'is_host_login' 'is_guest_login' 'count' 'srv_count' 'serror_rate'
 'srv_serror_rate' 'rerror_rate' 'srv_rerror_rate' 'same_srv_rate'
 'diff_srv_rate' 'srv_diff_host_rate' 'dst_host_count'
 'dst_host_srv_count' 'dst_host_same_srv_rate' 'dst_host_diff_srv_rate'
 'dst_host_same_src_port_rate' 'dst_host_srv_diff_host_rate'
 'dst_host_serror_rate' 'dst_host_srv_serror_rate' 'dst_host_rerror_rate'
 'dst_host_srv_rerror_rate' 'label' 'difficulty_level']<jupyter_text>Training Set We're using "KDDTrain+" specifically, as this is the full training data set.<jupyter_code>ftrain = os.path.join('NSL_KDD','KDDTrain+.csv')
kdd_train_data = pd.read_csv(ftrain, header=None, names=header_names)

# split off labels
kdd_train_labels = kdd_train_data.filter(['label','difficulty_level'])

# ...and drop them as well as the difficulty level
kdd_train_data.drop(['label', 'difficulty_level'],axis=1,inplace=True)

kdd_train_data.tail()<jupyter_output><empty_output><jupyter_text>Test Set As with the Training set, we're using the full "KDDTest+" file.<jupyter_code>ftest = os.path.join('NSL_KDD','KDDTest+.csv')
kdd_test_data = pd.read_csv(ftest, header=None, names=header_names)

# split off labels..
kdd_test_labels = kdd_test_data.filter(['label','difficulty_level'])

# ...and drop them as well as the difficulty level
kdd_test_data.drop(['label', 'difficulty_level'],axis=1,inplace=True)

kdd_test_data.tail()<jupyter_output><empty_output><jupyter_text>Attack Label Encoding As the labels are still plaintext, they need to be converted to simple integer representations.<jupyter_code>all_labels = pd.concat([kdd_train_labels, kdd_test_labels])
label_tokenizer = Tokenizer(num_words=len(all_labels['label'].unique())+1, filters='')
label_tokenizer.fit_on_texts(all_labels['label'].unique())

filename = os.path.join('NSL_KDD', 'kdd_label_wordindex.json')
print('Writing encoder data to file {}: {}'.format(filename, label_tokenizer.word_index))
with open(filename, 'w') as outfile:
    json.dump(label_tokenizer.word_index, outfile)<jupyter_output>Writing encoder data to file NSL_KDD/kdd_label_wordindex.json: {'normal': 1, 'neptune': 2, 'warezclient': 3, 'ipsweep': 4, 'portsweep': 5, 'teardrop': 6, 'nmap': 7, 'satan': 8, 'smurf': 9, 'pod': 10, 'back': 11, 'guess_passwd': 12, 'ftp_write': 13, 'multihop': 14, 'rootkit': 15, 'buffer_overflow': 16, 'imap': 17, 'warezmaster': 18, 'phf': 19, 'land': 20, 'loadmodule': 21, 'spy': 22, 'perl': 23, 'saint': 24, 'mscan': 25, 'apache2': 26, 'snmpgetattack': 27, 'processtable': 28, 'httptunnel': 29, 'ps': 30, 'snmpguess': 31, 'mailbomb': 32, 'named': 33, 'sendmail': 34, 'xterm': 35, 'worm': 36, 'xlock': 37, 'xsnoop': 38, 'sqlattack': 39, 'udpstorm': 40}<jupyter_text>Train Data<jupyter_code>encoded_train_labels = pd.DataFrame(columns=['label_encoded'],dtype=np.int8,data=label_tokenizer.texts_to_sequences(kdd_train_labels['label']))
kdd_train_labels = pd.concat([kdd_train_labels, encoded_train_labels], axis=1)
kdd_train_labels.head()<jupyter_output><empty_output><jupyter_text>Test Data<jupyter_code>encoded_test_labels = pd.DataFrame(columns=['label_encoded'],dtype=np.int8,data=label_tokenizer.texts_to_sequences(kdd_test_labels['label']))
kdd_test_labels = pd.concat([kdd_test_labels, encoded_test_labels], axis=1)
kdd_test_labels.head()<jupyter_output><empty_output><jupyter_text>Remaining Column Encoding Besides the label, there are three more columns that need to be translated from text to integer data: __protocol_type__, __service__ and __flag__.  As these are done the same way, I define a function that fits a tokenizer, transforms the texts and appends the encoded column.  The plaintext columns are dropped, as this is needed for normalization. The regarding class indexes are written as JSON to *dataset_columnName_wordindex.json* along with the pickles.<jupyter_code># create one big dataframe for training the encoders
all_data = pd.concat([kdd_train_data, kdd_test_data])
def encode_column_to_int(dataframe, column, filename):
    # instantiate and fit a tokenizer on exactly the number of values present in the column
    col_encoder = Tokenizer(num_words=len(all_data[column].unique())+1, filters='')
    col_encoder.fit_on_texts(all_data[column].unique())
    
    
    print('Writing encoder data to file {}: {}'.format(filename, col_encoder.word_index))
    with open(filename, 'w') as outfile:
        json.dump(col_encoder.word_index, outfile)

    # then transform the column in question
    encoded_col = pd.DataFrame(columns=[column],dtype=np.int8,data=col_encoder.texts_to_sequences(dataframe[column]))
    
    #drop the source column
    dataframe.drop([column],axis=1,inplace=True)

    # and append the encoded one to the original dataframe
    dataframe = pd.concat([dataframe, encoded_col], axis=1)
    return dataframe
kdd_train_data = encode_column_to_int(kdd_train_data, 'protocol_type', os.path.join('NSL_KDD','kdd_train_data_protocol_type_wordindex.json'))
kdd_test_data = encode_column_to_int(kdd_test_data, 'protocol_type', os.path.join('NSL_KDD','kdd_test_data_protocol_type_wordindex.json'))
kdd_train_data = encode_column_to_int(kdd_train_data, 'service', os.path.join('NSL_KDD','kdd_train_data_service_wordindex.json'))
kdd_test_data = encode_column_to_int(kdd_test_data, 'service', os.path.join('NSL_KDD','kdd_test_data_service_wordindex.json'))
kdd_train_data = encode_column_to_int(kdd_train_data, 'flag', os.path.join('NSL_KDD','kdd_train_data_flag_wordindex.json'))
kdd_test_data = encode_column_to_int(kdd_test_data, 'flag', os.path.join('NSL_KDD','kdd_test_data_flag_wordindex.json'))
kdd_test_data.tail()<jupyter_output><empty_output><jupyter_text>Feature Standardization As the scaler converts the DataFrames to numpy arrays, save the header info to recreate a DataFrame afterwards.<jupyter_code>train_data_header = list(kdd_train_data.columns.values)
test_data_header = list(kdd_test_data.columns.values)<jupyter_output><empty_output><jupyter_text>As many ML implementations behave badly if confronted with non-scaled inputs, we go ahead and transform all features to center, then scale it.<jupyter_code>scaler = preprocessing.MinMaxScaler()
scaler.fit(kdd_train_data) # fit the scaler on the training data

# transform both samples without any refitting
kdd_train_data = scaler.transform(kdd_train_data)
kdd_test_data = scaler.transform(kdd_test_data)
kdd_train_data = pd.DataFrame(columns=train_data_header, data=kdd_train_data)
kdd_test_data = pd.DataFrame(columns=test_data_header, data=kdd_test_data)
kdd_test_data.head()<jupyter_output><empty_output><jupyter_text>Serialization So at this point, we have training and test sets with data and labels. The data parts are encoded and scaled, the encoded indizes are written away as json files.  It would be nice if this data could be used for future runs, right? Right!  That's why we serialize each dataframe into a python binary pickle on it's own (which is a feature directly supported by [Pandas](https://pandas.pydata.org/pandas-docs/stable/api.htmlid12) - nice, eh?)<jupyter_code>def write_to_pickle(dataframe, filename):
    dataframe.to_pickle(os.path.join('NSL_KDD', filename+'.pkl'))
write_to_pickle(kdd_train_data, 'kdd_train_data')
write_to_pickle(kdd_test_data, 'kdd_test_data')
write_to_pickle(kdd_train_labels, 'kdd_train_labels')
write_to_pickle(kdd_test_labels, 'kdd_test_labels')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11416216216216216, 'fraction_numerical': 0.009513513513513514, 'mean_word_length': 7.768720379146919, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '190884', 'license': 'Apache-2.0', 'n_tokens_mistral': 3316, 'n_tokens_neox': 3186, 'n_words': 749, 'path': 'Pickle-NSL-KDD.ipynb', 'repo_name': 'twseptian/Master_IPython', 'size': '9250'}