<jupyter_start><jupyter_text>Why are we doing this?* This dataset is real CAN logs * The given format is not right for direct input to models.* all ml experiments start with cleaning the dataset* we plan to spend 80% of time understanding the data vs 20% actually modelling and predicting, because of the complexity of the networks and the problem that we're working with.  Data PreparationAs we know, knowing our data and understanding how it changes is a key factor in successfully analysing it. In this notebook, we explore the CAN messages dataset that we got from Korea University researchers on the CAN of a Kia Soul being attacked using different kinds of known attacks.<jupyter_code>import os
import pandas as pd
df = pd.read_csv('/content/drive/My Drive/Colab Notebooks/CS581-TestResults/project-data/Attack_free_dataset.txt')
df.columns
df1 = pd.read_csv('/content/drive/My Drive/Colab Notebooks/CS581-TestResults/project-data/DoS_attack_dataset.txt')
df1<jupyter_output><empty_output><jupyter_text>this doesn't look right, let's try opening it as a normal text file and see what's up<jupyter_code>with open('/content/drive/My Drive/Colab Notebooks/CS581-TestResults/project-data/DoS_attack_dataset.txt','r') as file:
    for line in file.readlines(1080):
        print(line)<jupyter_output>Timestamp:          0.000000        ID: 0000    000    DLC: 8    00 00 00 00 00 00 00 00

Timestamp:          0.000271        ID: 0080    000    DLC: 8    00 17 dc 09 16 11 16 bb

Timestamp:          0.000495        ID: 0000    000    DLC: 8    00 00 00 00 00 00 00 00

Timestamp:          0.000736        ID: 0081    000    DLC: 8    40 84 87 00 00 00 00 6b

Timestamp:          0.000983        ID: 0000    000    DLC: 8    00 00 00 00 00 00 00 00

Timestamp:          0.001239        ID: 0165    000    DLC: 8    00 08 80 00 00 00 08 80

Timestamp:          0.001484        ID: 0000    000    DLC: 8    00 00 00 00 00 00 00 00

Timestamp:          0.001736        ID: 018f    000    DLC: 8    00 3f 16 00 00 3f 00 00

Timestamp:          0.001984        ID: 0000    000    DLC: 8    00 00 00 00 00 00 00 00

Timestamp:          0.002229        ID: 02a0    000    DLC: 8    62 00 87 9d bc 0c b7 02

Timestamp:          0.002465        ID: 0000    000    DLC: 8    00 00 00 00 00 00 00 00

Timestamp:[...]<jupyter_text>DoS attack Dataset<jupyter_code>#Read the file
count = len(open('/content/drive/My Drive/Colab Notebooks/CS581-TestResults/project-data/DoS_attack_dataset.txt','r').readlines())
count
dfdict = {}
#formatting data set 
with open('/content/drive/My Drive/Colab Notebooks/CS581-TestResults/project-data/DoS_attack_dataset.txt','r') as file:
    for line in file.readlines():
        line = line.split()
        line.remove('Timestamp:')
        line.remove('ID:')
        line.remove('DLC:')
        # if len(line) > 4:
        #     line[4] = "".join(line[4:]) #line[4:] #int(''.join(line[4:]),base=16)
        #     line = line[:5]            
        dfdict[float(line[0])] = line[1:]
        # print(line)<jupyter_output><empty_output><jupyter_text>I'll just clean it up in excel with a little reference from: https://ocslab.hksecurity.net/Dataset/CAN-intrusion-dataset<jupyter_code>#creating dataframe from dictionary dfdict
df = pd.DataFrame(dict([ (k,pd.Series(v)) for k,v in dfdict.items() ]))
df = df.T
df
df.dtypes
#set timestamp as index and name other columns
df.index = df.index.rename('Timestamp')
df.columns = ['CAN ID','RTR','DLC','Data1','Data2','Data3','Data4','Data5','Data6','Data7','Data8'] 
df.head(15)
df.dtypes
# replace NaNs with 0s
df = df.fillna('0')
#change can ids from hex to int, convert other fields to int as well
df['CAN ID'] = df['CAN ID'].apply(lambda x: int(x, 16))
df['DLC'] = df['DLC'].astype('int32')
df['RTR'] = df['RTR'].astype('int32')
b16 = lambda x: int(x,16)
for col in ['Data1','Data2','Data3','Data4','Data5','Data6','Data7','Data8']:
    df[col] = df[col].apply(b16)
print(df.dtypes)
df.head(20)
df.head(20)
# [int(x,base=16) for x in df['ID'][0.015:0.018]],list(df['ID'][0.015:0.018])
df.index.max()
df.to_csv('/content/drive/My Drive/Colab Notebooks/CS581-TestResults/project-data/DoS_attack_dataset.csv')
deviceids = df['CAN ID'].unique()
df.groupby('CAN ID').count()<jupyter_output><empty_output><jupyter_text>Fuzzy Attack Dataset Now exploring the format of the other datasets, we see that we have a uniform formatting, and thus can apply the same transformations to get a csv file that we can use for further processing. We have a nice pipeline established.<jupyter_code>#read the file
count = len(open('/content/drive/My Drive/Colab Notebooks/CS581-TestResults/project-data/Fuzzy_attack_dataset.txt','r').readlines())
count
dfdict2 = {}
#remove the old format and put into dictionary 
with open('/content/drive/My Drive/Colab Notebooks/CS581-TestResults/project-data/Fuzzy_attack_dataset.txt','r') as file:
    for line in file.readlines():
        line = line.split()
        line.remove('Timestamp:')
        line.remove('ID:')
        line.remove('DLC:')
        # if len(line) > 4:
        #     line[4] = line[4:]
        #     line = line[:5]            
        dfdict2[float(line[0])] = line[1:]
        # print(line)
list(dfdict2.values())[:10]
#create df from dictionary
df2 = pd.DataFrame(dict([ (k,pd.Series(v)) for k,v in dfdict2.items() ]))
df2 = df2.T
df2 = df2.iloc[:,:-1]
df2
#set index and column names
# df2.set_index('Timestamp')
df2.columns = ['CAN ID','RTR','DLC','Data1','Data2','Data3','Data4','Data5','Data6','Data7','Data8'] 
df2.index = df2.index.rename('Timestamp')
df2
# replace NaNs with 0s
df2 = df2.fillna('0')
print(df2.head(20))
#convert hex strings and objects to ints
print(df2.dtypes)
#change datatypes from hex to int
df2['CAN ID'] = df2['CAN ID'].apply(lambda x: int(x, 16))
df2['DLC'] = df2['DLC'].astype('int32')
df2['RTR'] = df2['RTR'].astype('int32')
b16 = lambda x: int(x,16)
for col in ['Data1','Data2','Data3','Data4','Data5','Data6','Data7','Data8']:
    df2[col] = df2[col].apply(b16)
print(df2.dtypes)
df2.head(20)
df2.index.max()
df2.to_csv('/content/drive/My Drive/Colab Notebooks/CS581-TestResults/project-data/Fuzzy_attack_dataset.csv')
deviceids = df2['CAN ID'].unique()
df2.groupby('CAN \
ID').count()<jupyter_output><empty_output><jupyter_text>Attack Free Dataset<jupyter_code>count = len(open('/content/drive/My Drive/Colab Notebooks/CS581-TestResults/project-data/Attack_free_dataset.txt','r').readlines())
count
dfdict3 = {}
with open('/content/drive/My Drive/Colab Notebooks/CS581-TestResults/project-data/Attack_free_dataset.txt','r') as file:
    for line in file.readlines():
        line = line.split()
        line.remove('Timestamp:')
        line.remove('ID:')
        line.remove('DLC:')
        # if len(line) > 4:
        #     line[4] = line[4:]
        #     line = line[:5]            
        dfdict3[float(line[0])] = line[1:]
        # print(line)
df3 = pd.DataFrame(dict([ (k,pd.Series(v)) for k,v in dfdict3.items() ]))
df3 = df3.T
df3
df3.index = df3.index.rename('Timestamp')
df3.columns = ['CAN ID','RTR','DLC','Data1','Data2','Data3','Data4','Data5','Data6','Data7','Data8'] 
# df3.set_index('Timestamp')
df3
# replace NaNs with 0s
df3 = df3.fillna('0')
print(df3.head(20))
#convert hex strings and objects to ints
print(df3.dtypes)
df3['CAN ID'] = df3['CAN ID'].apply(lambda x: int(x, 16))
df3['DLC'] = df3['DLC'].astype('int32')
df3['RTR'] = df3['RTR'].astype('int32')
b16 = lambda x: int(x,16)
for col in ['Data1','Data2','Data3','Data4','Data5','Data6','Data7','Data8']:
    df3[col] = df3[col].apply(b16)
print(df3.dtypes)
df3.head(20)
df3.index.max()
df3.to_csv('/content/drive/My Drive/Colab Notebooks/CS581-TestResults/project-data/Attack_free_dataset.csv')<jupyter_output><empty_output><jupyter_text>Impersonation Attack Dataset<jupyter_code>count = len(open('/content/drive/My Drive/Colab Notebooks/CS581-TestResults/project-data/Impersonation_attack_dataset.txt','r').readlines())
count
dfdict4 = {}
with open('/content/drive/My Drive/Colab Notebooks/CS581-TestResults/project-data/Impersonation_attack_dataset.txt','r') as file:
    for line in file.readlines():
        line = line.split()
        line.remove('Timestamp:')
        line.remove('ID:')
        line.remove('DLC:')
        # if len(line) > 4:
        #     line[4] = line[4:]
        #     line = line[:5]            
        dfdict4[float(line[0])] = line[1:]
        # print(line)
df4 = pd.DataFrame(dict([ (k,pd.Series(v)) for k,v in dfdict4.items() ]))
df4 = df4.T
df4.columns = ['CAN ID','RTR','DLC','Data1','Data2','Data3','Data4','Data5','Data6','Data7','Data8'] 
df4.index = df4.index.rename('Timestamp')
df4
df4.index.max()
# replace NaNs with 0s
df4 = df4.fillna('0')
print(df4.head(20))
#convert hex strings and objects to ints
print(df4.dtypes)
df4['CAN ID'] = df4['CAN ID'].apply(lambda x: int(x, 16))
df4['DLC'] = df4['DLC'].astype('int32')
df4['RTR'] = df4['RTR'].astype('int32')
b16 = lambda x: int(x,16)
for col in ['Data1','Data2','Data3','Data4','Data5','Data6','Data7','Data8']:
    df4[col] = df4[col].apply(b16)
print(df4.dtypes)
df4.head(20)
df4.to_csv('/content/drive/My Drive/Colab Notebooks/CS581-TestResults/project-data/Impersonation_attack_dataset.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15881590319792566, 'fraction_numerical': 0.07087294727744166, 'mean_word_length': 3.791407867494824, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 32, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '593030', 'license': 'MIT', 'n_tokens_mistral': 3714, 'n_tokens_neox': 3327, 'n_words': 803, 'path': 'code/Data_Preparation.ipynb', 'repo_name': 'mehrotrasan16/CS581-CAN-DO-Project', 'size': '9256'}