<jupyter_start><jupyter_text>**Imports and configurations**<jupyter_code>%reload_ext autoreload
%autoreload 2
import os
import time

import pandas as pd
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import plotly.graph_objs as go
import plotly.express as px

from IPython.display import IFrame
from plotly.offline import init_notebook_mode
init_notebook_mode(connected=True)
from helpers import *
from learners import *
from plot_utils import *<jupyter_output><empty_output><jupyter_text>Data loading :<jupyter_code>folder_path = 'C:/Users/yaass/OneDrive/Bureau/Parser/ransom_datasets'

files = get_file_list(folder_path, extensions=['gz'])

datasets = prepare_datasets(datasets_path = folder_path)<jupyter_output>apistats_counts
loaded data shape :   (2618, 301)
fileops_created_nested_files
loaded data shape :   (1232, 9747)
fileops_deleted_nested_files
loaded data shape :   (1029, 5537)
fileops_exists_nested_files
loaded data shape :   (1545, 6847)
fileops_failed_nested_files
loaded data shape :   (1300, 3073)
fileops_opened_nested_files
loaded data shape :   (2094, 12066)
fileops_read_nested_files
loaded data shape :   (1508, 10430)
fileops_recreated_nested_files
loaded data shape :   (539, 445)
fileops_summary
loaded data shape :   (2392, 12)
fileops_written_nested_files
loaded data shape :   (1265, 9891)
loaded_dll_onehot
loaded data shape :   (2168, 821)
pe_entropy_analysis
loaded data shape :   (3051, 715)
pe_imports_advapi32
loaded data shape :   (1532, 455)
pe_imports_comctl32
loaded data shape :   (598, 92)
pe_imports_gdi32
loaded data shape :   (883, 541)
pe_imports_kernel32
loaded data shape :   (2425, 886)
pe_imports_libraries
loaded data shape :   (3058, 286)
pe_imports_msvcrt
load[...]<jupyter_text>**Models evaluation**  1. apistats (count encoded) In this section, on top of studying apistats, we'll also compare the performance of different SMOTE techniques to decide which gives the highest performance.<jupyter_code>datasets['apistats_counts'].head()
class_distribution(datasets['apistats_counts'], label='sublabel')
start = time.time()

#create a smote pipeline
X, y = get_X_y(datasets['apistats_counts'], label='sublabel')
smoter = smote(X, y, category='adaptive', over_strategy=0.5, under_strategy=0.8, k_neighbors=5, fit=False)

#compute cross-validation scores
names, results = get_evaluation_results(df = datasets['apistats_counts'], 
                                        label='sublabel', 
                                        smoter=smoter, 
                                        scaler=MinMaxScaler(),
                                        scoring ='accuracy')

print(f'Cross-validation {len(names)} models required a duration of {time.time()-start:.2f} seconds')
#standard smote
print_evaluation_results(results, names)
#bordeline smote
print_evaluation_results(results, names)
#svm smote
print_evaluation_results(results, names)
#adaptive smote
print_evaluation_results(results, names)<jupyter_output>LogisticRegression             	: 92.734% ( (+/-) 1.438% )
KNN                            	: 94.466% ( (+/-) 1.365% )
Decision tree                  	: 95.785% ( (+/-) 1.346% )
Random Forest                  	: 98.084% ( (+/-) 0.900% )
SVM                            	: 92.366% ( (+/-) 1.182% )
RegularNets                    	: 86.998% ( (+/-) 3.656% )
LDA                            	: 87.380% ( (+/-) 2.593% )
Gaussian Naive Bayes           	: 60.115% ( (+/-) 2.599% )<jupyter_text>**Conclusion** : The adaptive SMOTE ADASYN gives the highest accuracy with minimum variability.<jupyter_code>fig = plot_evaluation_boxplots(results, names, title = 'Models Performance On Count Encoded "apistats" Data', y_axis = 'Accuracy')

figure_path = 'figures/ransomware/evaluation_count_apistats.html'

save_figures_to_html(figure_path, [fig])

IFrame(figure_path, width=900, height=600)<jupyter_output><empty_output><jupyter_text>2. dll_loaded (one-hot encoded)<jupyter_code>df = datasets['loaded_dll_onehot']
class_distribution(df, label='sublabel')
df.head()
start = time.time()

names, results = get_evaluation_results(df, label='sublabel')

print(f'Cross-validation {len(names)} models required a duration of {time.time()-start:.2f} seconds')
start = time.time()

#create a smote pipeline
X, y = get_X_y(df, label='sublabel')

smote_params = {
    'category' : 'adaptive',
    'over_strategy' : 0.5,
    'under_strategy' : 0.8,
    'k_neighbors' : 5,
}

smoter = smote(X, y, **smote_params, fit=False)

print_new_distribution(X, y, smote_params)


#compute cross-validation scores
names, results = get_evaluation_results(df = df, 
                                        label='sublabel', 
                                        smoter=smoter, 
                                        scaler=MinMaxScaler(),
                                        scoring ='accuracy')

print(f'Cross-validation {len(names)} models required a duration of {time.time()-start:.2f} seconds')
print_evaluation_results(results, names)
fig = plot_evaluation_boxplots(results, names, title = 'Models Performance On One-Hot Encoded "dll_loaded" Data', y_axis = 'Accuracy')

figure_path = 'figures/ransomware/evaluation_onehot_dll_data.html'

save_figures_to_html(figure_path, [fig])

IFrame(figure_path, width=900, height=600)<jupyter_output><empty_output><jupyter_text>3. File operations  In this section we'll study the summary information about file operations. As for the specific nested file operation of each category, as they are composed of a number of features in the order of 10k, we'll study them in the feature selection notebook, in order to perform variable selection before training. **File Operations (Counts Summary)**<jupyter_code>df = datasets['fileops_summary']
class_distribution(df, label='sublabel')
df.head()
start = time.time()

#create a smote pipeline
X, y = get_X_y(df, label='sublabel')

smote_params = {
    'category' : 'adaptive',
    'over_strategy' : 0.5,
    'under_strategy' : 0.8,
    'k_neighbors' : 5,
}

smoter = smote(X, y, **smote_params, fit=False)

print_new_distribution(X, y, smote_params)


#compute cross-validation scores
names, results = get_evaluation_results(df = df, 
                                        label='sublabel', 
                                        smoter=smoter, 
                                        scaler=MinMaxScaler(),
                                        scoring ='accuracy')

print(f'Cross-validation {len(names)} models required a duration of {time.time()-start:.2f} seconds')
print_evaluation_results(results, names)
fig = plot_evaluation_boxplots(results, names, title = 'Models Performance On File Operation Counts Data', y_axis = 'Accuracy')

figure_path = 'figures/ransomware/evaluation_file_operation_counts.html'

save_figures_to_html(figure_path, [fig])

IFrame(figure_path, width=900, height=600)<jupyter_output><empty_output><jupyter_text>4. Registry Key Operations In this section we'll study the summary information about registry key operations. As for the specific nested registry keys of each category, as they are composed of a number of features in the order of 10k, we'll study them in the feature selection notebook, in order to perform variable selection before training. **Registry Key Operations (Counts Summary)**<jupyter_code>df = datasets['regkeys_summary']
class_distribution(df, label='sublabel')
df.head()
start = time.time()

#create a smote pipeline
X, y = get_X_y(df, label='sublabel')

smote_params = {
    'category' : 'adaptive',
    'over_strategy' : 0.5,
    'under_strategy' : 0.8,
    'k_neighbors' : 5,
}

smoter = smote(X, y, **smote_params, fit=False)

print_new_distribution(X, y, smote_params)


#compute cross-validation scores
names, results = get_evaluation_results(df = df, 
                                        label='sublabel', 
                                        smoter=smoter, 
                                        scaler=MinMaxScaler(),
                                        scoring ='accuracy')

print(f'Cross-validation {len(names)} models required a duration of {time.time()-start:.2f} seconds')
print_evaluation_results(results, names)
fig = plot_evaluation_boxplots(results, names, title = 'Models Performance On RegKeys Operation Counts Data', y_axis = 'Accuracy')

figure_path = 'figures/ransomware/evaluation_registry_key_counts.html'

save_figures_to_html(figure_path, [fig])

IFrame(figure_path, width=900, height=600)<jupyter_output><empty_output><jupyter_text>5. PE Entropy<jupyter_code>df = datasets['pe_entropy_analysis']
class_distribution(df, label='sublabel')
df.head()
start = time.time()

#create a smote pipeline
X, y = get_X_y(df, label='sublabel')

smote_params = {
    'category' : 'adaptive',
    'over_strategy' : 0.5,
    'under_strategy' : 0.8,
    'k_neighbors' : 5,
}

smoter = smote(X, y, **smote_params, fit=False)

print_new_distribution(X, y, smote_params)


#compute cross-validation scores
names, results = get_evaluation_results(df = df, 
                                        label='sublabel', 
                                        smoter=smoter, 
                                        scaler=MinMaxScaler(),
                                        scoring ='accuracy')

print(f'Cross-validation {len(names)} models required a duration of {time.time()-start:.2f} seconds')
print_evaluation_results(results, names)
fig = plot_evaluation_boxplots(results, names, title = 'Models Performance On PE Entropy Data', y_axis = 'Accuracy')

figure_path = 'figures/ransomware/evaluation_pe_entropy.html'

save_figures_to_html(figure_path, [fig])

IFrame(figure_path, width=900, height=600)<jupyter_output><empty_output><jupyter_text>6. PE Imports (One-Hot Encoded) Considering all pe import functions at once ends up generating a dataset that has over 20 000 columns. So we will analyse imports belonging to different libraries (e.g. kernel32.dll, user32.dll...) separately and try to select either the ones with the highest feature importance in the most recurrent libraries. For now, let's first study the performance on training on just the libraries, before checking the individual pe imports. **Libraries**<jupyter_code>df = datasets['pe_imports_libraries']
class_distribution(df, label='sublabel')
df.head()
start = time.time()

#create a smote pipeline
X, y = get_X_y(df, label='sublabel')

smote_params = {
    'category' : 'adaptive',
    'over_strategy' : 0.5,
    'under_strategy' : 0.8,
    'k_neighbors' : 5,
}

smoter = smote(X, y, **smote_params, fit=False)

print_new_distribution(X, y, smote_params)


#compute cross-validation scores
names, results = get_evaluation_results(df = df, 
                                        label='sublabel', 
                                        smoter=smoter, 
                                        scaler=MinMaxScaler(),
                                        scoring ='accuracy')

print(f'Cross-validation {len(names)} models required a duration of {time.time()-start:.2f} seconds')
print_evaluation_results(results, names)
fig = plot_evaluation_boxplots(results, names, title = 'Models Performance On PE Entropy Libraries Data', y_axis = 'Accuracy')

figure_path = 'figures/ransomware/evaluation_pe_imports.html'

save_figures_to_html(figure_path, [fig])

IFrame(figure_path, width=900, height=600)<jupyter_output><empty_output><jupyter_text>**Selecting Libraries**<jupyter_code>from learners import *
X, y = get_X_y(df, label='sublabel')
smote_params = {
    'category' : 'adaptive',
    'over_strategy' : 0.5,
    'under_strategy' : 0.8,
    'k_neighbors' : 5,
    }
smoter = smote(X, y, **smote_params, fit=False)
k_best_selected_libraries = select_k_best(df, select_function=chi2, k=10, label='sublabel', smoter=smoter)
k_best_selected_libraries
limit= 15
most_recurrent_libraries = get_most_recurrent(df, portion=0.05, limit=limit)
most_recurrent_libraries
#Retrieve the most recurrent libraries in PE imports across all data samples
selected_libraries_indices = []
for column in k_best_selected_libraries:
    if column in most_recurrent_libraries.index.tolist():
        selected_libraries_indices.append(most_recurrent_libraries.index.tolist().index(column))

#Plot
colors = np.repeat('lightslategray', limit)
colors[selected_libraries_indices] = 'lightblue'

fig = go.Figure( data = [ go.Bar(x=most_recurrent_libraries.index, 
                                 y=most_recurrent_libraries.values,  
                                 marker_color=colors) ] )

fig.update_layout(title='The top 5% most recurrent libraries in PE imports', title_x=0.3)

fig.show()<jupyter_output><empty_output><jupyter_text>Let's explore what's the majority class in function of the involved libraries in PE imports:<jupyter_code>def plot_class_distribution(df, libraries, title, label='sublabel'):
    
    malware = []
    ransomware = []
    
    #Balancing:
    X, y = get_X_y(df, label=label)
    smote_params = {
    'category' : 'adaptive',
    'over_strategy' : 0.5,
    'under_strategy' : 0.8,
    'k_neighbors' : 5,
    }
    X, y = smote(X, y, **smote_params, fit=True)
    X['sublabel'] = y
    df = X.copy()
    
    for library in libraries:
        subset = df[df[library] != 0]
        total = len(subset)
        malware.append(subset[label].sum())
        ransomware.append(subset[label].sum() - total)

    fig = go.Figure(data=[
        go.Bar(name='Malware', x=libraries, y=malware, marker_color=np.repeat('crimson', len(libraries))),
        go.Bar(name='Ransomware', x=libraries, y=ransomware, marker_color=np.repeat('lightslategray', len(libraries))) ])
    
    fig.update_layout(barmode='group', 
                      title=title, 
                      title_x=0.2)
    fig.show()
#Most recurrent libraries: 
plot_class_distribution(df = df,
                        libraries = most_recurrent_libraries.iloc[:10].index.tolist(),
                        title = 'The class distribution for the most recurrent libraries in PE imports')
#Most recurrent libraries: 
plot_class_distribution(df = df,
                        libraries = k_best_selected_libraries,
                        title = 'The class distribution for the k-selected libraries in PE imports')<jupyter_output><empty_output><jupyter_text>**kernel32.dll** We can look at the performance on PE Imports of the most recurrent library kernel32.dll as an example. As for our final model, We'll pick the data from the 8 most recurrent libraries as they will provide enough observation data for training.<jupyter_code>df = datasets['pe_imports_kernel32']
class_distribution(df, label='sublabel')
df.head()
start = time.time()

#create a smote pipeline
X, y = get_X_y(df, label='sublabel')

smote_params = {
    'category' : 'adaptive',
    'over_strategy' : 0.5,
    'under_strategy' : 0.8,
    'k_neighbors' : 5,
}

smoter = smote(X, y, **smote_params, fit=False)

print_new_distribution(X, y, smote_params)


#compute cross-validation scores
names, results = get_evaluation_results(df = df, 
                                        label='sublabel', 
                                        smoter=smoter, 
                                        scaler=MinMaxScaler(),
                                        scoring ='accuracy')

print(f'Cross-validation {len(names)} models required a duration of {time.time()-start:.2f} seconds')
print_evaluation_results(results, names)
fig = plot_evaluation_boxplots(results, names, title = 'Models Performance On PE Imports of kernel32.dll library Data', y_axis = 'Accuracy')

figure_path = 'figures/ransomware/evaluation_pe_imports.html'

save_figures_to_html(figure_path, [fig])

IFrame(figure_path, width=900, height=600)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.107161092672964, 'fraction_numerical': 0.021891753893285678, 'mean_word_length': 3.4704707560627677, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 30, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '437588', 'license': 'CC0-1.0', 'n_tokens_mistral': 5217, 'n_tokens_neox': 4827, 'n_words': 1192, 'path': 'model-building/initial-model-selection-ransomware.ipynb', 'repo_name': 'DSTI-A20-RW/ransomware-preencryption-detector', 'size': '15668'}