<jupyter_start><jupyter_text>1. Loading data.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>1.1. Firstly we load the data and filter the columns.<jupyter_code>df = pd.read_csv("/home/alberto/Documentos/MatchingLearning/Practicas/Moriarty2.csv",
                  usecols=["UUID","ActionType"])

df2 = pd.read_csv("/home/alberto/Documentos/MatchingLearning/Practicas/T4.csv", 
                  usecols=["UUID", "CPU_0", "CPU_1", "CPU_2", "CPU_3", 
                           "Traffic_TotalRxBytes", "Traffic_TotalTxBytes", "MemFree"])<jupyter_output><empty_output><jupyter_text>1.2. Merging the two datasets. The first thing we need to do it is to convert column 'UUID' (which is a timestamp in milliseconds) into a date timestamp.<jupyter_code>df['UUID'] = pd.to_datetime(df['UUID'], unit="ms")
df['UUID'] = df['UUID'].dt.round('t')


df2['UUID'] = pd.to_datetime(df['UUID'], unit="ms")
df2['UUID'] = df['UUID'].dt.round('t')<jupyter_output><empty_output><jupyter_text>Merge the two datasheets on the column to set the desired combinations.<jupyter_code>data = pd.merge(df,df2, on=['UUID'])<jupyter_output><empty_output><jupyter_text>1.3. Replace ActionType. We need numeric values in the columns, so we replace ActionType malicious/benign by 1/0 respectively. And finally we don't need column 'UUID' for the predicting model so we remove it.<jupyter_code>data['ActionType'] = data['ActionType'].replace(['malicious'], 1)
data['ActionType'] = data['ActionType'].replace(['benign'], 0)
data = data.drop('UUID', 1)<jupyter_output><empty_output><jupyter_text>2. PCA.  2.1. Scalation.<jupyter_code>from sklearn import preprocessing
scaler = preprocessing.StandardScaler()
datanorm = scaler.fit_transform(data)<jupyter_output><empty_output><jupyter_text>2.2. Modelling (PCA).<jupyter_code>from sklearn.decomposition import PCA
n_components = 2
estimator = PCA(n_components)
X_pca = estimator.fit_transform(datanorm)<jupyter_output><empty_output><jupyter_text>3. KNN Classification.<jupyter_code>x = X_pca
y = data['ActionType']
from sklearn import neighbors
from matplotlib.colors import ListedColormap

h = .02  # step size in the mesh

# Create color maps
cmap_light = ListedColormap(['#FFAAAA', '#AAFFAA', '#AAAAFF'])
cmap_bold = ListedColormap(['#FF0000', '#00FF00', '#0000FF'])
for weights in ['uniform', 'distance']:
    n_neighbors = 15
    # Instance of Neighbours Classifier and fit the data.
    clf = neighbors.KNeighborsClassifier(n_neighbors, weights=weights)
    clf.fit(x,y)
    
    # Plot the decision boundary. For that, we will assign a color to each
    # point in the mesh [x_min, x_max]x[y_min, y_max].
    x_min, x_max = x[:, 0].min() - 1, x[:, 0].max() + 1
    y_min, y_max = x[:, 1].min() - 1, x[:, 1].max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))
    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
    
    # Put the result into a color plot
    Z = Z.reshape(xx.shape)
    plt.figure()
    plt.pcolormesh(xx, yy, Z, cmap=cmap_light)

    # Plot also the training points
    plt.scatter(x[:, 0], x[:, 1], c=y, cmap=cmap_bold,
                edgecolor='k', s=20)
    plt.xlim(xx.min(), xx.max())
    plt.ylim(yy.min(), yy.max())
    plt.title("3-Class classification (k = %i, weights = '%s')"
              % (n_neighbors, weights))
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15118978525827045, 'fraction_numerical': 0.016831108531630876, 'mean_word_length': 4.480127186009539, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '616286', 'license': 'MIT', 'n_tokens_mistral': 1296, 'n_tokens_neox': 1254, 'n_words': 282, 'path': 'notebooks/MILESTONE 2/1.0-ass-pca-and-KNN_classification.ipynb', 'repo_name': 'itSQualL/machine-learning-techinques', 'size': '3446'}