<jupyter_start><jupyter_text>Asset Classification (Supervised)  Authors- <NAME> (NVIDIA)- <NAME> (NVIDIA)  Table of Contents * Introduction* Dataset* Reading in the datasets* Training and inference* References  Introduction In this notebook we will show how to predict the function of a server with Windows Event Logs using `cudf`, `cuml` and `pytorch`. The machines are labeled as DC, SQL, WEB, DHCP, MAIL and SAP. The dependent variable will be the type of the machine. The features are selected from Windows Event Logs which is in a tabular format.  Library imports<jupyter_code>import cudf
import torch
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as torch_optim
from torch.utils.dlpack import from_dlpack
from cuml.preprocessing import LabelEncoder
from cuml.preprocessing import train_test_split
from sklearn.metrics import accuracy_score, f1_score, confusion_matrix
import pandas as pd
batch_size = 10000
label_col = '19'
epochs = 15<jupyter_output><empty_output><jupyter_text>Read the dataset into a GPU dataframe with `cudf.read_csv()`<jupyter_code>win_events_on_gpu = cudf.read_csv('win_events_18_features.csv')
features = {
    "1" : "eventcode",
    "2" : "keywords",
    "3" : "privileges",
    "4" : "message",
    "5" : "sourcename", 
    "6" : "taskcategory",
    "7" : "account_for_which_logon_failed_account_domain",
    "8" : "detailed_authentication_information_authentication_package",
    "9" : "detailed_authentication_information_key_length",
    "10" : "detailed_authentication_information_logon_process",
    "11" : "detailed_authentication_information_package_name_ntlm_only",
    "12" : "logon_type",
    "13" : "network_information_workstation_name",
    "14" : "new_logon_security_id",
    "15" : "impersonation_level",
    "16" : "network_information_protocol",
    "17" : "network_information_direction",
    "18" : "filter_information_layer_name"
}
win_events_on_gpu.head()<jupyter_output><empty_output><jupyter_text>Categorize the Labels<jupyter_code>for col in win_events_on_gpu.columns:
    win_events_on_gpu[col] = win_events_on_gpu[col].astype('str')
    win_events_on_gpu[col] = win_events_on_gpu[col].fillna("NA")
    win_events_on_gpu[col] = LabelEncoder().fit_transform(win_events_on_gpu[col])
for col in win_events_on_gpu.columns:
    win_events_on_gpu[col] = win_events_on_gpu[col].astype('int16')
win_events_on_gpu.to_csv("test12")<jupyter_output><empty_output><jupyter_text>Split the dataset into training and validation sets using cuML `train_test_split` functionColumn 19 is the dependent variable hence we save it as labels.<jupyter_code>X, val_X, Y, val_Y = train_test_split(win_events_on_gpu, label_col, train_size=0.9)
val_X.index = val_Y.index
val_X.head()
val_Y.head()
X, test_X, Y, test_Y = train_test_split(X,Y, train_size=0.9)
X.index = Y.index
test_X.index = test_Y.index
X.head()
Y.head()<jupyter_output><empty_output><jupyter_text>Print Labels<jupyter_code>test_Y.head()<jupyter_output><empty_output><jupyter_text>Embedding columns, check values in the columns<jupyter_code>embedded_cols = {}
for col in X.columns:
    catergories_cnt = X[col].max()+2
    if catergories_cnt > 1:
        embedded_cols[col] = catergories_cnt
embedded_cols
X[label_col] = Y
val_X[label_col] = val_Y
test_X[label_col] = test_Y<jupyter_output><empty_output><jupyter_text>Choosing columns for embedding<jupyter_code>embedded_col_names = embedded_cols.keys()
embedded_col_names 
embedded_cols.items()
embedding_sizes = [(n_categories, min(100, (n_categories+1)//2)) for _,n_categories in embedded_cols.items()]
embedding_sizes<jupyter_output><empty_output><jupyter_text>Partition the dataframe<jupyter_code>def get_partitioned_dfs(df, batch_size):
    dataset_len = df.shape[0]
    prev_chunk_offset = 0
    partitioned_dfs = []
    while prev_chunk_offset < dataset_len:
        curr_chunk_offset = prev_chunk_offset + batch_size
        chunk = df.iloc[prev_chunk_offset:curr_chunk_offset:1]
        partitioned_dfs.append(chunk)
        prev_chunk_offset = curr_chunk_offset
    return partitioned_dfs
train_part_dfs = get_partitioned_dfs(X, batch_size)
val_part_dfs = get_partitioned_dfs(val_X, batch_size)
test_part_dfs = get_partitioned_dfs(test_X, batch_size)
train_part_dfs[0].head()
del win_events_on_gpu
del X
del val_X
del test_X<jupyter_output><empty_output><jupyter_text>Check GPU availabilityIf there's a GPU, data is moved on to the GPU.<jupyter_code>def get_default_device():
    """Pick GPU if available, else CPU"""
    if torch.cuda.is_available():
        return torch.device('cuda')
    else:
        return torch.device('cpu')
def to_device(data, device):
    """Move tensor(s) to chosen device"""
    if isinstance(data, (list,tuple)):
        return [to_device(x, device) for x in data]
    return data.to(device, non_blocking=True)
device = get_default_device()
device
def bn_drop_lin(n_in, n_out, bn, p, actn):
    "Sequence of batchnorm (if `bn`), dropout (with `p`) and linear (`n_in`,`n_out`) layers followed by `actn`."
    layers = [nn.BatchNorm1d(n_in)] if bn else []
    if p != 0: layers.append(nn.Dropout(p))
    layers.append(nn.Linear(n_in, n_out))
    if actn is not None: layers.append(actn)
    return layers<jupyter_output><empty_output><jupyter_text>The bn_drop_lin function returns a sequence of batch normalization, dropout and a linear layer. This custom layer is usually used at the end of a model.n_in represents the size of the input, n_out the size of the output, bn whether we want batch norm or not, p how much dropout, and actn (optional parameter) adds an activation function at the end.Reference: https://github.com/fastai/fastai/blob/master/fastai/layers.pyL44  Define Model Class<jupyter_code>class TabularModel(nn.Module):
    "Basic model for tabular data"
    
    def __init__(self, emb_szs, n_cont, out_sz, layers, drops, 
                 emb_drop, use_bn, is_reg, is_multi):
        super().__init__()
        
        self.embeds = nn.ModuleList([nn.Embedding(ni, nf) for ni,nf in emb_szs])
        self.emb_drop = nn.Dropout(emb_drop)
        self.bn_cont = nn.BatchNorm1d(n_cont)
        n_emb = sum(e.embedding_dim for e in self.embeds)
        self.n_emb,self.n_cont = n_emb,n_cont
        sizes = [n_emb + n_cont] + layers + [out_sz]
        actns = [nn.ReLU(inplace=True)] * (len(sizes)-2) + [None]
        layers = []
        for i,(n_in,n_out,dp,act) in enumerate(zip(sizes[:-1],sizes[1:],[0.]+drops,actns)):
            layers += bn_drop_lin(n_in, n_out, bn=use_bn and i!=0, p=dp, actn=act)
        self.layers = nn.Sequential(*layers)
    
    def forward(self, x_cat, x_cont):
        if self.n_emb != 0:
                x = [e(x_cat[:,i]) for i,e in enumerate(model.embeds)]
                x = torch.cat(x, 1)
                x = self.emb_drop(x)
        if self.n_cont != 0:
            x_cont = self.bn_cont(x_cont) 
            x = torch.cat([x, x_cont], 1) if self.n_emb != 0 else x_cont
        x = self.layers(x)
        return x.squeeze()
model = TabularModel(embedding_sizes, 0, 6, [200,100], [0.001,0.01], emb_drop=0.04, is_reg=False,is_multi=True, use_bn=True)
to_device(model, device)<jupyter_output><empty_output><jupyter_text>Define the Optimizer<jupyter_code>def get_optimizer(model, lr = 0.001, wd = 0.0):
    parameters = filter(lambda p: p.requires_grad, model.parameters())
    optim = torch_optim.Adam(parameters, lr=lr, weight_decay=wd)
    return optim<jupyter_output><empty_output><jupyter_text>Training function<jupyter_code>def train_model(model, optim, dfs):
    model.train()
    total = 0
    sum_loss = 0
    
    xb_cont_tensor = torch.zeros(0, 0)
    xb_cont_tensor.cuda()
    
    for df in dfs:
        batch = df.shape[0] 
        train_set = df.drop(label_col).to_dlpack()
        train_set = from_dlpack(train_set).long()
        
        output = model(train_set, xb_cont_tensor)
        train_label = df[label_col].to_dlpack()
        train_label = from_dlpack(train_label).long()
        
        loss = F.cross_entropy(output, train_label)   
        
        optim.zero_grad()
        loss.backward()
        optim.step()
        total += batch
        sum_loss += batch*(loss.item())
        
    return sum_loss/total<jupyter_output><empty_output><jupyter_text>Evaluation function<jupyter_code>def val_loss(model, dfs):
    model.eval()
    total = 0
    sum_loss = 0
    correct = 0
    
    xb_cont_tensor = torch.zeros(0, 0)
    xb_cont_tensor.cuda()
    
    for df in dfs:
        current_batch_size = df.shape[0]
        
        val = df.drop(label_col).to_dlpack()
        val = from_dlpack(val).long()
        
        out = model(val, xb_cont_tensor)
        
        val_label = df[label_col].to_dlpack()
        val_label = from_dlpack(val_label).long()
        
        loss = F.cross_entropy(out, val_label)
        sum_loss += current_batch_size*(loss.item())
        total += current_batch_size
        
        pred = torch.max(out, 1)[1]
        correct += (pred == val_label).float().sum().item()
    print("valid loss %.3f and accuracy %.3f" % (sum_loss/total, correct/total))
    
    return sum_loss/total, correct/total<jupyter_output><empty_output><jupyter_text>Prediction function to be used with a test set<jupyter_code>def predict(model, test_set):
    xb_cont_tensor = torch.zeros(0, 0)
    xb_cont_tensor.cuda()
    
    current_batch_size = test_set.shape[0]

    test_set = test_set.to_dlpack()
    test_set = from_dlpack(test_set).long()
    
    out = model(test_set, xb_cont_tensor)
    pred = torch.max(out, 1)[1].view(-1).tolist()
    return pred
def train_loop(model, epochs, lr=0.01, wd=0.0):
    optim = get_optimizer(model, lr = lr, wd = wd)
    for i in range(epochs): 
        loss = train_model(model, optim, train_part_dfs)
        print("training loss: ", loss)
        val_loss(model, val_part_dfs)
def cleanup_cache():
    # release memory.
    torch.cuda.empty_cache()<jupyter_output><empty_output><jupyter_text>Training<jupyter_code>train_loop(model, epochs=epochs, lr=0.03, wd=0.00001)
def inference(model, test_part_dfs):
    pred_results = []
    true_results = []
    for df in test_part_dfs:
        pred_results.append(predict(model, df))
        true_results.append(df[label_col].values_host)                           
    pred_results = np.concatenate(pred_results).astype(np.int32)
    true_results = np.concatenate(true_results)
    f1_score_ = f1_score(pred_results, true_results,average='micro')
    print('micro F1 score: %s'%(f1_score_))
    return true_results, pred_results
true_results, pred_results = inference(model, test_part_dfs)
cleanup_cache()
labels = ["DC","DHCP","MAIL","SAP","SQL","WEB"]
a = confusion_matrix(true_results, pred_results)
pd.DataFrame(a, index=labels, columns=labels)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11414805892708237, 'fraction_numerical': 0.012971370332622996, 'mean_word_length': 4.069985908877407, 'pattern_counts': {'":': 0, '<': 64, '<?xml version=': 0, '>': 64, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '495670', 'license': 'Apache-2.0', 'n_tokens_mistral': 3949, 'n_tokens_neox': 3836, 'n_words': 795, 'path': 'notebooks/network_mapping/Asset_Classification_Supervised.ipynb', 'repo_name': 'gbatmaz/clx', 'size': '10813'}