<jupyter_start><jupyter_text>Network Intrusion Detection in Netflow using RAPIDS LODA  Introduction In this workflow, we show how to use `Loda` anomaly detector for intrusion detection usecase. Loda is trained to identify attacks in the form of bots in Netflow data. We use [cic_ids2017](https://www.unb.ca/cic/datasets/ids-2017.html) benchmark dataset for the evaluation. For training, we use the activities collected from Monday that don't have any attacks on it. For evaluation, we use Friday flow data that is a mix of flows of benign and bot attacks.<jupyter_code>%config IPCompleter.use_jedi = False
import pandas as pd 
import cupy as cp 
import cudf
import cuml
import matplotlib.pylab as plt 
import os 
from cuml.decomposition import PCA as cu_pca
from itertools import permutations
from clx.analytics.loda import Loda
from util import metrics, average_precision_score, plot_pr, plot_roc
%matplotlib inline
cp.random.seed(101)<jupyter_output><empty_output><jupyter_text>Load and preprocess datasetDownload `GeneratedLabelledFlows.zip` from the above link and extract all files in a selected path.<jupyter_code>dir_path = "put/path/extracted/cic_ids2017/"
datasets = os.listdir(dir_path)
def load_dataset(dir_path):
    df = cudf.read_csv(dir_path) 
    df.columns = [ '_'.join(col.split()) for col in df.columns.str.strip()]
    return df
def drop_nonunique_features(df, nunique=1):
    # drop features with less than 2 unique variables. 
    var_col = []
    for i in df.columns:
        if df[i].value_counts().nunique() <= nunique:
            var_col.append(i)
    return df.drop(var_col, axis=1, inplace=False)
def remove_naninf(df):
    # Remove nan, inf rows. 
    temp_df = df.select_dtypes('float64')
    na_rows = temp_df.isin([cp.inf, cp.nan, -cp.inf]).any(1)
    return df[~na_rows]<jupyter_output><empty_output><jupyter_text>We use the `Monday-WorkingHours` dataset for fitting a `Loda` model, this flow dataset doesn't have any attacks. For evaluation, we test the trained `Loda` model on activities that contain both benign and attacks. We use `Friday-WorkingHours-Morning` that has an attack in the form of bots.<jupyter_code># Training days
mon_df = load_dataset(os.path.join(dir_path, "Monday-WorkingHours.pcap_ISCX.csv")).pipe(drop_nonunique_features).pipe(remove_naninf).to_pandas()

# Testing days 
frid_df = load_dataset(os.path.join(dir_path, "Friday-WorkingHours-Morning.pcap_ISCX.csv")).pipe(drop_nonunique_features).pipe(remove_naninf).to_pandas()

# Proportion of anomalies in test data 
print(f"Proportion of Benign to Bots class in the evaluation dataset\n {frid_df.Label.value_counts() / frid_df.shape[0]}")<jupyter_output>Proportion of Benign to Bots class in the evaluation dataset
 BENIGN    0.989754
Bot       0.010246
Name: Label, dtype: float64<jupyter_text>Feature extraction and transformation methods.We tried two approaches to transform the categorical features, such as `Source_IP`, `Destination_IP`, `Source_Port`, `Destination_Port` into numerical features.1. Transform categorical features into count/frequency format.<jupyter_code>def frequent_encoding(features, df):
    
    # Econde the categorical values based on the frequency of usage normalized over all rows.
    for feature in features:
        num_unique = df[feature].nunique()
        freq_count = df.groupby(feature)[feature].count()/ num_unique
        df[feature] = df[feature].map(freq_count)
        df[feature]  = (df[feature] - df[feature].min())/(df[feature].max() - df[feature].min())
    return onehot_encode('Protocol', df)

def onehot_encode(feature, df):
    dum_protocol = cudf.get_dummies(df, columns=[feature], prefix=[feature])
    protocol = dum_protocol.iloc[:, -3:] # hard coded. 
    df = cudf.concat([df, protocol], axis=1)
    return df.drop(feature, inplace=False, axis=1)<jupyter_output><empty_output><jupyter_text>2. Compute entropy of categorical values based on subgroups formed from other related categorical feature.$$H(X) = -\sum_{x_i \in X} p(x_i) \log p(x_i)$$ Where $p(x_i) = \frac{count(x_i)}{count(X)}$We compute the entropy of sub-groups formed from `['Source_IP', 'Destination_IP', 'Source_Port', 'Destination_Port']` and compute all pairwise pairwise entropy among each other. Then the new numerical features are append to the original dataframe.<jupyter_code>def entropy(series, base=None):
    vc = series.value_counts(normalize=True, sort=False)
    return -(vc * np.log(vc)/np.log(series.shape[0])).sum()
def encode_entropy(df, feature , target):
    feature_entropy = df.groupby(feature).agg(**{
        target+"_"+feature: (target, entropy)
        })
    return df.join(feature_entropy, on=feature, rsuffix="r_")
cat_features = ['Source_IP', 'Destination_IP', 'Source_Port', 'Destination_Port']
# Encode based on entropy of pairwise of categorical features.
for feature, target in permutations(cat_features,2):
    print(feature, target)
    mon_df = encode_entropy(mon_df, feature, target)
    frid_df = encode_entropy(frid_df, feature, target)


def remove_categorical_features(df):
    """
    Remove categorical features from the cic dataset
    """
    df = df.select_dtypes(['float64','int64'])
    return df.drop(["Protocol", "Source_Port", "Destination_Port"], axis=1, inplace=False)

def remove_correlated_features(df):
    """
    Remove features that are highly correlated. Mostly remove derived features.
    param df: pd.DataFrame
    """
    corr_mat = df.corr()
    # Select upper triangle of correlation matrix
    upper = corr_mat.where(np.triu(np.ones(corr_mat.shape), k=1).astype(np.bool))
    # Find features with correlation greater than 0.95
    to_drop = [column for column in upper.columns if any(upper[column] > 0.95)]
    return to_drop, df.drop(to_drop, inplace=False, axis=1)<jupyter_output><empty_output><jupyter_text>Now remove all highly correlated features and extract label from test data for evaluation.<jupyter_code># Training data
X_train = remove_categorical_features(mon_df)
removed_cols, X_train = remove_correlated_features(X_train)

# Test data
label = (frid_df['Label']=="BENIGN").astype(int)
X_test = remove_categorical_features(frid_df)
X_test.drop(removed_cols, inplace=True, axis=1)
# Frequent_encoding 
#X_test = frequent_encoding(['Source IP', 'Destination IP', 'Source Port', 'Destination Port'], X_test.copy())
# Normalize if needed. 
normalize = lambda x : (x - x.min()) /(x.max() - x.min() + 0.0000001)

# Transform back from pandas to cudf to work with Loda. 
X_train = cudf.from_pandas(X_train)
X_test = cudf.from_pandas(X_test)<jupyter_output><empty_output><jupyter_text>Train and test anomaly detector Now we fit a `Loda` to the training dataset. We can simply train the anomaly detector and generate scores from the trained model. For this experiment, we will fit the clean data from `Monday working hours` and generate scores for the `Friday working hours`We train `Loda` algorithm with `number_random_cuts=1000`.<jupyter_code>ld = Loda(n_random_cuts=1000)
ld.fit(X_train.values)
x_scores = ld.score(X_test.values)<jupyter_output><empty_output><jupyter_text>Model evaluationFollowing is the AUC (Area under ROC curve) and average precision (AP) scores of the test data.<jupyter_code>auc, ap = metrics(label, x_scores)
print(f"AUC: {auc} and Average precision (AP): {ap}")<jupyter_output>AUC: 0.5286834239959717 and Average precision (AP): 0.991349485858245<jupyter_text>AP is good but AUC is poor, we can improve this by reducing some of the noisy features. One way is to use PCA to remove irrelevant feature, it improves the performance by reducing the noise features.  Removing redundant features using PCAWe use `cudf` PCA implementation as follows.<jupyter_code>def pca_loda(x_train, x_test, variance):
    
    x_train, n_component = pca_transform(x_train, variance=variance)
    x_test  = cu_pca(n_components=n_component, whiten=False).fit_transform(x_test)
    
    loda_pca = Loda(n_random_cuts=500)
    loda_pca.fit(x_train)
    loda_score = loda_pca.score(x_test)
    return metrics(label, loda_score), loda_score, n_component

def pca_transform(df, variance=0.99):
    """Apply PCA and keep #components with #variance kept"""
    pca = cu_pca(n_components=df.shape[1])
    pca_xtrain = pca.fit_transform(df)
    variance_exp = cp.cumsum(pca.explained_variance_ratio_)
    n_components = variance_exp<=variance
    return pca_xtrain[:,n_components], sum(n_components)
ap_auc, pca_scores, n_components_used = pca_loda(X_train.values, X_test.values, variance=0.999) 
fig = plt.figure(figsize=(12,5))
plt.subplot(1,2,1)
plot_pr(label, pca_scores)
plt.subplot(1,2,2)
plot_roc(label, pca_scores)
print(f"Total number features used {n_components_used}")<jupyter_output>Total number features used 11
{'fraction_non_alphanumeric': 0.10592686002522068, 'fraction_numerical': 0.014903129657228018, 'mean_word_length': 5.289834174477289, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 35, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '126395', 'license': 'Apache-2.0', 'n_tokens_mistral': 2992, 'n_tokens_neox': 2834, 'n_words': 798, 'path': 'notebooks/ids_detection/IDS_using_LODA.ipynb', 'repo_name': 'acaldwell-pixel/clx', 'size': '8723'}