<jupyter_start><jupyter_text>Anomaly Exploration (understanding 'Odd')In this notebook we're going to be using the zat Python module for processing, transformation and anomaly detection on Zeek network data. We're going to look at 'normal' http traffic and demonstrate the use of Isolation Forests for anomaly detection. We'll then explore those anomalies with clustering and PCA.**Software**- zat: https://github.com/SuperCowPowers/zat- Pandas: https://github.com/pandas-dev/pandas- Scikit-Learn: http://scikit-learn.org/stable/index.html**Techniques**- One Hot Encoding: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html- Isolation Forest: http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html- PCA: http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html**Related Notebooks**- Zeek to Scikit-Learn: https://nbviewer.jupyter.org/github/SuperCowPowers/zat/blob/master/notebooks/Zeek_to_Scikit_Learn.ipynb**Note:** A previous version of this notebook used a large http log (1 million rows) but we wanted people to be able to run the notebook themselves, so we've changed it to run on the local example http.log.<jupyter_code>import zat
from zat.log_to_dataframe import LogToDataFrame
from zat.dataframe_to_matrix import DataFrameToMatrix
print('zat: {:s}'.format(zat.__version__))
import pandas as pd
print('Pandas: {:s}'.format(pd.__version__))
import numpy as np
print('Numpy: {:s}'.format(np.__version__))
import sklearn
from sklearn.ensemble import IsolationForest
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
print('Scikit Learn Version:', sklearn.__version__)
# Create a Pandas dataframe from the Zeek HTTP log
log_to_df = LogToDataFrame()
zeek_df = log_to_df.create_dataframe('../data/http.log')
print('Read in {:d} Rows...'.format(len(zeek_df)))
zeek_df.head()<jupyter_output>Read in 150 Rows...<jupyter_text>So... what just happened?**Yep it was quick... the two little lines of code above turned a Zeek log (any log) into a Pandas DataFrame. The zat package also supports streaming data from dynamic/active logs, handles log rotations and in general tries to make your life a bit easier when doing data analysis and machine learning on Zeek data.****Now that we have the data in a dataframe there are a million wonderful things we could do for data munging, processing and analysis but that will have to wait for another time/notebook.**<jupyter_code># We're going to pick some features that might be interesting
# some of the features are numerical and some are categorical
features = ['id.resp_p', 'method', 'resp_mime_types', 'request_body_len']<jupyter_output><empty_output><jupyter_text>Our HTTP features are a mix of numeric and categorical dataWhen we look at the http records some of the data is numerical and some of it is categorical so we'll need a way of handling both data types in a generalized way. zat has a DataFrameToMatrix class that handles a lot of the details and mechanics of combining numerical and categorical data, we'll use below.<jupyter_code># Show the dataframe with mixed feature types
zeek_df[features].head()<jupyter_output><empty_output><jupyter_text>Transformers**We'll now use a scikit-learn tranformer class to convert the Pandas DataFrame to a numpy ndarray (matrix). Yes it's awesome... I'm not sure it's Optimus Prime awesome.. but it's still pretty nice.**<jupyter_code># Use the zat DataframeToMatrix class (handles categorical data)
# You can see below it uses a heuristic to detect category data. When doing
# this for real we should explicitly convert before sending to the transformer.
to_matrix = DataFrameToMatrix()
zeek_matrix = to_matrix.fit_transform(zeek_df[features], normalize=True)
print(zeek_matrix.shape)
zeek_matrix[:1]
# Train/fit and Predict anomalous instances using the Isolation Forest model
odd_clf = IsolationForest(behaviour='new', contamination=0.25) # Marking 25% odd
odd_clf.fit(zeek_matrix)
# Now we create a new dataframe using the prediction from our classifier
odd_df = zeek_df[features][odd_clf.predict(zeek_matrix) == -1]
print(odd_df.shape)
odd_df.head()
# Now we're going to explore our odd dataframe with help from KMeans and PCA algorithms
odd_matrix = to_matrix.fit_transform(odd_df)
# Just some simple stuff for this example, KMeans and PCA
kmeans = KMeans(n_clusters=4).fit_predict(odd_matrix)  # Change this to 3/5 for fun
pca = PCA(n_components=3).fit_transform(odd_matrix)

# Now we can put our ML results back onto our dataframe!
odd_df['x'] = pca[:, 0] # PCA X Column
odd_df['y'] = pca[:, 1] # PCA Y Column
odd_df['cluster'] = kmeans
odd_df.head()
# Plotting defaults
%matplotlib inline
import matplotlib.pyplot as plt
plt.rcParams['font.size'] = 14.0
plt.rcParams['figure.figsize'] = 15.0, 6.0

# Helper method for scatter/beeswarm plot
def jitter(arr):
    stdev = .02*(max(arr)-min(arr))
    return arr + np.random.randn(len(arr)) * stdev
# Jitter so we can see instances that are projected coincident in 2D
odd_df['jx'] = jitter(odd_df['x'])
odd_df['jy'] = jitter(odd_df['y'])

# Now use dataframe group by cluster
cluster_groups = odd_df.groupby('cluster')

# Plot the Machine Learning results
colors = {0:'green', 1:'blue', 2:'red', 3:'orange', 4:'purple', 5:'brown'}
fig, ax = plt.subplots()
for key, group in cluster_groups:
    group.plot(ax=ax, kind='scatter', x='jx', y='jy', alpha=0.5, s=250,
               label='Cluster: {:d}'.format(key), color=colors[key])
# Now print out the details for each cluster
pd.set_option('display.width', 1000)
for key, group in cluster_groups:
    print('\nCluster {:d}: {:d} observations'.format(key, len(group)))
    print(group[features].head())<jupyter_output>Cluster 0: 8 observations
                               id.resp_p method        resp_mime_types  request_body_len
ts                                                                                      
2013-09-15 23:44:47.464160919         80    GET  application/x-dosexec                 0
2013-09-15 23:44:47.464160919         80    GET  application/x-dosexec                 0
2013-09-15 23:44:49.221977949         80    GET  application/x-dosexec                 0
2013-09-15 23:44:50.805350065         80    GET  application/x-dosexec                 0
2013-09-15 23:44:51.404618979         80    GET  application/x-dosexec                 0

Cluster 1: 7 observations
                               id.resp_p   method resp_mime_types  request_body_len
ts                                                                                 
2013-09-15 23:48:06.495719910         80  OPTIONS      text/plain                 0
2013-09-15 23:48:07.495719910         80  OPTIONS      text/plain      [...]<jupyter_text>Categorical variables that are anomalous- Cluster 0: application/x-dosexec mime_types- Cluster 1: http method of OPTIONS (instead of normal GET/POST)- Cluster 2: See Below- Cluster 3: response port of 8080 (instead of 80) Numerical variable outliers- Cluster 2: The request_body_len values are outliers (for this demo dataset)**The important thing here is that both categorical and numerical variables were properly handled and the machine learning algorithm 'did the right thing' when marking outliers (for categorical and numerical fields)**<jupyter_code># Distribution of the request body length
Zeek_df[['request_body_len']].hist()
print('\nFor this small demo dataset almost all request_body_len are 0\nCluster 2 represents outliers')<jupyter_output>For this small demo dataset almost all request_body_len are 0
Cluster 2 represents outliers
{'fraction_non_alphanumeric': 0.09282700421940929, 'fraction_numerical': 0.032041139240506326, 'mean_word_length': 3.9868507560815254, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '135789', 'license': 'Apache-2.0', 'n_tokens_mistral': 2492, 'n_tokens_neox': 2233, 'n_words': 794, 'path': 'notebooks/Anomaly_Detection.ipynb', 'repo_name': 'z-3zh/zat', 'size': '7584'}