<commit_before># Network Anomaly Detection Pipeline using Dataflow and  BQ-ML 
A dataflow streaming pipeline to process large scale network security log data from GCS or PubSub and find outliers using K-Means clustering. In high level, this repo will walk you through following three areas:

1. Aggregate Network logs in N-mins window.
	- Fixed Window with After Processing time triggers 
	- Using schema inferring to group by subnet and subscriberId and aggregate data. 
	- There are 4 types of aggregations (Min, Max, Avg and  Approximate Unique Count )
	- Use N minutes intervals (FILE_LOAD) to batch insert to BQ in a partition & clustered table.  
2. Create K-Means Clustering model using BQ ML.
	- Using BQ ML query to create K-Means model. This is done using stored procedure and schedule query in BQ.
	- Normalized the data by finding STD DEV for each cluster to help with outlier detection. 
3. Prediction (Batch/Online)
	- Find Z-Score and check if it's 2 STD DEV above the mean.
	- Use dataflow to calculate nearest distance from centroid and score between sample and centroid. 
	- Streaming insert to BQ table if outliers found.
 

## Reference Architecture

![ref_arch](diagram/ref-arch.png)

##  How It works?

### Example input log data and output after aggregation

Sample Input Data

```
{
 \"subscriberId\": \"100\",
 \"srcIP\": \"192.168.127.12",<commit_msg>in progress read me to merge before demo<commit_after># Anomaly Detection Using Machine Learning To Predict Cyber Security Threat 
A dataflow streaming pipeline to process network flow log from GCS and/or PubSub to find outliers in real time by using a K-Means Clustering Model created from BQ-ML. In high level, this repo can be used to demo following areas:

1. Streaming Analytics at Scale using Dataflow (Feature Extraction)
	- Fixed Window with After Processing time triggers 
	- Using schema inferring to group by subnet and subscriberId and aggregate data. 
	- There are 4 types of aggregations (Min, Max, Avg and  Approximate Unique Count )
	- Use N minutes intervals (FILE_LOAD) to batch insert to BQ in a partition & clustered table.  
2. K-Means Clustering model using BQ ML.
	- Using BQ ML query to create K-Means model. This is done using stored procedure and schedule query in BQ.
	- Normalized the data by finding STD DEV for each cluster to help with outlier detection. 
3. Anomaly detection in near real time (Online Prediction)
	- Find Z-Score and check if it's 2 STD DEV above the mean.
	- Use dataflow to calculate nearest distance from centroid and score between sample and centroid. 
	- Streaming insert to BQ table if outliers found.
 

## Reference Architecture

![ref_arch](diagram/ref-arch.png)

## Quickly Setup a Customer Demo
   
##  How It works?

### Example input log data and output after aggregation

Sample Input Data

```
{
 \"subscriberId\": \"100\",
 \"srcIP\": \"192.168.127.12",
{'fraction_non_alphanumeric': 0.0614853195164076, 'fraction_numerical': 0.013126079447322971, 'mean_word_length': 3.8266666666666667, 'pattern_counts': {'":': 4, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4679672', 'n_tokens_mistral': 860, 'n_tokens_neox': 771, 'n_words': 420}