<jupyter_start><jupyter_text>Unsupervised Anomaly Detection Anomaly detection detects data points in data that does not fit well with the rest of data. In this notebook we demonstrate how to do anomaly detection for 1-D data using Chronos's dbscan detector, autoencoder detector and threshold detector. For demonstration, we use the publicly available cluster trace data cluster-trace-v2018 of Alibaba Open Cluster Trace Program. You can find the dataset introduction here. In particular, we use machine usage data to demonstrate anomaly detection, you can download the separate data file directly with machine_usage.  Helper functions This section defines some helper functions to be used in the following procedures. You can refer to it later when they're used.<jupyter_code>def get_drop_ts_and_len(df, target_col="cpu_usage", allow_missing_num=3):
    """
    get start index which with consecutive missing num larger than allow_missing_num
    """
    missing_num = df[target_col].isnull().astype(int).groupby(df[target_col].notnull().astype(int).cumsum()).sum()
    missing_num = missing_num[missing_num > 0]
    drop_ts = df.iloc[(missing_num.index + missing_num.cumsum() - missing_num).values].index
    drop_ts = drop_ts[missing_num > allow_missing_num]
    drop_missing_num = missing_num[missing_num > allow_missing_num]
    drop_len = drop_missing_num.values
    return drop_ts, drop_len
def rm_missing_day(start_dts, missing_lens, df):
    """
    drop days with consecutive missing values.
    If consecutive missing values across day, we remove the data for an extra hour.
    """ 
    for start_time, l in zip(start_dts, missing_lens):
        start = start_time.replace(hour=0, minute=0, second=0)
        start_day_end = start + pd.Timedelta(hours=24)
        start_day_end = start_day_end - pd.Timedelta(minutes=1)
        end_time = start_time + l*pd.Timedelta(minutes=1)
        if start_day_end < end_time:
            end = end_time + pd.Timedelta(hours=1)
        else:
            end = start_day_end
        df = df.drop(df[start:end].index)
    return df
def get_result_df(df, ano_index):
    """
    Add anomalies value to dataframe.
    """
    result_df = df.copy()
    result_df['anomalies'] = 0
    result_df.loc[df.index[ano_index], 'anomalies'] = 1
    result_df['anomalies'] = result_df['anomalies'] > 0
    return result_df
def plot_anomalies_value(date, y, anomalies):
    """
    plot the anomalies value
    """
    fig, axs = plt.subplots(figsize=(16,6))
    
    axs.plot(date, y, color='blue', label='y')
    axs.scatter(date[anomalies].tolist(), y[anomalies], color='red', label='anomalies value')
    axs.set_title('the anomalies value')
    
    plt.xlabel('datetime')
    plt.legend(loc='upper left')
    plt.show()<jupyter_output><empty_output><jupyter_text>Download raw dataset and load into dataframe Now we download the dataset and load it into a pandas dataframe.Steps are as below:* First, download the raw data machine_usage. Or run the script `get_data.sh` to download the raw data.It will download the resource usage of each machine from m_1932 to m_2085. * Second, run `grep m_1932 machine_usage.csv > m_1932.csv` to extract records of machine 1932. Or run `extract_data.sh`.We use machine 1932 as an example in this notebook.You can choose any machines in the similar way.* Finally, use pandas to load `m_1932.csv` into a dataframe as shown below.<jupyter_code>import os 
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
df_1932 = pd.read_csv("m_1932.csv", header=None, usecols=[1,2,3], names=["time_step", "cpu_usage","mem_usage"])<jupyter_output><empty_output><jupyter_text>Below are some example records of the data<jupyter_code>df_1932.head()
df_1932.sort_values(by="time_step").plot(y="cpu_usage", x="time_step", figsize=(16,6),title="cpu_usage of machine 1932")<jupyter_output><empty_output><jupyter_text>Data pre-processing Now we need to do data cleaning and preprocessing on the raw data. Note that this part could vary for different dataset. For the machine_usage data, the pre-processing contains 2 parts:1. Change the time unit from seconds to minutes, which is calculated by averaging `cpu_usage` within the minutes.2. Check missing values and handle missing data(fill or drop) Change the time unit to minute.<jupyter_code>df_1932["ts_min"] = df_1932["time_step"]//60<jupyter_output><empty_output><jupyter_text>Calculate the average of cpu_usage in the same time and draw the line chart.<jupyter_code>resampled_df = df_1932.groupby("ts_min").agg({"cpu_usage":"mean"})
resampled_df.plot(figsize=(16,6),title="mean cpu_usage of machine 1932")<jupyter_output><empty_output><jupyter_text>We need datetime information to generate related features,so re-adjust time series.<jupyter_code>resampled_df.index = pd.to_datetime(resampled_df.index, unit='m', origin=pd.Timestamp('2018-01-01'))<jupyter_output><empty_output><jupyter_text>Ensure the time interval of dataframe is the same.<jupyter_code>full_idx = pd.date_range(start=resampled_df.index.min(), end=resampled_df.index.max(),freq='T')
resampled_df = resampled_df.reindex(full_idx)
resampled_df.head()<jupyter_output><empty_output><jupyter_text>Here, we drop day with more than 3 consecutive missing values and fill other missing values remained.<jupyter_code>drop_ts,drop_len = get_drop_ts_and_len(resampled_df,allow_missing_num=3)
df = rm_missing_day(drop_ts, drop_len,resampled_df)
df.head()
print("no. of n/a values:")
print(df.isna().sum())
df.ffill(inplace=True)
df.plot(figsize=(16,6))<jupyter_output><empty_output><jupyter_text>Anomaly Detection by DBScan Detector DBScanDetector uses DBSCAN clustering for anomaly detection. The DBSCAN algorithm tries to cluster the points and label the points that do not belong to any clusters as -1. It thus detects outliers detection in the input time series. DBScanDetector assigns anomaly score 1 to anomaly samples, and 0 to normal samples.<jupyter_code>from zoo.chronos.model.anomaly import DBScanDetector

ad = DBScanDetector(eps=0.1, min_samples=6)
ad.fit(df['cpu_usage'].to_numpy())
anomaly_scores = ad.score()
anomaly_indexes = ad.anomaly_indexes()

print("The anomaly scores are:", anomaly_scores)
print("The anomaly indexes are:", anomaly_indexes)<jupyter_output>The anomaly scores are: [0. 0. 0. ... 1. 1. 1.]
The anomaly indexes are: [ 263  368  386  387  388  389  390  391  392  393  394  395  396  403
  404  452  568  605  610  611  614  624  664  665  666  667  668  669
  670  671  725  726  850 1108 1124 1128 1428 1444 1445 1801 1829 1830
 1831 1832 1833 1834 1835 1836 1837 1838 1846 1851 1852 1870 1885 1886
 2096 2112 2113 2114 2195 2518 2519 2669 2670 2678 3024 3215 3216 3217
 3218 3219 3220 3221 3222 3223 3224 3225 3226 3233 3234 3235 3236 3244
 3259 3260 3279 3280 3285 3304 3324 3326 3327 3328 3329 3332 3333 3359
 3360 3361 3373 3388 3394 3399 3400 3412 3422 3423 3476 3580 3581 3582
 3584 3585 3721 3729 3918 3922 4074 4293 4302 4312 4466 4625 4676 4683
 4684 4685 4686 4687 4688 4689 4690 4691 4692 4693 4694 4708 4726 4727
 4728 4756 4757 4766 4767 4801 4802 4830 4831 4832 4857 4861 5362 5363
 5364 5453 5456 5458 5459 5460 5465 5472 5481 5485 5487 5492 6094 6104
 6126 6127 6128 6129 6130 6131 6132 6133 6134 6135 6136 6137 6138 6139
 614[...]<jupyter_text>Draw anomalies in line chart.<jupyter_code>result_df = get_result_df(df, anomaly_indexes)
plot_anomalies_value(result_df.index, result_df.cpu_usage, result_df.anomalies)<jupyter_output><empty_output><jupyter_text>Anomaly Detection by AutoEncoder Detector AEDetector is unsupervised anomaly detector. It builds an autoencoder network, try to fit the model to the input data, and calcuates the reconstruction error. The samples with larger reconstruction errors are more likely the anomalies.<jupyter_code>from zoo.chronos.model.anomaly import AEDetector

ad = AEDetector(roll_len=10, ratio=0.05)
ad.fit(df['cpu_usage'].to_numpy())
anomaly_scores = ad.score()
anomaly_indexes = ad.anomaly_indexes()

print("The anomaly scores are:", anomaly_scores)
print("The anomaly indexes are:", anomaly_indexes)<jupyter_output>The anomaly scores are: [0.22100376 0.20443486 0.20183592 ... 0.96199906 0.96200335 0.9604706 ]
The anomaly indexes are: [2673 3228 4300 1131 2668 5425 5502 1130 3929 5422 5468 1106 7111 4314
 7408 5446 7106 2670  669 4313 2678 5476 7203 7104 6659 4311 4293 2676
 5431 7107 4307 5353 4304 5434 5433 7105  665 1839 5739 7112 6140 7110
 7199 5423 5427 1108 7102 7401 7108 5435 4309 3024 7109 5501 7204 5471
 7161 5500  667 3928 7113 1126 7175 6649 7402 6651 1133 1123 1127 5477
 5426 4301 3916  668 3215 5472 6126 5424 7162 5436 7163 5469 7164 6652
 5478 1121 6650 4308 5428 5429 1122 1129 7170 5498 7171 2669 3227 5432
 4303 5356 4312 7165 7411 6658 7166  666 1125 5479 7173 5467 1128 5499
 7172 5450 6654 7174 7403 3923 1132 6653 7409 7405 3925 5449 4302 3924
 7169 3917 1124 7167 6655 5357 8466 5496 7168 1829 5369 5451 3926 5482
 5495 6656 3927 5466 3919 6657 3921 3922 6139 7564 3920 5480 5483 5368
 7404 5359 3918 5358 5367 5360 5494 8467 5361 5363 5362 5365 5366 5490
 5465 5491 5462 4683 5481 5[...]<jupyter_text>Draw anomalies in line chart.<jupyter_code>result_df = get_result_df(df, anomaly_indexes)
plot_anomalies_value(result_df.index, result_df.cpu_usage, result_df.anomalies)<jupyter_output><empty_output><jupyter_text>Anomaly Detection by Threshold Detector ThresholdDetector is a simple anomaly detector that detectes anomalies based on threshold. The target value for anomaly testing can be either 1) the sample value itself or 2) the difference between the forecasted value and the actual value. In this notebook we demostrate the first type. The thresold can be set by user or esitmated from the train data accoring to anomaly ratio and statistical distributions.<jupyter_code>from zoo.chronos.model.anomaly import ThresholdDetector

thd=ThresholdDetector()
thd.set_params(threshold=(20, 80))
thd.fit(df['cpu_usage'].to_numpy())
anomaly_scores = thd.score()
anomaly_indexes = thd.anomaly_indexes()

print("The anomaly scores are:", anomaly_scores)
print("The anomaly indexes are:", anomaly_indexes)<jupyter_output>The anomaly scores are: [0. 0. 0. ... 1. 1. 1.]
The anomaly indexes are: [6656, 6657, 7168, 7169, 8548, 8549, 8550, 8551, 8552, 8630, 8554, 7199, 8555, 7204, 8556, 8631, 8559, 8562, 4684, 4685, 4686, 4687, 4688, 4689, 1106, 4690, 4692, 1108, 4691, 4693, 8564, 1124, 8568, 1128, 8569, 5739, 1132, 2669, 2670, 8570, 8571, 2678, 8572, 8573, 8574, 8614, 8575, 8576, 3216, 3217, 3218, 3219, 8577, 3221, 3222, 8578, 3224, 3225, 8579, 8580, 8581, 8582, 8584, 8586, 8587, 4293, 4302, 4303, 8589, 4308, 4312, 7167, 226, 227, 8594, 7403, 7404, 7405, 5361, 5362, 5363, 5364, 5365, 5366, 7409, 7411, 8623, 8599, 263, 8601, 8603, 8469, 8470, 8471, 8472, 8473, 8474, 8475, 8476, 8477, 8478, 8479, 8480, 8481, 8482, 8483, 8484, 8485, 8486, 8487, 8488, 8489, 1834, 1835, 1836, 8490, 8491, 8492, 8493, 8494, 8495, 8496, 8497, 8498, 8499, 8500, 5432, 8501, 8502, 8503, 5436, 8504, 8505, 8506, 8507, 8508, 8509, 8510, 8511, 8512, 8513, 8514, 8515, 8516, 8517, 8518, 8519, 5453, 5454, 3918, 5456, 5457, 5458, 5459, 5460,[...]<jupyter_text>Draw anomalies in line chart.<jupyter_code>result_df = get_result_df(df, anomaly_indexes)
plot_anomalies_value(result_df.index, result_df.cpu_usage, result_df.anomalies)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08705985915492957, 'fraction_numerical': 0.19339788732394367, 'mean_word_length': 5.088424437299035, 'pattern_counts': {'":': 1, '<': 55, '<?xml version=': 0, '>': 59, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 31, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '248280', 'license': 'Apache-2.0', 'n_tokens_mistral': 5653, 'n_tokens_neox': 3900, 'n_words': 1318, 'path': 'pyzoo/zoo/chronos/use-case/AIOps/AIOps_anomaly_detect_unsupervised.ipynb', 'repo_name': 'block4chain/analytics-zoo', 'size': '11360'}