<jupyter_start><jupyter_text>Endpoints Multivariate Anomaly Detection Endpoint`multi-ad-ppe.ppe.cognitiveservices.azure.com/anomalydetector/v1.1-preview`Multivariate Anomaly Detection APIs- List models `[GET] /multivariate/models`- Train a model `[POST] /multivariate/models`- Get a model `[GET] /multivariate/models/{model_id}`- Delete a model `[DELETE] /multivariate/models/{model_id}`- Detect anomalies `[POST] /multivariate/models/{model_id}/detect`- Export a model `[GET] /multivariate/models/{model_id}/export`- Get detection result `[GET] /multivariate/results/{result_id}`<jupyter_code>ENDPOINT = "[Place_holder].cognitiveservices.azure.com/anomalydetector/v1.1-preview"
HEADERS = {
    "Ocp-Apim-Subscription-Key": "[Place_holder]"
}
API_MODEL = "https://{endpoint}/multivariate/models"
API_MODEL_STATUS = "https://{endpoint}/multivariate/models/{model_id}"
API_MODEL_INFERENCE = "https://{endpoint}/multivariate/models/{model_id}/detect"
API_RESULTS = "https://{endpoint}/multivariate/results/{result_id}"
API_EXPORT = "https://{endpoint}/multivariate/models/{model_id}/export"
API_DELETE = "https://{endpoint}/multivariate/models/{model_id}"
SOURCE_BLOB_SAS = "https://aka.ms/AnomalyDetector/MVADSampleData"<jupyter_output><empty_output><jupyter_text>List Models```http[GET] https://{endpoint}/multivariate/models?$skip=0&$top=5``` Sample - Request```jsonheader ={"Content-Type": "application/json", "Ocp-Apim-Subscription-Key": "subscription_key"}``` - Response```jsonresponse={  "models": [    {        "createdTime":"2020-12-01T09:43:45Z",        "displayName":"DevOps-Test",        "lastUpdatedTime":"2020-12-01T09:46:13Z",        "modelId":"b4c1616c-33b9-11eb-824e-0242ac110002",        "status":"READY",        "variablesCount":18    },    {        "createdTime":"2020-12-01T09:43:30Z",        "displayName":"DevOps-Test",        "lastUpdatedTime":"2020-12-01T09:45:10Z",        "modelId":"ab9d3e30-33b9-11eb-a3f4-0242ac110002",        "status":"READY",        "variablesCount":18    },  ],  "currentCount": 1,  Current count of trained multivariate models.  "maxCount": 50,  Max number of models that can be trained for this subscription.  "nextLink": "string"  next link to fetch more models}``` - Error Response ```json{"code": "string","message": "string"} ```<jupyter_code>res = requests.get(API_MODEL.format(endpoint=ENDPOINT), headers=HEADERS)
assert res.status_code == 200, f"Error occured. Error message: {res.content}"
print(res.content)<jupyter_output><empty_output><jupyter_text>Train Model  ```HTTP[POST] https://{endpoint}/multivariate/models```-  Request  - **slidingWindow** An optional field, indicates how many history points will be used to determine the anomaly score of one subsequent point.  - **alignPolicy** An optional field, since those multivariate need to be aligned in the same timestamp before starting the detection.  - **alignMode**, An optional field, indicates how we merge different variables into the same time-range which is required by the model  ```json  {Inner, Outer}  ```  - **fillNAMethod**, optional field, indicates how missed valus will be filled with. Can not be set to *NotFill*, when alignMode is *Outer*  ```json  {Previous, Subsequent, Linear, Zero, Fix, NotFill}  ```  - **paddingValue**, optional field, only be usefull if fillNAMethod is set to *Fix*.   - **source**, required field, source path contain the training zip file.    ```bash  Source file link of the input variables, each variable will be a csv with two columns, the first column will be timestamp, the second column will be value.Besides these variable csv files, an extra meta.json can be included in th zip file if you would like to rename a variable.Be default, the file name of the variable will be used as the variable name.",example: "https://multiadsample.blob.core.windows.net/data/sample_data_2_1000.zip?sp=rl&st=2020-12-04T06:03:47Z&se=2022-12-05T06:03:00Z&sv=2019-12-12&sr=b&sig=AZTbvZ7fcp3MdqGY%2FvGHJXJjUgjS4DneCGl7U5omq5c%3D"  ```  - **startTime**, require field, it means start time of data you want to use for training, should be %Y-%m-%dT%H:%M:%SZ   format.  - **endTime**, require field, it means end time of data you want to use for training, should be %Y-%m-%dT%H:%M:%SZ format.  - **displayName**, optional field, it means model name.  -  Response  - model location in headers-  Sample  - Request  ```jsonheader ={"Content-Type": "application/json", "Ocp-Apim-Subscription-Key": "subscription_key"}request={    "slidingWindow": 200,    "alignPolicy": {        "alignMode": "Outer",        "fillNAMethod": "Linear",         "paddingValue": 0    },    "source": SOURCE_BLOB_SAS,    "startTime": "2021-01-01T00:00:00Z",     "endTime": "2021-01-02T12:00:00Z",     "displayName": "SampleRequest"}  ```  - Response  ```json  Location:	{endpoint}/multivariate/models/927e2af8-a657-11ea-8cae-0242ac110002  GUID  ``` - Error Response ```json {   "code": "string",   "message": "string" } ```<jupyter_code>SLIDING_WINDOW = 200
data = {
    'slidingWindow': SLIDING_WINDOW,
    'alignPolicy': {
        'alignMode': 'Outer',
        'fillNAMethod': 'Linear', 
        'paddingValue': 0
    },
    'source': SOURCE_BLOB_SAS,
    'startTime': '2021-01-01T00:00:00Z', 
    'endTime': '2021-01-02T12:00:00Z', 
    'displayName': 'SampleRequest'
}

res = requests.post(API_MODEL.format(endpoint=ENDPOINT), data=json.dumps(data), headers=HEADERS)
assert res.status_code == 201, f"Error occured. Error message: {res.content}"
print(res.content)
location = res.headers['Location']
print(location)
model_id = location[location.rindex('/')+1:]
print(model_id)<jupyter_output><empty_output><jupyter_text>Get Multivaraite Model Status by modelid```http[GET] https://{endpoint}/multivariate/models/{model_id}``` Sample - Request```jsonheader ={"Content-Type": "application/json", "Ocp-Apim-Subscription-Key": "subscription_key"}``` - Response ```jsonresponse = {    "createdTime":"2020-07-01T07:58:37Z",    "lastUpdatedTime":"2020-07-01T07:59:55Z",    "modelId":"ab888466-bb70-11ea-958f-0242ac110002",    "modelInfo":{  Training Status of the model.        "diagnoseInfo":{            "modelState":{                "epochIds":[10, 20, 30, 40, 50, 60, 70, 80, 90, 100],  100 epoc in total                "latenciesInSeconds":[0.5837657451629639, 0.5688292980194092, 0.5959596633911133, 0.5251538753509521, 0.6021878719329834, 0.6459534168243408, 0.5391685962677002, 0.5622642040252686, 0.5487074851989746, 0.6336326599121094],                "trainLosses":[1.682054042816162, 0.7844524383544922, 0.6616984605789185, 0.6293938159942627, 0.6323581337928772, 0.6257774233818054, 0.5985430479049683, 0.6037595868110657, 0.5779791474342346, 0.5583345293998718],                "validationLosses":[0.8330008387565613, 0.6937242150306702, 0.7329594492912292, 0.6103720664978027, 0.6125020980834961, 0.5729937553405762, 0.5761528611183167, 0.5710235238075256, 0.5679566264152527, 0.5674979090690613]            },            "variableStates":[                {                     "effectiveCount":1441,                    "endTime":"2019-04-02T00:00:00Z",                    "errors":[],                    "filledNARatio":0.0,                    "startTime":"2019-04-01T00:00:00Z",                    "variable":"established_connections"                },                {                     "effectiveCount":1441,  Effective time-series points count.                    "endTime":"2019-04-02T00:00:00Z",  End time of a variable                    "errors":[],                    "filledNARatio":0.0,  NA ratio of a variable.                    "startTime":"2019-04-01T00:00:00Z",  Start time of a variable                    "variable":"memory"                }            ]        },        "errors":[],  Error message when creating or training model fails.        "status":"READY",         "alignPolicy":{                "alignMode":"Outer",                "fillNAMethod":"Linear",                "paddingValue":0         },         "displayName":"DevOps-Test",         "endTime":"2019-04-02T00:00:00Z",         "slidingWindow":28,         "source":"/data/sample_data.zip",         "startTime":"2019-04-01T00:00:00Z"    }}  ``` - Error Response ```json{"code": "string","message": "string"} ```<jupyter_code>res = requests.get(API_MODEL_STATUS.format(endpoint=ENDPOINT, model_id = model_id), headers=HEADERS)
assert res.status_code == 200, f"Error occured. Error message: {res.content}"
res_content = json.loads(res.content)
print(json.dumps(res_content))
print(res_content['modelInfo']['status'])<jupyter_output><empty_output><jupyter_text>Detection with the trained model```http[POST] https://{endpoint}/multivariate/models/{model_id}/detect```inference api will return an resultid, you can get result by get inference result api.-  Request   - **source**, required field, source path contain the training zip file.```json Source file link of the input variables, each variable will be a csv with two columns, the first column will be timestamp, the second column will be value.Besides these variable csv files, an extra meta.json can be included in th zip file if you would like to rename a variable.Be default, the file name of the variable will be used as the variable name.",example: "https://multiadsample.blob.core.windows.net/data/sample_data_2_1000.zip?sp=rl&st=2020-12-04T06:03:47Z&se=2022-12-05T06:03:00Z&sv=2019-12-12&sr=b&sig=AZTbvZ7fcp3MdqGY%2FvGHJXJjUgjS4DneCGl7U5omq5c%3D"```   - **startTime**, a require field, it means start time of data you want to use to inference, should be %Y-%m-%dT%H:%M:%SZ format.    - **endTime**, a require field, it means end time of data you want to use to inference, should be %Y-%m-%dT%H:%M:%SZ format.  -  Response  - result location in headers  -  Sample  - Request```jsonheader ={"Content-Type": "application/json", "Ocp-Apim-Subscription-Key": "subscription_key"}request={  "source": SOURCE_BLOB_SAS,  "startTime": "2020-01-01T00:00:00Z",  "endTime": "2020-02-01T00:00:00Z"}```- Response```jsonLocation:    {endpoint}/multivariate/results/927e2af8-a657-11ea-8cae-0242ac110002  GUID ```- Error Response ```json{"code": "string","message": "string"} ```<jupyter_code>data = {
    'source': SOURCE_BLOB_SAS,
    'startTime': '2021-01-02T12:00:00Z', 
    'endTime': '2021-01-03T00:00:00Z', 
}

res = requests.post(API_MODEL_INFERENCE.format(endpoint=ENDPOINT, model_id=model_id), 
                    data=json.dumps(data), headers=HEADERS)
assert res.status_code == 201, f"Error occured. Error message: {res.content}"
print(res.content)
result_id = res.headers['location'].split("/")[-1]
print(f"result id = {result_id}")<jupyter_output><empty_output><jupyter_text>Get Detection Result```http[GET] https://{endpoint}/multivariate/results/{result_id}```-  Sample- Request```jsonheader ={"Content-Type": "application/json", "Ocp-Apim-Subscription-Key": "subscription_key"}```- Response```jsonResponse={  "resultId": "45aad126-aafd-11ea-b8fb-d89ef3400c5f",  "summary": {    "status": "READY",  Multivariate anomaly detection status    "errors": [  Error message when inference fails.      {        "code": "string",        "message": "string"      }    ],    "variableStates": [      {        "variable": "ad_input",        "filledNARatio": 0,        "effectiveCount": 26,        "startTime": "2019-04-01T00:00:00Z",        "endTime": "2019-04-01T00:25:00Z",        "errors": []      },      {        "variable": "ad_ontimer_output",        "filledNARatio": 0,        "effectiveCount": 26,        "startTime": "2019-04-01T00:00:00Z",        "endTime": "2019-04-01T00:25:00Z",        "errors": []      }    ],    "setupInfo": {      "source": "/data/{$zipfile_name}",      "startTime": "2019-04-01T00:15:00Z",      "endTime": "2019-04-01T00:40:00Z"    }  },  "results": [    {      "timestamp": "2019-04-01T00:19:00Z",      "errors": [        {          "code": "InsufficientHistoricalData",          "message": "historical data is not enough."        }      ]    },    {      "timestamp": "2019-04-01T00:20:00Z",      "value":{          "contributors": [], isAnomaly is false, contributors is empty.          "isAnomaly": false,          "severity": 0.3509107994398884,          "score": 0.34231      },      "errors": []    },    {      "timestamp": "2019-04-01T00:21:00Z",      "value":{          "contributors": [          {              "contributionScore": 0.0007775013367514271,  The higher the contributionScore is, the more likely the contributor to be the root cause of a anomaly.              "variable": "ad_ontimer_output"  Variable name of a contributor          },          {              "contributionScore": 0.0007989604079048129,              "variable": "ad_input"          }        ],          "isAnomaly": true,  To indicate whether current timestamp is anomaly or not          "severity": 0.42135109874230336,  severity of the current timestamp, the more significant an anomaly is, the higher the severity will be          "score": 0.23485905670108112      },      "errors": []    }   ]}``` - Error Response ```json{"code": "string","message": "string"} ```<jupyter_code>res = requests.get(API_RESULTS.format(endpoint=ENDPOINT, result_id=result_id), headers=HEADERS)
assert res.status_code == 200, f"Error occured. Error message: {res.content}"
print(res.content)<jupyter_output><empty_output><jupyter_text>Visualize ResultsDemo code to draw results. Additional python package is required.<jupyter_code>from bokeh.io import output_file, show, output_notebook, save
from bokeh.layouts import gridplot
from bokeh.plotting import figure
from matplotlib import pyplot
from bokeh.models import ColumnDataSource, HoverTool
from bokeh.palettes import Dark2_5 as palette
import pandas as pd
import numpy as np
import os
import itertools  
import shutil
import uuid
import zipfile
from urllib.request import urlretrieve
%matplotlib inline
output_notebook()

def unzip_file(zip_src, dst_dir):
    r = zipfile.is_zipfile(zip_src)
    if r:
        fz = zipfile.ZipFile(zip_src, 'r')
        print(fz)
        for file in fz.namelist():
            fz.extract(file, dst_dir)
    else:
        print('This is not zip')
        
def load_data(local_data_path, start, end):
    cwd = os.getcwd()
    new_dir = os.path.join(cwd, str(uuid.uuid1()))
    shutil.rmtree(new_dir, ignore_errors=True)
    os.mkdir(new_dir)
    unzip_file(local_data_path, new_dir)
    files = os.listdir(new_dir)
    frames = []
    for file in files:
        if file[-4:] != '.csv':
            continue
        frame = pd.read_csv('{}\\{}'.format(new_dir, file))
        var = file[:file.find('.csv')]
        frame = frame.rename(columns={'value': var})
        frame = frame[frame['timestamp'] >= start]
        frame = frame[frame['timestamp'] <= end]
        frame['timestamp'] = pd.to_datetime(frame['timestamp'])
        frame.set_index(['timestamp'], inplace=True)
        frames.append(frame)
    shutil.rmtree(new_dir, ignore_errors=True)
    return frames


def plot_lines_multi(x, y, p, color, name, t_str="hover,save,pan,box_zoom,reset,wheel_zoom", t_loc='above'):
    '''...
    '''
    p.line(x, y, color=color, legend_label=name)

def draw(data_source, local_data_path, result_id, sensitivity, start, end):
    urlretrieve(data_source, local_data_path)
    print(local_data_path, result_id, sensitivity, start, end)
    series = load_data(local_data_path, start, end)
    p_list = []
    colors = itertools.cycle(palette)
    # p_value = figure(background_fill_color="#fafafa", x_axis_type="datetime")
    for var, color in zip(series, colors):
        name = var.columns.values[0]
        p_value = figure(background_fill_color="#fafafa", x_axis_type="datetime")
        plot_lines_multi(var.index, var[name], p_value, color, name)
        p_list.append(p_value)
    header = HEADERS
    raw_result = json.loads(requests.get(API_RESULTS.format(endpoint=ENDPOINT, result_id=result_id), headers=header).content)
    if raw_result['summary']['status'] != 'READY':
        print("result not ready")
        return
    filter_item = list(filter(lambda x: 'value' in x and 'isAnomaly' in x['value'] and x['timestamp'] >= start and x['timestamp'] <= end, raw_result['results']))
    timestamps = [item['timestamp'] for item in filter_item]
    isAnomaly = [item['value']['isAnomaly'] for item in filter_item]
    RawScore = [item['value']['score'] for item in filter_item]
    Severity = [item['value']['severity'] for item in filter_item]
    result = pd.DataFrame({'Timestamp': timestamps, 'isAnomaly': isAnomaly, 'RawScore': RawScore, 'Severity': Severity})
    result.loc[(result.Severity <= (1 - sensitivity)) & (result.isAnomaly == True), 'isAnomaly'] = False
    result['Timestamp'] = pd.to_datetime(result['Timestamp'])
    result.set_index(['Timestamp'], inplace=True)
    result = result.reindex(['isAnomaly', 'RawScore', 'Severity'], axis=1)
    colors = ['red', 'blue', 'black']
    for col, color in zip(result.columns, colors):
        p = figure(background_fill_color="#fafafa", x_axis_type="datetime")
        p.line(result.index, result[col], color=color, alpha=0.8, legend_label=col)
        p_list.append(p)
    grid = gridplot([[x] for x in p_list], sizing_mode='scale_width', plot_height=50)
    show(grid)
    result = result.sort_values(by=['RawScore'], ascending=False)
    top_anomaly = list(result[result.isAnomaly].index.strftime('%Y-%m-%dT%H:%M:%SZ'))[0]
    print("Top Anomaly Timestamp is : {0}".format(top_anomaly))
    return series, raw_result, top_anomaly

def show_contribution(series, raw_result, anomaly_timestamp):
    anomaly_result = [x for x in raw_result['results'] if 'contributors' in x['value'] and x['timestamp'] == top_anomaly][0]
    contributors = [x['variable'] for x in anomaly_result['value']['contributors']]
    series_index = pd.DataFrame({'index': list(range(0, len(series))), 'name': [x.columns[0] for x in series]})
    series_index = series_index.set_index('name')
    sorted_series = [series[i][series[i].index <= top_anomaly].tail(SLIDING_WINDOW) for i in series_index.reindex(contributors)['index'].values]
    p_list = []
    colors = itertools.cycle(palette)
    for var, color in zip(sorted_series, colors):
        name = var.columns.values[0]
        p_value = figure(background_fill_color="#fafafa", x_axis_type="datetime")
        plot_lines_multi(var.index, var[name], p_value, color, name)
        p_list.append(p_value)
    grid = gridplot([[x] for x in p_list], sizing_mode='scale_width', plot_height=50)
    show(grid)
data_source = SOURCE_BLOB_SAS
local_data_path = "sample_data_5_3000.zip"
severity = 0.7
start_date = "2021-01-02T12:00:00Z"
end_date = "2021-01-03T00:00:00Z"
series, raw_result, top_anomaly = draw(data_source, local_data_path, result_id, severity, start_date, end_date)
show_contribution(series, raw_result, top_anomaly)<jupyter_output><empty_output><jupyter_text>Export model by modelid```http[GET] https://{endpoint}/multivariate/models/{model_id}/export```you can take the model zip file to other env to do inference. Sample - Request```jsonheader ={"Content-Type": "application/json", "Ocp-Apim-Subscription-Key": "subscription_key"}``` - Response```jsonzip file``` - Error Response ```json{"code": "string","message": "string"} ```<jupyter_code>res = requests.get(API_EXPORT.format(endpoint=ENDPOINT, model_id=model_id), headers=HEADERS)
assert res.status_code == 200, f"Error occured. Error message: {res.content}"<jupyter_output><empty_output><jupyter_text>Delete model by modelid```http[DELETE] https://{endpoint}/multivariate/models/{model_id}``` Sample - Request```jsonheader ={"Content-Type": "application/json", "Ocp-Apim-Subscription-Key": "subscription_key"}``` - Response```jsonresponse={}``` - Error Response ```json{"code": "string","message": "string"} ```<jupyter_code>res = requests.delete(API_DELETE.format(endpoint=ENDPOINT, model_id=model_id), headers=HEADERS)
assert res.status_code == 204, f"Error occured. Error message: {res.content}"
print(res.content)<jupyter_output><empty_output><jupyter_text>Model list has been updated.<jupyter_code>res = requests.get(API_MODEL.format(endpoint=ENDPOINT), headers=HEADERS)
assert res.status_code == 200, f"Error occured. Error message: {res.content}"
print(res.content)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14994808918771937, 'fraction_numerical': 0.0683245167350571, 'mean_word_length': 4.128803245436106, 'pattern_counts': {'":': 137, '<': 45, '<?xml version=': 0, '>': 43, 'https://': 16, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '206922', 'license': 'MIT', 'n_tokens_mistral': 7584, 'n_tokens_neox': 6588, 'n_words': 1423, 'path': 'ipython-notebook/Multivariate API Demo Notebook.ipynb', 'repo_name': 'moreOver0/AnomalyDetector', 'size': '20227'}