<jupyter_start><jupyter_text>Run Backtesting on Historical Data for Amazon Lookout for MetricsAmazon Lookout for Metrics also supports backtesting against your historical information and in this notebook we will demonstrate this functionality on the same e-commerce data. Once the backtesting job has completed you can see all of the anomalies that Amazon Lookout for Metrics detected in the last 20% of your historical data. From here you can begin to unpack the kinds of results you will see from Amazon Lookout for Metrics in the future when you start streaming in new data. **NOTE YOU MUST CREATE A NEW DETECTOR TO LEVERAGE REAL TIME DATA. BACKTESTING IS ONLY FOR EXPLORATION.**This notebook assumes that you already completed the prerequisites in the `1.PrereqSetupPackages.ipynb` and `2.PrereqSetupData.ipynb`. If you have not, go back and complete those first, then give backtesting a whirl while you are waiting for the real time anomalies to alert you!First restore the variables from the previous notebook and then import the libraries needed.<jupyter_code>%store -r<jupyter_output><empty_output><jupyter_text>Just as in the last notebook, connect to AWS through the SDK.<jupyter_code>import boto3
import utility
L4M = boto3.client( "lookoutmetrics", region_name="us-west-2" )<jupyter_output><empty_output><jupyter_text>Creating A DetectorThis step is identical to the one in the previous notebook. Here, we're simply creating a diffrent `detector` that we will use for backtesting.<jupyter_code>project = "initial-lookoutmetrics-backtesting"

frequency = "PT1H" # one of 'P1D', 'PT1H', 'PT10M' and 'PT5M'
response = L4M.create_anomaly_detector( 
    AnomalyDetectorName = project + "-detector",
    AnomalyDetectorDescription = "My Detector",
    AnomalyDetectorConfig = {
        "AnomalyDetectorFrequency" : "PT1H",
    },
)

anomaly_detector_arn = response["AnomalyDetectorArn"]
anomaly_detector_arn<jupyter_output><empty_output><jupyter_text>Define MetricsAfter creating a detector, we need to point it to the s3 path for our backtest data. This process is also, similar to the one from the previous notebook.First, let's create a role that can work with the Amazon Lookout for Metrics service:<jupyter_code>role_name = "L4MTestRole"
role_arn = utility.get_or_create_iam_role(role_name)<jupyter_output><empty_output><jupyter_text>Now, let's create a metric set for our detector that point to the backtest data in S3:<jupyter_code>s3_path_backtest = 's3://'+ s3_bucket + '/ecommerce/backtest/'
s3_path_backtest
params = {
    "AnomalyDetectorArn": anomaly_detector_arn,
    "MetricSetName" : project + '-metric-set-1',
    "MetricList" : [
        {
            "MetricName" : "views",
            "AggregationFunction" : "AVG",
        },
        {
            "MetricName" : "revenue",
            "AggregationFunction" : "SUM",
        },
    ],

    "DimensionList" : [ "platform", "marketplace" ],

    "TimestampColumn" : {
        "ColumnName" : "timestamp",
        "ColumnFormat" : "yyyy-MM-dd HH:mm:ss",
    },

    #"Delay" : 120, # seconds the detector will wait before attempting to read latest data per current time and detection frequency below
    "MetricSetFrequency" : frequency,

    "MetricSource" : {
        "S3SourceConfig": {
            "RoleArn" : role_arn,
            "HistoricalDataPathList": [
                s3_path_backtest,
            ],
#            "TemplatedPathList": [
#                s3_path_format,
#            ],

            "FileFormatDescriptor" : {
                "CsvFormatDescriptor" : {
                    "FileCompression" : "NONE",
                    "Charset" : "UTF-8",
                    "ContainsHeader" : True,
                    "Delimiter" : ",",
#                    "HeaderList" : [
#                        "platform",
#                        "marketplace",
#                        "timestamp",
#                        "views",
#                        "revenue"
#                    ],
                    "QuoteSymbol" : '"'
                },
            }
        }
    },
}

params
response = L4M.create_metric_set( ** params )

metric_set_arn = response["MetricSetArn"]
metric_set_arn<jupyter_output><empty_output><jupyter_text>Activate the Detector and execute backtestingNow that the MetricSet has been specified, we are ready to start backtesting, that's done by activating the back test anomaly detector.<jupyter_code>L4M.back_test_anomaly_detector(AnomalyDetectorArn = anomaly_detector_arn)
utility.wait_anomaly_detector( L4M, anomaly_detector_arn )<jupyter_output><empty_output><jupyter_text>Validate resultsAfter backtesting is finished, you can visually validate the historical anomalies via the console or inspect the results by running the following commands:<jupyter_code>anomaly_groups = []
next_token = None

while True:    
    params = {
        "AnomalyDetectorArn" : anomaly_detector_arn,
        "SensitivityThreshold" : 50,
        "MaxResults" : 100,
    }
    
    if next_token:
        params["NextToken"] = next_token
    
    response = L4M.list_anomaly_group_summaries( **params )
    stats = response["AnomalyGroupStatistics"]
    
    anomaly_groups += response["AnomalyGroupSummaryList"]
    
    if "NextToken" in response:
        next_token = response["NextToken"]
        continue
    break

stats
anomaly_groups<jupyter_output><empty_output><jupyter_text>And to dive even deeper into a specific anomaly group, simlpy choose your anomaly group of interest and drill down to it's time-series. Here we will use the first anomaly group in the list.<jupyter_code>anomaly_group_id = anomaly_groups[0]["AnomalyGroupId"]

response = L4M.list_anomaly_group_time_series(AnomalyDetectorArn = anomaly_detector_arn,
                                   AnomalyGroupId=anomaly_group_id,
                                   MetricName='views',
                                   MaxResults=10,
                                  )
response<jupyter_output><empty_output><jupyter_text>Clean up resourcesOnce we have completed backtesting, we can start to cleanup the resources we created. Before cleaning up, you can visit the "Anomalies" page of the Amazon Lookout for Metrics console, and visually check the detected anomalies.<jupyter_code>answer = input("Delete resources? (y/n)")
if answer=="y":
    delete_resources = True
else:
    delete_resources = False
    
if delete_resources:
    #L4M.delete_anomaly_detector( AnomalyDetectorArn = anomaly_detector_arn )
    utility.wait_delete_anomaly_detector( L4M, anomaly_detector_arn )
else:
    print("Not deteleting resources.")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08100347589542088, 'fraction_numerical': 0.0069517908417711955, 'mean_word_length': 3.188607594936709, 'pattern_counts': {'":': 5, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '556011', 'license': 'MIT-0', 'n_tokens_mistral': 1993, 'n_tokens_neox': 1867, 'n_words': 558, 'path': 'getting_started/4.BacktestingWithHistoricalData.ipynb', 'repo_name': 'dast1/amazon-lookout-for-metrics-samples', 'size': '6617'}