<jupyter_start><jupyter_text>Anomaly Detection in HTTP LogsThis sample notebook demonstrates working with HTTP request logs data stored in BigQuery.Google Cloud Logging in the Google Cloud Platform makes it simple to export HTTP request logs from AppEngine applications directly into BigQuery for further analysis. This log data includes information such as the requested resource, HTTP status code, etc. One possible use of these logs is to mine them as they are collected to detect anomalies in response latency, since this can be a signal for some unexpected deployment issue.The sample data used in this notebook is similar to AppEngine logs. It represents anonymized HTTP logs from a hypothetical application.Related Links:* [Cloud Logging](https://cloud.google.com/logging/docs/)* [BigQuery](https://cloud.google.com/bigquery/what-is-bigquery)* [Pandas](http://pandas.pydata.org/) for data analysis* [Matplotlib](http://matplotlib.org/) for data visualization<jupyter_code>from __future__ import division
import google.datalab.bigquery as bq
import matplotlib.pyplot as plot
import numpy as np<jupyter_output><empty_output><jupyter_text>Understanding the Logs DataIt's helpful to inspect the dataset, the schema, and a sample of the data we're working with. Usually, logs are captured as multiple tables within a dataset, with new tables added per time window (such as daily logs).<jupyter_code>%bq tables list --project cloud-datalab-samples --dataset httplogs
%bq tables describe -n cloud-datalab-samples.httplogs.logs_20140615
%%bq query -n logs
SELECT timestamp, latency, status, method, endpoint
FROM `cloud-datalab-samples.httplogs.logs_20140615`
ORDER by timestamp
%%bq sample --query logs --count 7<jupyter_output><empty_output><jupyter_text>Transforming Logs into a Time SeriesWe're going to build a timeseries over latency. In order to make it a useful metric, we'll look at 99th percentile latency of requests within a fixed 5min window using this SQL query issued to BigQuery (notice the grouping over a truncated timestamp, and quantile aggregation).<jupyter_code>%%bq query -n timeseries
SELECT DIV(UNIX_SECONDS(timestamp), 300) * 300 AS five_minute_window,
       APPROX_QUANTILES(latency, 99)[SAFE_ORDINAL(99)] as latency
FROM `cloud-datalab-samples.httplogs.logs_20140615`
WHERE endpoint = 'Recent'
GROUP BY five_minute_window
ORDER by five_minute_window
%%bq sample --query timeseries --count 10<jupyter_output><empty_output><jupyter_text>Visualizing the Time Series DataIts helpful to visualize the data. In order to visualize this timeseries, we'll use python, pandas and matplotlib.<jupyter_code># Execute and convert the results to a Pandas dataframe
timeseries_df = timeseries.execute(output_options=bq.QueryOutput.dataframe()).result()

timeseries_values = timeseries_df['latency'].values
timeseries_len = len(timeseries_values)
plot.plot(np.array(range(timeseries_len)), timeseries_values)
plot.yscale('log')
plot.grid()<jupyter_output><empty_output><jupyter_text>Anomaly Detection A visual inspection of the chart, above, highlights the obvious anomalies, but we want to construct something that can inspect the data, learn from it, and detect anomalies in an automated way.  Anomaly DetectorThe code below is a simple anomaly detector created for purposes of a sample. It uses a simple algorithm that tracks mean and standard deviation. You can give it a value indicating the number of instances is uses to train a model before it starts detecting anomalies. As new values are passed to it, it continues to update the model (to account for slowly evolving changes) and report anomalies. Any value that is off from the mean by 3x the standard deviation is considered as an anomaly.<jupyter_code>class AnomalyDetector(object):

  def __init__(self, window = 10):
    self._index = 0
    self._window = window
    self._values = np.zeros(window)
    self._valuesSq = np.zeros(window)
    self._mean = 0
    self._variance = 0
    self._count = 0

  def observation(self, value):
    anomaly = False

    threshold = 3 * np.sqrt(self._variance)
    if self._count > self._window:
      if value > self._mean + threshold:
        value = self._mean + threshold
        anomaly = True
      elif value < self._mean - threshold:
        value = self._mean - threshold
        anomaly = True
    else:
      self._count += 1

    prev_value = self._values[self._index]
    self._values[self._index] = value
    self._valuesSq[self._index] = value ** 2
    self._index = (self._index + 1) % self._window

    self._mean = self._mean - prev_value / self._window + value / self._window
    self._variance = sum(self._valuesSq) / self._window - (self._mean ** 2)

    return anomaly, self._mean<jupyter_output><empty_output><jupyter_text>With the anomaly detector implemented, let's run the timeseries through it to collect any anomalies and the expected mean along each point.<jupyter_code>anomalies = np.zeros(timeseries_len)
means = np.zeros(timeseries_len)

anomaly_detector = AnomalyDetector(36)

for i, value in enumerate(timeseries_values):
  anomaly, mean = anomaly_detector.observation(value)
    
  anomalies[i] = anomaly
  means[i] = mean<jupyter_output><empty_output><jupyter_text>Then, plot the same time series, but overlay the anomalies and the mean values as well.<jupyter_code>ticks = np.array(range(timeseries_len))

plot.plot(ticks, timeseries_values)
plot.plot(ticks[anomalies == 1], timeseries_values[anomalies == 1], 'ro')
plot.plot(ticks, means, 'g', linewidth = 1)
plot.yscale('log')
plot.grid()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07989228007181329, 'fraction_numerical': 0.010233393177737882, 'mean_word_length': 4.951923076923077, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 31, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '93621', 'license': 'Apache-2.0', 'n_tokens_mistral': 1691, 'n_tokens_neox': 1571, 'n_words': 590, 'path': 'docs/samples/Anomaly Detection in HTTP Logs.ipynb', 'repo_name': 'JeffersonH44/datalab-r', 'size': '5570'}