FIELD

The present disclosure relates to automated analysis of observed metrics and more particularly to computerized determination of expected metric performance.

BACKGROUND

Modest computing systems can have hundreds or even thousands of processor cores, memory arrays, storage arrays, networking ports, and additional peripherals. In large-scale computing systems, such as a data center or supercomputer, the number of processor cores can be in the hundreds of thousands to millions. Each hardware component may have a number of associated parameters, such as clock speed, temperature, idle time, etc. Some of these parameters may be reported and/or measured by the computing system itself. Other parameters may be monitored by an associated monitoring system.

These parameters are referred to in this disclosure as metrics and may be defined at a component level, such as available space on a given magnetic disk, or may be at a subsystem level, such as amount of available storage space in a storage area network. Metrics may also be defined at a system level, such as number of transactions per second in the database, delay in returning results for a query, or the length of execution time of a particular function. A monitoring system for a large computing system may measure and/or collect thousands, millions, or even billions of time-series metrics (that is, metrics that are measured repeatedly over time). Monitoring metrics allows for problems to be quickly identified and resolved, hopefully before problems negatively affect business outcomes. Negative effects may include alienating users, missing revenue, decreasing productivity, etc.

Currently, problems are detected by skilled system administrators who manually create rules to generate alerts for specific metrics. For example, an administrator may set a threshold for available disk space such that an alert will be generated when available disk space decreases below 10% of total disk space. For many metrics, the “correct” threshold may not be known a priori. Instead, the administrator may have to observe the metric over time and infer a threshold based on the historical metric values.

Administrators may watch scores of metrics, such as in a dashboard, and use experience and intuition to determine if any of the metrics indicate the onset of a problem. However, regardless of how many computer screens are used, the number of metrics that can be visually tracked is limited.

Big data principles have been applied to the problem of monitoring systems to address the limitations of visual and programmatic oversight by human administrators. Automated processes may evaluate every single metric, a significant advance compared to the tiny fraction that a human administrator can review, and determine normal historical behavior for each metric.

The background description provided here is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

SUMMARY

A system includes a metric data store configured to receive and store a time-series of values of a first metric. The first metric corresponds to a measurement of computing system performance. The system also includes a seasonal trend identification module configured to determine a periodicity profile for the first metric based on the time-series of values and a modeling module configured to generate an autoregressive moving average (ARMA) model of the time-series of values based on the periodicity profile for the first metric. The modeling module includes a seasonal model module configured to generate a first model that represents a seasonal characteristic of the time-series of values, a non-seasonal model module configured to generate a second model that represents a non-seasonal characteristic of the time-series of values, and a combination module configured to generate a third model based on the first model and the second model. The modeling module is configured to, in response to the periodicity profile indicating that a first periodicity profile describes the time-series of values, output the third model as the ARMA model of the time-series of values. The system further includes an envelope determination module configured to determine an operating envelope that indicates normal behavior of the first metric based on the ARMA model of the time-series of values, an anomaly identification module configured to identify an anomaly in present values of the first metric in response to the present values deviating outside the operating envelope, and an alert module configured to selectively (i) generate an alert message in response to the identified anomaly and (ii) transmit the alert message to a designated user over a network interface.

In other features, the modeling module includes an autocorrelation module configured to calculate a downsampled autocorrelation function (ACF) based on the time-series of values and the periodicity profile. The seasonal model module is configured to generate the first model based on the downsampled ACF. In further features, a rate of the downsampled ACF is proportional to the first periodicity profile.

In yet further features, calculating the downsampled ACF includes calculating an ACF at an integer multiple of a period length associated with the first periodicity profile. In other features, the modeling module includes a residual calculation unit configured to determine a residual of the first model. The non-seasonal model module is configured to generate the second model based on the determined residual of the first model.

In yet other features, the modeling module includes an outlier removal module configured to filter the time-series of values to remove irregularities. Filtering the time-series of values includes discarding values that are either greater than or less than a preceding value by a predetermined threshold. The outlier removal module if further configured to provide the filtered time-series of values to the seasonal trend identification module and the modeling module.

In further features, the modeling module includes a time-series module configured to interpolate and detrend the time-series of values prior to the filtering of the time-series of values by the outlier removal module. In other features, the system includes an anomaly removal module configured to, prior to processing by the modeling module, remove values of the first metric associated with the anomaly identified by the anomaly identification module from the stored time-series of values in the metric data store.

In yet other features, the modeling module includes a standard model module configured to generate a fourth model that represents the time-series of values. The modeling module is configured to, in response to the periodicity profile indicating that no periodicity profile describes the time-series of values, output the fourth model as the ARMA model of the time-series of values. In further features, the fourth model represents the non-seasonal characteristic of the time-series of values. The standard model module is configured to determine parameters of the fourth model based only on the time-series of values.

A computer-implemented monitoring method includes receiving and storing a time-series of values of a first metric that corresponds to a measurement of computing system performance, determining a periodicity profile for the first metric based on the time-series of values, and generating an autoregressive moving average (ARMA) model of the time-series of values based on the periodicity profile for the first metric. Generating the ARMA model of the time-series values includes generating a first model that represents a seasonal characteristic of the time-series of values, generating a second model that represents a non-seasonal characteristic of the time-series of values, generating a third model based on the first model and the second model, and in response to the periodicity profile indicating that a first periodicity profile describes the time-series of values, outputting the third model as the ARMA model of the time-series of values. The method further includes determining an operating envelope that indicates normal behavior of the first metric based on the ARMA model of the time-series of values, identifying an anomaly in present values of the first metric in response to the present values deviating outside the operating envelope, and selectively (i) generating an alert message in response to the identified anomaly and (ii) transmitting the alert message to a designated user over a network interface.

In other features, generating the first model includes (i) calculating a downsampled autocorrelation function (ACF) based on the time-series of values and the periodicity profile and (ii) generating the first model based on the downsampled ACF. In further features, a rate of the downsampled ACF is proportional to the first periodicity profile. In yet further features, calculating the downsampled ACF includes calculating an ACF at an integer multiple of a period length associated with the first periodicity profile.

In other features, generating the second model includes (i) determining a residual of the first model and (ii) generating the second model based on the determined residual of the first model. In yet other features, the method includes filtering the time-series of values to remove irregularities by discarding values that are either greater than or lesser than a preceding value by a predetermined threshold. Generating the first model includes calculating a downsampled ACF based on the filtered time-series of values and the periodicity profile.

In further features, the method includes interpolating and detrending the time-series of values prior to the filtering of the time-series of values. In other features, the method includes removing values of the first metric associated with the identified anomaly from the stored time-series of values.

In yet other features, generating the ARMA model of the time-series values includes generating a fourth model that represents a non-seasonal characteristic of the time-series of values. In response to the periodicity profile indicating that no periodicity profile describes the time-series of values, outputting the fourth model as the ARMA model of the time-series of values. In other features, the method includes in response to receiving new time-series values of the first metric, (i) generating an updated ARMA model and (ii) determining an updated operating envelope that indicates normal behavior of the first metric based on the updated ARMA model.

Further areas of applicability of the present disclosure will become apparent from the detailed description, the claims, and the drawings. The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure.

