Much of the worlds data is streaming, time-series data, where anomalies give significant information in critical situations. Yet detecting anomalies in streaming data is a difficult task, requiring detectors to process data in real-time, and learn while simultaneously making predictions. We present a novel anomaly detection technique based on an on-line sequence memory algorithm called Hierarchical Temporal Memory (HTM). We show results from a live application that detects anomalies in financial metrics in real-time. We also test the algorithm on NAB, a published benchmark for real-time anomaly detection, where our algorithm achieves best-in-class results.
Anomaly detection is being regarded as an unsupervised learning task as anomalies stem from adversarial or unlikely events with unknown distributions. However, the predictive performance of purely unsupervised anomaly detection often fails to match the required detection rates in many tasks and there exists a need for labeled data to guide the model generation. Our first contribution shows that classical semi-supervised approaches, originating from a supervised classifier, are inappropriate and hardly detect new and unknown anomalies. We argue that semi-supervised anomaly detection needs to ground on the unsupervised learning paradigm and devise a novel algorithm that meets this requirement. Although being intrinsically non-convex, we further show that the optimization problem has a convex equivalent under relatively mild assumptions. Additionally, we propose an active learning strategy to automatically filter candidates for labeling. In an empirical study on network intrusion detection data, we observe that the proposed learning methodology requires much less labeled data than the state-of-the-art, while achieving higher detection accuracies. THE detection of outliers has mainly been considered for single random samples, although some recent work deals also with standard linear models; see, for example, Anscombe (1960) and Kruskal (1960). Essentially similar problems arise in time series (Burman, 1965) but there seems no published work taking into account correlations between successive observations. In the past, the search for outliers in time series has been based on the assumption that the observations are independently and identically normally distributed. This assumption leads to analyses which will be called random sample procedures. Two types of outlier that may occur in a time series are considered in this paper. A Type I outlier corresponds to the situation in which a gross error of observation or recording error affects a single observation. A Type II outlier corresponds to the situation in which a single "innovation" is extreme. This will affect not only the particular observation but also subsequent observations. For the development of tests and the interpretation of outliers, it is necessary to distinguish among the types of outlier likely to be contained in the process. The present approach is based on four possible formulations of the problem: the outliers are all of Type I; the outliers are all of Type II; the outliers are all of the same type but whether they are of Type I or of Type II is not known; and the outliers are a mixture of the two types. Since more practical solutions than those given by likelihood ratio methods are often obtained from simplifications of likelihood ratio criteria, some simpler criteria are derived. These criteria are of the form &2a, where A is the estimated error in the observation tested and ^ is the estimated standard error of A. Throughout this paper, trend and seasonal components are assumed either negligible or to have been eliminated. The method adopted to remove these components might affect the results in some way.