Adaptive Model Pooling for Online Deep Anomaly Detection from a Complex Evolving Data Stream
Online anomaly detection from a data stream is critical for the safety and
security of many applications but is facing severe challenges due to complex
and evolving data streams from IoT devices and cloud-based infrastructures.
Unfortunately, existing approaches fall too short for these challenges; online
anomaly detection methods bear the burden of handling the complexity while
offline deep anomaly detection methods suffer from the evolving data
distribution. This paper presents a framework for online deep anomaly
detection, ARCUS, which can be instantiated with any autoencoder-based deep
anomaly detection methods. It handles the complex and evolving data streams
using an adaptive model pooling approach with two novel techniques:
concept-driven inference and drift-aware model pool update; the former detects
anomalies with a combination of models most appropriate for the complexity, and
the latter adapts the model pool dynamically to fit the evolving data streams.
In comprehensive experiments with ten data sets which are both high-dimensional
and concept-drifted, ARCUS improved the anomaly detection accuracy of the
streaming variants of state-of-the-art autoencoder-based methods and that of
the state-of-the-art streaming anomaly detection methods by up to 22% and 37%,
respectively.