Interpretable Anomaly Detection for Monitoring of High Performance Computing Systems
The Trinity supercomputer at Los Alamos National Laboratory collects data from a large number of sensors, producing a massive stream of data on the order of terabytes per day. Previous supercomputers relied on humans skimming this data for incorrect behaviors, but this approach is not feasible with current machines, or future exascale class machines, which will produce approximately 25 times more data than Trinity. Instead, an automated system that can scale to the large data stream will be required. We present Interpretable and Interactive Classifier Adjusted Density Estimation with Temporal components (iCADET) an extension of classifieradjusted density estimation, for anomaly detection and exploratory data analysis. Our methods take advantage of the interpretability of random forest classifiers to present explanations to an analyst regarding why the algorithm ranks certain data points as more likely to be anomalous. We also include temporal relational features in our analysis, and provide a mechanism for analysts to modify the anomaly detection method in real time in order to silence uninteresting anomalies. We find that iCADET successfully recovers a variety of synthetic anomalies injected into the Trinity supercomputer System Environment Data Collections (SEDC) data set. We consult domain experts to confirm that our methods uncover behaviors in the un altered data set that are informative for systems analysts and troubleshooters.