The monitoring and management of high-volume feature-rich traffic in large networks offers significant challenges in storage, transmission and computational costs. The predominant approach to reducing these costs is based on performing a linear mapping of the data to a low-dimensional subspace such that a certain large percentage of the variance in the data is preserved in the low-dimensional representation. This variance-based subspace approach to dimensionality reduction forces a fixed choice of the number of dimensions, is not responsive to real-time shifts in observed traffic patterns, and is vulnerable to normal traffic spoofing. Based on theoretical insights proved in this paper, we propose a new distance-based approach to dimensionality reduction motivated by the fact that the real-time structural differences between the covariance matrices of the observed and the normal traffic is more relevant to anomaly detection than the structure of the training data alone. Our approach, called the distance-based subspace method, allows a different number of reduced dimensions in different time windows and arrives at only the number of dimensions necessary for effective anomaly detection. We present centralized and distributed versions of our algorithm and, using simulation on real traffic traces, demonstrate the qualitative and quantitative advantages of the distance-based subspace approach.
In other related work, PCA-based methods have been decentralized for a variety of purposes including anomaly detection @cite_21 @cite_8 @cite_19 @cite_7 @cite_0 . A distributed framework for PCA is proposed in @cite_7 to achieve accurate detection of network anomalies through monitoring of only the local data. A distributed implementation of PCA is developed for decomposable Gaussian graphical models in @cite_0 to allow decentralized anomaly detection in backbone networks. Distributed gossip algorithms using only local communication for subspace estimation have been used in the context of sensor networks @cite_26 @cite_10 @cite_23 . Our work in this paper extends the distributed average consensus protocol proposed in @cite_23 to estimate the principal subspace for the context of anomaly detection in network traffic data.
@cite_26 @cite_7 @cite_8 @cite_21 @cite_0 @cite_19 @cite_23 @cite_10
2158339409 2171604933 2111161380 2152426255
In this paper, we consider principal component analysis (PCA) in decomposable Gaussian graphical models. We exploit the prior information in these models in order to distribute PCA computation. For this purpose, we reformulate the PCA problem in the sparse inverse covariance (concentration) domain and address the global eigenvalue problem by solving a sequence of local eigenvalue problems in each of the cliques of the decomposable graph. We illustrate our methodology in the context of decentralized anomaly detection in the Abilene backbone network. Based on the topology of the network, we propose an approximate statistical graphical model and distribute the computation of PCA. We consider the problem of network anomaly detection in large distributed systems. In this setting, Principal Component Analysis (PCA) has been proposed as a method for discovering anomalies by continuously tracking the projection of the data onto a residual subspace. This method was shown to work well empirically in highly aggregated networks, that is, those with a limited number of large nodes and at coarse time scales. This approach, however, has scalability limitations. To overcome these limitations, we develop a PCA-based anomaly detector in which adaptive local data filters send to a coordinator just enough data to enable accurate global detection. Our method is based on a stochastic matrix perturbation analysis that characterizes the tradeoff between the accuracy of anomaly detection and the amount of data communicated over the network. We propose the use of commute distance, a random walk metric, to discover anomalies in network traffic data. The commute distance based anomaly detection approach has several advantages over Principal Component Analysis (PCA), which is the method of choice for this task: (i) It generalizes both distance and density based anomaly detection techniques while PCA is primarily distance-based (ii) It is agnostic about the underlying data distribution, while PCA is based on the assumption that data follows a Gaussian distribution and (iii) It is more robust compared to PCA, i.e., a perturbation of the underlying data or changes in parameters used will have a less significant effect on the output of it than PCA. Experiments and analysis on simulated and real datasets are used to validate our claims. The increasing number of network attacks causes growing problems for network operators and users. Thus, detecting anomalous traffic is of primary interest in IP networks management. In this paper we address the problem considering a method based on PCA for detecting network anomalies. In more detail, we present a new technique that extends the state of the art in PCA based anomaly detection. Indeed, by means of the Kullback-Leibler divergence we are able to obtain great improvements with respect to the performance of the "classical" approach. Moreover we also introduce a method for identifying the flows responsible for an anomaly detected at the aggregated level. The performance analysis, presented in this paper, demonstrates the effectiveness of the proposed method.