The traffic behavior of University of Louisville network with the interconnected backbone routers and the number of Virtual Local Area Network (VLAN) subnets is investigated using the Random Matrix Theory (RMT) approach. We employ the system of equal interval time series of traffic counts at all router to router and router to subnet connections as a representation of the inter-VLAN traffic. The cross-correlation matrix C of the traffic rate changes between different traffic time series is calculated and tested against null-hypothesis of random interactions. The majority of the eigenvalues i of matrix C fall within the bounds predicted by the RMT for the eigenvalues of random correlation matrices. The distribution of eigenvalues and eigenvectors outside of the RMT bounds displays prominent and systematic deviations from the RMT predictions. Moreover, these deviations are stable in time. The method we use provides a unique possibility to accomplish three concurrent tasks of traffic analysis. The method verifies the uncongested state of the network, by establishing the profile of random interactions. It recognizes the system-specific large-scale interactions, by establishing the profile of stable in time non-random interactions. Finally, by looking into the eigenstatistics we are able to detect and allocate anomalies of network traffic interactions.
The urgent need for a network-wide, scalable approach to the problem of healthy network traffic profile creation is expressed in works of @cite_20 @cite_15 @cite_23 @cite_10 @cite_0 @cite_25 . There are several studies with the promising results, which demonstrate that the traffic anomalous events cause the temporal changes in statistical properties of traffic features. Lakhina, Crovella and Diot presented the characterization of the network-wide anomalies of the traffic flows. The authors studied three different types of traffic flows and fused the information from flow measurements taken throughout the entire network. They obtained and classified a different set of anomalies for different traffic types using the subspace method @cite_15 .
@cite_0 @cite_23 @cite_15 @cite_10 @cite_25 @cite_20
2119742497 2164210932 2009182597 2144936818 40609341 2949350626 2152426255 2273128077 2043890595 2039423859 1484844389 2090888999 2057165483 2102283838 2094331586 2884121921 2055654888 2158018483 2009439582 2963024417
We introduce an Internet traffic anomaly detection mechanism based on large deviations results for empirical measures. Using past traffic traces we characterize network traffic during various time-of-day intervals, assuming that it is anomaly-free. We present two different approaches to characterize traffic: (i) a model-free approach based on the method of types and Sanov's theorem, and (ii) a model-based approach modeling traffic using a Markov modulated process. Using these characterizations as a reference we continuously monitor traffic and employ large deviations and decision theory results to ldquocomparerdquo the empirical measure of the monitored traffic with the corresponding reference characterization, thus, identifying traffic anomalies in real-time. Our experimental results show that applying our methodology (even short-lived) anomalies are identified within a small number of observations. Throughout, we compare the two approaches presenting their advantages and disadvantages to identify and classify temporal network anomalies. We also demonstrate how our framework can be used to monitor traffic from multiple network elements in order to identify both spatial and temporal anomalies. We validate our techniques by analyzing real traffic traces with time-stamped anomalies. The increasing practicality of large-scale flow capture makes it possible to conceive of traffic analysis methods that detect and identify a large and diverse set of anomalies. However the challenge of effectively analyzing this massive data source for anomaly diagnosis is as yet unmet. We argue that the distributions of packet features (IP addresses and ports) observed in flow traces reveals both the presence and the structure of a wide range of anomalies. Using entropy as a summarization tool, we show that the analysis of feature distributions leads to significant advances on two fronts: (1) it enables highly sensitive detection of a wide range of anomalies, augmenting detections by volume-based methods, and (2) it enables automatic classification of anomalies via unsupervised learning. We show that using feature distributions, anomalies naturally fall into distinct and meaningful clusters. These clusters can be used to automatically classify anomalies and to uncover new anomaly types. We validate our claims on data from two backbone networks (Abilene and Geant) and conclude that feature distributions show promise as a key element of a fairly general network anomaly diagnosis framework. Abstract This paper proposes a mathematical justification of the phenomenon of extreme congestion at a very limited number of nodes in very large networks. It is argued that this phenomenon occurs as a combination of the negative curvature property of the network together with minimum-length routing. More specifically, it is shown that in a large n-dimensional hyperbolic ball B of radius R viewed as a roughly similar model of a Gromov hyperbolic network, the proportion of traffic paths transiting through a small ball near the center is Θ(1), whereas in a Euclidean ball, the same proportion scales as Θ(1 R n−1). This discrepancy persists for the traffic load, which at the center of the hyperbolic ball scales as volume2(B), whereas the same traffic load scales as volume1+1 n (B) in the Euclidean ball. This provides a theoretical justification of the experimental exponent discrepancy observed by Narayan and Saniee between traffic loads in Gromov-hyperbolic networks from the Rocketfuel database and synthetic ... Detecting and understanding anomalies in IP networks is an open and ill-defined problem. Toward this end, we have recently proposed the subspace method for anomaly diagnosis. In this paper we present the first large-scale exploration of the power of the subspace method when applied to flow traffic. An important aspect of this approach is that it fuses information from flow measurements taken throughout a network. We apply the subspace method to three different types of sampled flow traffic in a large academic network: multivariate timeseries of byte counts, packet counts, and IP-flow counts. We show that each traffic type brings into focus a different set of anomalies via the subspace method. We illustrate and classify the set of anomalies detected. We find that almost all of the anomalies detected represent events of interest to network operators. Furthermore, the anomalies span a remarkably wide spectrum of event types, including denial of service attacks (single-source and distributed), flash crowds, port scanning, downstream traffic engineering, high-rate flows, worm propagation, and network outage. The traffic networks reflect the pulse and structure of a city and shows some dynamic characteristic. Previous research in mining structure from networks mostly focus on static networks and fail to exploit the temporal patterns. In this paper, we aim to solve the problem of discovering the urban spatio-temporal structure from time-evolving traffic networks. We model the time-evolving traffic networks into a 3-order tensor, each element of which indicates the volume of traffic from i-th origin area to j-th destination area in k-th time domain. Considering traffic data and urban contextual knowledge together, we propose a regularized Non-negative Tucker Decomposition (rNTD) method, which discovers the spatial clusters, temporal patterns and relations among them simultaneously. Abundant experiments are conducted in a large dataset collected from Beijing. Results show that our method outperforms the baseline method. This paper introduces a novel graph-analytic approach for detecting anomalies in network flow data called GraphPrints. Building on foundational network-mining techniques, our method represents time slices of traffic as a graph, then counts graphlets -- small induced subgraphs that describe local topology. By performing outlier detection on the sequence of graphlet counts, anomalous intervals of traffic are identified, and furthermore, individual IPs experiencing abnormal behavior are singled-out. Initial testing of GraphPrints is performed on real network data with an implanted anomaly. Evaluation shows false positive rates bounded by 2.84 at the time-interval level, and 0.05 at the IP-level with 100 true positive rates at both. The increasing number of network attacks causes growing problems for network operators and users. Thus, detecting anomalous traffic is of primary interest in IP networks management. In this paper we address the problem considering a method based on PCA for detecting network anomalies. In more detail, we present a new technique that extends the state of the art in PCA based anomaly detection. Indeed, by means of the Kullback-Leibler divergence we are able to obtain great improvements with respect to the performance of the "classical" approach. Moreover we also introduce a method for identifying the flows responsible for an anomaly detected at the aggregated level. The performance analysis, presented in this paper, demonstrates the effectiveness of the proposed method. Internet traffic contains a rich set of periodic patterns. Examples include regular packet transmissions along bottleneck links, periodic routing information exchange, and periodicities inside Denial-of-Service attack streams. Analyzing such periodic patterns has wide applications, including a better understanding of network traffic dynamics, diagnosis of network anomalies, and detection of Denial-of-Service attacks. However, current understanding of periodic behavior in aggregate traffic is quite limited. Many previous approaches often analyze traffic on a per-flow basis, and are not suited to analyze high speed aggregate traffic. In addition, a number of approaches only indicate that they may reveal periodic patterns, but fall short of proposing automatic detection algorithms and quantitatively evaluating their performance. This thesis explores the application of spectral and statistical methods to detect periodic patterns in Internet traffic. In our approach we first apply spectral techniques to obtain the traffic spectrum, and then use algorithms based on rigorous statistical methods to automatically detect periodic patterns from the traffic spectrum. One salient feature of our approach is that it operates at the aggregate traffic level and does not require flow separation. We first carry out controlled lab experiments to demonstrate the spectral characteristics of various periodic patterns. We then propose four non-parametric detection algorithms and evaluate their performance using real-world Internet traffic. Results show that one of them, the Top-Frequency Algorithm, is the best choice in terms of detection performance and algorithm simplicity. It can provide excellent accuracy (up to 95 ) for detecting the periodic pattern caused a bottleneck link even when the traffic through the bottleneck accounts for less than 10 of the aggregate traffic observed at the monitoring point. We also investigate two extensions to our algorithms. The first one is to utilize harmonics, and the second one is to have parametric detection that considers the variation of traffic spectra according to other factors, such as traffic volume. Evaluation results show that we can get significant improvement by considering harmonics for traffic similar to the training data and marginal improvement by considering traffic volume for parametric detection. The systematic study of large-scale networks has unveiled the ubiquitous presence of connectivity patterns characterized by large-scale heterogeneities and unbounded statistical fluctuations. These features affect dramatically the behavior of the diffusion processes occurring on networks, determining the ensuing statistical properties of their evolution pattern and dynamics. In this article, we present a stochastic computational framework for the forecast of global epidemics that considers the complete worldwide air travel infrastructure complemented with census population data. We address two basic issues in global epidemic modeling: (i) we study the role of the large scale properties of the airline transportation network in determining the global diffusion pattern of emerging diseases; and (ii) we evaluate the reliability of forecasts and outbreak scenarios with respect to the intrinsic stochasticity of disease transmission and traffic flows. To address these issues we define a set of quantitative measures able to characterize the level of heterogeneity and predictability of the epidemic pattern. These measures may be used for the analysis of containment policies and epidemic risk assessment. This paper contains a quantitative evaluation of probabilistic traffic assignment models and proposes an alternate formulation. First, the concept of stochastic-user-equilibration (S-U-E) is formalized as an extension of Wardrop's user-equilibration criterion. Then, the stochastic-network-loading (S-N-L) problem (a special case of S-U-E for networks with constant link costs) is analyzed in detail and an expression for the probability of route choice which is based on two general postulates of user behavior is derived. The paper also discusses the weaknesses of existing S-N-L techniques with special attention paid to Dial's multipath method and compares them to the suggested approach. The proposed model seems reasonable and does not exhibit the inherent weaknesses of the logit model when applied to sets of routes which overlap heavily. The discussion is supported by several numerical examples on small contrived networks. The paper concludes with the discussion of two techniques that can be used to approximate the link flows resulting from the proposed model in large networks. We propose a probabilistic characterization of network traffic. This characterization can handle traffic with heavy-tailed distributions in performance analysis. We show that queue size, output traffic, virtual delay, aggregate traffic, etc. at various points in a network can easily be characterized within the framework. This characterization is measurable and allows for a simple probabilistic method for regulating network traffic. All of these properties of the proposed characterization enable a systematic approach for providing end-to-end probabilistic QoS guarantees A crucial problem for the efficient design and management of integrated services networks is how to best allocate network resources for heterogeneous and bursty traffic streams in multiplexers that support prioritized service disciplines. In this paper, we introduce a new approach for determining per-connection performance parameters such as delay-bound violation probability and loss probability in multi-service networks. The approach utilizes a traffic characterization consisting of the variances of a stream's rate distribution over multiple interval lengths, which captures its burstiness properties and autocorrelation structure. From this traffic characterization, we provide a simple and efficient resource allocation algorithm by deriving stochastic delay-bounds for static priority schedulers and employing a Gaussian approximation over intervals. To evaluate the scheme, we perform trace-driven simulation experiments with long traces of MPEG-compressed video and show that our approach is accurate enough to capture most of the inherent statistical multiplexing gain, achieving average network utilizations of up to 90 for these traces and substantially outperforming previous "effective bandwidth" techniques. A review of dynamic nonlinear features of spatiotemporal congested patterns in freeway traffic is presented. The basis of the review is a comparison of theoretical features of the congested patterns that are shown by a microscopic traffic flow model in the context of the Kerner's three-phase traffic theory and empirical microscopic and macroscopic pattern characteristics measured on different freeways over various days and years. In this test of the microscopic three-phase traffic flow theory, a model of an "open" road is applied: Empirical time-dependence of traffic demand and drivers' destinations are used at the upstream model boundaries. At downstream model boundary conditions for vehicle freely leaving a modeling freeway section(s) are given. Spatiotemporal congested patterns emerge, develop, and dissolve in this open freeway model with the same types of bottlenecks as those in empirical observations. It is found that microscopic three-phase traffic models can explain all known macroscopic and microscopic empirical congested pattern features (e.g., probabilistic breakdown phenomenon as a first-order phase transition from free flow to synchronized flow, moving jam emergence in synchronized flow rather than in free flow, spatiotemporal features of synchronized flow and general congested patterns at freeway bottlenecks, intensification of downstream congestion due to upstream congestion at adjacent bottlenecks). It turns out that microscopic optimal velocity (OV) functions and time headway distributions are not necessarily qualitatively different, even if local congested traffic behavior is qualitatively different. Model performance with respect to spatiotemporal pattern emergence and evolution cannot be tested using these traffic characteristics. The reason for this is that important spatiotemporal features of congested traffic patterns are lost in these and many other macroscopic and microscopic traffic characteristics, which are widely used as the empirical basis for a test of traffic flow models. As a key approach to securing large networks, existing anomaly detection techniques focus primarily on network traffic data. However, the sheer volume of such data often renders detailed analysis very expensive and reduces the effectiveness of these tools. In this paper, we propose a light-weight anomaly detection approach based on unproductive DNS traffic, namely, the failed DNS queries, with a novel tool - DNS failure graphs. A DNS failure graph captures the interactions between hosts and failed domain names. We apply a graph decomposition algorithm based on the tri-nonnegative matrix factorization technique to iteratively extract coherent co-clusters (dense subgraphs) from DNS failure graphs. By analyzing the co-clusters in the daily DNS failure graphs from a 3-month DNS trace captured at a large campus network, we find these co-clusters represent a variety of anomalous activities, e.g., spamming, trojans, bots, etc.. In addition, these activities often exhibit distinguishable subgraph structures. By exploring the temporal properties of the co-clusters, we show our method can identify new anomalies that likely correspond to unreported domain-flux bots. Understanding and solving performance-related issues of current and future (5G+) networks requires the availability of realistic, yet simple and manageable, traffic models which capture and regenerate various properties of real traffic with sufficient accuracy and minimum number of parameters. Traffic in wireless cellular networks must be modeled in the space domain as well as the time domain. Modeling traffic in the time domain has been investigated well. However, for modeling the User Equipment (UE) distribution in the space domain, either the unrealistic uniform Poisson model, or some non-adjustable model, or specifc data from operators, is commonly used. In this paper, stochastic geometry is used to explain the similarities of traffic modeling in the time domain and the space domain. It is shown that traffic modeling in the time domain is a special one-dimensional case of traffic modeling in the space domain. Unified and non-parameterized metrics for characterizing the heterogeneity of traffic in the time domain and the space domain are proposed and their equivalence to the inter-arrival time, a well accepted metric in the time domain, is demonstrated. Coefficient of Variation (CoV), the normalized second-order statistic, is suggested as an appropriate statistical property of traffic to be measured. Simulation results show that the proposed metrics capture the properties of traffic more accurately than the existing metrics. Finally, the performance of LTE networks under modeled traffic using the new metrics is illustrated. Advanced information and communication technologies can be used to facilitate traffic incident management. If an incident is detected and blocks a road link, in order to reduce the incident-induced traffic congestion, a dynamic strategy to deliver incident information to selected drivers and help them make detours in urban areas is proposed by this work. Time-dependent shortest path algorithms are used to generate a subnetwork where vehicles should receive such information. A simulation approach based on an extended cell transmission model is used to describe traffic flow in urban networks where path information and traffic flow at downstream road links are well modeled. Simulation results reveal the influences of some major parameters of an incident-induced congestion dissipation process such as the ratio of route-changing vehicles to the total vehicles, operation time interval of the proposed strategy, traffic density in the traffic network, and the scope of the area where traffic incident information is delivered. The results can be used to improve the state of the art in preventing urban road traffic congestion caused by incidents. We report a study of the correlations among topological, weighted and spatial properties of large infrastructure networks. We review the empirical results obtained for the air-transportation infrastructure that motivates a network modeling approach which integrates the various attributes of this network. In particular, we describe a class of models which include a weight-topology coupling and the introduction of geographical attributes during the network evolution. The inclusion of spatial features is able to capture the appearance of non-trivial correlations between the traffic flows, the connectivity pattern and the actual distances of vertices. The anomalous fluctuations in the betweenness-degree correlation function observed in empirical studies are also recovered in the model. The presented results suggest that the interplay between topology, weights and geographical constraints is a key ingredient in order to understand the structure and evolution of many real-world networks. Several types of network traffic have been shown to exhibit long-range dependence (LRD). In this work, we show that the busy period of an ATM system driven by a long-range dependent process can be very large. We introduce a new traffic model based on a fractional Brownian motion envelope process. We show that this characterization can be used to predict queueing dynamics. Furthermore, we derive a new framework for computing delay bounds in ATM networks based on this traffic model. We show that it agrees with results given by large deviation theory with less computational complexity. This paper addresses the issue of unsupervised network anomaly detection. In recent years, networks have played more and more critical roles. Since their outages cause serious economic losses, it is quite significant to monitor their changes over time and to detect anomalies as early as possible. In this paper, we specifically focus on the management of the whole network. In it, it is important to detect anomalies which make great impact on the whole network, and the other local anomalies should be ignored. Further, when we detect the former anomalies, it is required to localize nodes responsible for them. It is challenging to simultaneously perform the above two tasks taking into account the nonstationarity and strong correlations between nodes. We propose a network anomaly detection method which resolves the above two tasks in a unified way. The key ideas of the method are: (1)construction of quantities representing feature of a whole network and each node from the same input based on eigen equation compression, and (2)incremental anomalousness scoring based on learning the probability distribution of the quantities. We demonstrate through the experimental results using two benchmark data sets and a simulation data set that anomalies of a whole network and nodes responsible for them can be detected by the proposed method. Non-recurring traffic congestion is caused by temporary disruptions, such as accidents, sports games, adverse weather, etc. We use data related to real-time traffic speed, jam factors (a traffic congestion indicator), and events collected over a year from Nashville, TN to train a multi-layered deep neural network. The traffic dataset contains over 900 million data records. The network is thereafter used to classify the real-time data and identify anomalous operations. Compared with traditional approaches of using statistical or machine learning techniques, our model reaches an accuracy of 98.73 percent when identifying traffic congestion caused by football games. Our approach first encodes the traffic across a region as a scaled image. After that the image data from different timestamps is fused with event- and time-related data. Then a crossover operator is used as a data augmentation method to generate training datasets with more balanced classes. Finally, we use the receiver operating characteristic (ROC) analysis to tune the sensitivity of the classifier. We present the analysis of the training time and the inference time separately.