In this paper, we describe and experimentally evaluate the performance of our Network Link Outlier Factor (NLOF) for detecting soft-failures in communication networks. The NLOF is computed using the throughput values derived from NetFlow records. The flow throughput values are clustered in two stages, outlier values are determined within each cluster, and the flow outliers are used to compute the outlier factor or score for each network link. When sampling NetFlow records across the full span of a network, NLOF enables the detection of soft-failures across the span of the network; large NLOF scores correlate well with links experiencing failure.
Much of the related work implements active probing techniques @cite_16 @cite_17 @cite_13 . These techniques use probe messages to infer the state of links and require optimal probe placement @cite_3 @cite_9 @cite_12 to trade off measurement with resource consumption. More recent work @cite_10 uses passive data (e.g., number of: flows, lost packets, average packet delay) and compares the performance of several machine learning techniques (e.g., random forests and multi-layer perceptrons) to localize faults. Their results are compared to the active probing technique in @cite_2 . Other recent work in the domain of optical networking uses passively collected physical layer data and machine learning to detect and or localize faults @cite_5 @cite_11 .
@cite_11 @cite_10 @cite_9 @cite_3 @cite_2 @cite_5 @cite_16 @cite_13 @cite_12 @cite_17
2158778126 2891623325 2159326294 2136283257
Packet delay and loss are two fundamental measures of performance. Using active probing to measure delay and loss typically involves sending Poisson probes, on the basis of the PASTA property (Poisson Arrivals See Time Averages), which ensures that Poisson probing yields unbiased estimates. Recent work, however, has questioned the utility of PASTA for probing and shown that, for delay measurements, i) a wide variety of processes other than Poisson can be used to probe with zero bias and ii) Poisson probing does not necessarily minimize the variance of delay estimates. In this paper, we determine optimal probing processes that minimize the mean-square error of measurement estimates for both delay and loss. Our contributions are twofold. First, we show that a family of probing processes, specifically Gamma renewal probing processes, has optimal properties in terms of bias and variance. The optimality result is general, and only assumes that the target process we seek to optimally measure via probing, such as a loss or delay process, has a convex auto-covariance function. Second, we use empirical datasets to demonstrate the applicability of our results in practice, specifically to show that the convexity condition holds true and that Gamma probing is indeed superior to Poisson probing. Together, these results lead to explicit guidelines on designing the best probe streams for both delay and loss estimation. Communication networks such as wireless sensor networks, Internet of Things and vehicular ad-hoc networks are becoming more complex and increasing in size. This leads to high overhead (network and computation) and difficulty in determining the accurate network topology, which is an important information for traffic engineering and network management. Localization of link failures in such networks is a challenging problem and requires a novel approach to achieve the goal without any prior information about the network topology. In this paper, we present a traffic engineering (TE)-based machine learning approach to detect and localize link failures. Instead of using topology information and actively injecting additional packets to localize a failed link, the proposed machine learning model adopts a passive mechanism to learn the network traffic behavior from propagation delay, number of flows and average packet loss at every node in the network under normal working conditions and failure scenarios. We train the learning model with machine learning algorithms such as naive Bayes, logistic regression, support vector machine, multi-layer perceptron, decision tree and random forest. We implement the proposed approach and carry out extensive experiments using the Mininet platform. The performance study shows that our proposed approach localizes link failures with at least 90 accuracy using random forest algorithm while requiring less time-to-localization of a link failure compared to other existing works. Despite many efforts over the past decade, the ability to generate topological maps of the Internet at the router-level accurately and in a timely fashion remains elusive. Mapping campaigns commonly involve traceroute-like probing that are usually non-adaptive and incomplete, thus revealing only a portion of the underlying topology. In this paper we demonstrate that standard probing methods yield datasets that implicitly contain information about much more than just the directly observed links and routers. Each probe, in addition to the underlying domain knowledge, returns information that places constraints on the underlying topology, and by integrating a large number of such constraints it is possible to accurately infer the existence of unseen components of the Internet. We describe DomainImpute, a novel data analysis methodology designed to accurately infer the unseen hop-count distances between observed routers. We use both synthetic and a large empirical dataset to validate the proposed methods. On our empirical real world dataset, we show that our methods can estimate over 55 of the unseen distances between observed routers to within a one-hop error. Increase in the network usage for more and more performance critical applications has caused a demand for tools that can monitor network health with minimum management traffic. Adaptive probing has the potential to provide effective tools for end-to-end monitoring and fault diagnosis over a network. Adaptive probing based algorithms adapt the probe set to localize faults in the network by sending less probes in healthy areas and more probes in the suspected areas of failure. In this paper we present adaptive probing tools that meet the requirements to provide an effective and efficient solution for fault diagnosis for modern communication systems. We present a system architecture for adaptive probing based fault diagnosis tool and propose algorithms for probe selection to perform failure detection and fault localization. We compare the performance and efficiency of the proposed algorithms through simulation results.