10897641
Reversible sketches for efficient and accurate change detection over network data streams
Traffic anomalies such as failures and attacks are increasing in frequency and severity, and thus identifying them rapidly and accurately is critical for large network operators. The detection typically treats the traffic as a collection of flows and looks for heavy changes in traffic patterns (<i>e.g.</i>, volume, number of connections). However, as link speeds and the number of flows increase, keeping per-flow state is not scalable. The recently proposed sketch-based schemes [14] are among the very few that can detect heavy changes and anomalies over massive data streams at network traffic speeds. However, sketches do not preserve the key (<i>e.g.</i>, source IP address) of the flows. Hence, even if anomalies are detected, it is difficult to infer the culprit flows, making it a big practical hurdle for online deployment. Meanwhile, the number of keys is too large to record.
 To address this challenge, we propose efficient <i>reversible hashing</i> algorithms to infer the keys of culprit flows from sketches without storing any explicit key information. No extra memory or memory accesses are needed for recording the streaming data. Meanwhile, the heavy change detection daemon runs in the background with space complexity and computational time sublinear to the key space size. This short paper describes the conceptual framework of the reversible sketches, as well as some initial approaches for implementation. See [23] for the optimized algorithms in details. comment We further apply various emph IP-mangling algorithms and emph bucket classification methods to reduce the false positives and false negatives. Evaluated with netflow traffic traces of a large edge router, we demonstrate that the reverse hashing can quickly infer the keys of culprit flows even for many changes with high accuracy.
887532
A Suite of Schemes for User-Level Network Diagnosis without Infrastructure
It is highly desirable and important for end users, with no special privileges, identify and pinpoint faults inside the network that degrade the performance of their applications. However, existing tools are inaccurate to infer the link-level loss rates and have large diagnosis granularity (in terms of the number of hops). To address these problems, we propose a suite of user-level diagnosis approaches in two categories: (1) only need to be deployed at the source and (2) deployed at both source and destination. For the former, we propose two fragmentation aided diagnosis approaches (FAD), Algebraic FAD and Opportunistic FAD, which uses IP fragmentation to enable accurate link-level loss rate inference. For the latter category, we propose Striped Probe Analysis (SPA) which significantly improves the diagnosis granularity over those of the source-only approaches. Internet experiments are applied to evaluate each individual schemes (including an improved version of the state-of-the-art tool, Tulip [1]) and various hybrid approaches. The results indicate that our approaches dramatically outperform existing work (especially for diagnosis granularity) and provide not only the best performance but also smooth tradeoff among deployment requirement, diagnosis accuracy and granularity.
6860435
On the stability of network distance estimation
Overlay network distance monitoring and estimation system can benefit many new applications and services, such as peer-to-peer overlay routing and location. However, there is a lack of such scalable system with small overhead, good usability, and good distance estimation accuracy and stability. Thus we propose a scalable overlay distance monitoring system, Internet Iso-bar, which clusters hosts based on the similarity of their perceived network distance, with no assumption about the underlying network topology. The centers of each cluster are then chosen as monitors to represent their clusters for probing and distance estimation. We compare it with other network distance estimation systems, such as Global Network Positioning (GNP) [1]. Internet Iso-bar is easy to implement and use, and has good scalability and small communication and computation cost for online monitoring. Preliminary evaluation on real Internet measurement data also shows that Internet Iso-bar has high prediction accuracy and stability. Finally, by adjusting the number of clusters, we can smoothly trade off the measurement and management cost for better distance estimation accuracy and stability.
166109
End-to-End Inference of Router Packet Forwarding Priority
Packet forwarding prioritization (PFP) in routers is one of the mechanisms commonly available to network administrators. PFP can have a significant impact on the performance of applications, the accuracy of measurement tools' results and the effectiveness of network troubleshooting procedures. Despite their potential impact, no information on PFP settings is readily available to end users. In this paper, we present an end-to-end approach for packet forwarding priority inference and its associated tool, POPI. This is the first attempt to infer router packet-forwarding priority through end-to-end measurement. Our POPI tool enables users to discover such network policies through the monitoring and rank classification of loss rates for different packet types. We validated our approach via statistical analysis, simulation, and wide-area experimentation in PlanetLab. As part of our wide-area experiments, we employed POPI to analyze 156 random paths across 162 PlanetLab nodes. We discovered 15 paths flagged with multiple priorities, 13 of which were further validated through hop-by-hop loss rates measurements. In addition, we surveyed all related network operators and received responses for about half of them confirming our inferences.
61999607
Towards unbiased end-to-end network diagnosis
Internet fault diagnosis is extremely important for end users, overlay network service providers (like Akamai [1]) and even Internet service providers (ISPs). However, because link-level properties cannot be uniquely determined from end-to-end measurements, the accuracy of existing statistical diagnosis approaches is subject to uncertainty from statistical assumptions about the network. In this paper, we propose a novel Least-biased End-to-end Network Diagnosis (in short, LEND) system for inferring link-level properties like loss rate. We define a minimal identifiable link sequence (MILS) as a link sequence of minimal length whose properties can be uniquely identified from end-to-end measurements. We also design efficient algorithms to find all the MILSes and infer their loss rates for diagnosis. Our LEND system works for any network topology and for both directed and undirected properties, and incrementally adapts to network topology and property changes. It gives highly accurate estimates of the loss rates of MILSes, as indicated by both extensive simulations and Internet experiments. Furthermore, we demonstrate that such diagnosis can be achieved with fine granularity and in near real-time even for reasonably large overlay networks. Finally, LEND can supplement existing statistical inference approaches and provide smooth tradeoff between diagnosis accuracy and granularity.
16325811
Collaborative detection and filtering of shrew DDoS attacks using spectral analysis
This paper presents a new spectral template-matching approach to countering shrew distributed denial-of-service (DDoS) attacks. These attacks are stealthy, periodic, pulsing, and low-rate in attack volume, very different from the flooding type of attacks. They are launched with high narrow spikes in very low frequency, periodically. Thus, shrew attacks may endanger the victim systems for a long time without being detected. In other words, such attacks may reduce the quality of services unnoticeably. Our defense method calls for collaborative detection and filtering (CDF) of shrew DDoS attacks. We detect shrew attack flows hidden in legitimate TCP/UDP streams by spectral analysis against pre-stored template of average attack spectral characteristics. This novel scheme is suitable for either software or hardware implementation. 
 
The CDF scheme is implemented with the NS-2 network simulator using real-life Internet background traffic mixed with attack datasets used by established research groups. Our simulated results show high detection accuracy by merging alerts from cooperative routers. Both theoretical modeling and simulation experimental results are reported here. The experiments achieved up to 95% successful detection of network anomalies along with a low 10% false positive alarms. The scheme cuts off malicious flows containing shrew attacks using a newly developed packet-filtering scheme. Our filtering scheme retained 99% of legitimate TCP flows, compared with only 20% TCP flows retained by using the Drop Tail algorithm. The paper also considers DSE FPGA, and network processor implementation issues and discusses limitations and further research challenges.
365943
Modeling and Storing Scientific Protocols
We propose an abstract model for scientific protocols, where several atomic operators are proposed for protocol composition We distinguish two different layers associated with scientific protocols: design and implementation, and discuss the mapping between them We illustrate our approach with a representative example and describe ProtocolDB, a scientific protocol repository currently in development Our approach benefits scientists by allowing the archiving of scientific protocols with the collected data sets to constitute a scientific portfolio for the laboratory to query, compare and revise protocols.
5682504
An Efficient XPath Query Processor for XML Streams
Streaming XPath evaluation algorithms must record a potentially exponential number of pattern matches when both predicates and descendant axes are present in queries, and the XML data is recursive. In this paper, we use a compact data structure to encode these pattern matches rather than storing them explicitly. We then propose a polynomial time streaming algorithm to evaluate XPath queries by probing the data structure in a lazy fashion. Extensive experiments show that our approach not only has a good theoretical complexity bound but is also efficient in practice.
32843088
Analysis of Zipf's law: An index approach
Abstract A rigorous analysis of Zipf's law is made using an index for the sequence of observed values of the variables in a Zipf-type relationship. Three important properties relating rank, count, and frequency are identified. Using this approach, the shape of Zipf-type curves can be described in terms of three distinct regions and two parameters of the Mandelbrot-Zipf law. This result has considerable practical significance, since it provides rigorous foundations for the application of Zipf's law.
8394681
ViteX: a streaming XPath processing system
We present ViteX, an XPath processing system on XML streams with polynomial time complexity. ViteX uses a polynomial-space data structure to encode an exponential number of pattern matches (in the query size) which are required to process queries correctly during a single sequential scan of XML. Then ViteX computes query solutions by probing the data structure in a lazy fashion without enumerating pattern matches.