The fast evolving nature of modern cyber threats and network monitoring needs calls for new, "software-defined", approaches to simplify and quicken programming and deployment of online (stream-based) traffic analysis functions. StreaMon is a carefully designed data-plane abstraction devised to scalably decouple the "programming logic" of a traffic analysis application (tracked states, features, anomaly conditions, etc.) from elementary primitives (counting and metering, matching, events generation, etc), efficiently pre-implemented in the probes, and used as common instruction set for supporting the desired logic. Multi-stage multi-step real-time tracking and detection algorithms are supported via the ability to deploy custom states, relevant state transitions, and associated monitoring actions and triggering conditions. Such a separation entails platform-independent, portable, online traffic analysis tasks written in a high level language, without requiring developers to access the monitoring device internals and program their custom monitoring logic via low level compiled languages (e.g., C, assembly, VHDL). We validate our design by developing a prototype and a set of simple (but functionally demanding) use-case applications and by testing them over real traffic traces.
   We have developed Gigascope, a stream database for network applications including traffic analysis, intrusion detection, router configuration analysis, network research, network monitoring, and performance monitoring and debugging. Gigascope is undergoing installation at many sites within the AT&T network, including at OC48 routers, for detailed monitoring. In this paper we describe our motivation for and constraints in developing Gigascope, the Gigascope architecture and query language, and performance issues. We conclude with a discussion of stream database research problems we have found in our application. Monitoring the traffic of wide area networks consisting of several autonomous systems connected through a high-speed backbone is a challenge due to the huge amount of traffic. Keeping logs for obtaining measurements is unfeasible. This work describes a distributed real-time strategy for backbone traffic monitoring that does not employ logs and allows arbitrary metrics to be collected about the traffic of the backbone as a whole. Traffic is sampled by monitors that are distributed across the backbone and are accessed by a Stream Processing Engine (SPE). Besides the distributed monitoring architecture, we present an implementation (BackStreamDB) that was deployed on a national backbone. Case studies are described that show the system flexibility. Experiments are reported in which we evaluated the amount of traffic that can be handled. Network monitoring is a complex task that generally requires the use of different tools for specific purposes. This paper describes a flexible network monitoring tool, called PaQueT, designed to meet a wide range of monitoring needs. The user can define metrics as queries in a process similar to writing queries on a database management system. This approach provides an easy mechanism to adapt the tool as system requirements evolve. PaQueT allows one to monitor values ranging from packet level metrics to those usually provided only by tools based on Netflow or SNMP. PaQueT has been developed as an extension of Borealis Data Stream Management System. The first advantage of our approach is the ability to generate measurements in real time, minimizing the volume of data stored; second, the tool can be easily extended to consider several types of network protocols. We have conducted an experimental study to verify the effectiveness of our approach, and to determine its capacity to process large volumes of data. The use of the Internet as a medium for real-time communications has grown significantly over the past few years. However, the best-effort model of this network is not particularly well-suited to the demands of users who are familiar with the reliability, quality and security of the Public Switched Telephone Network. If the growth is to continue, monitoring and real time analysis of communication data will be needed in order to ensure good call quality, and should degradation occur, to take corrective action. Writing this type of monitoring application is difficult and time consuming: VoIP traffic not only tends to use dynamic ports, but its real-time nature, along with the fact that its packets tend to be small, impose non-trivial performance requirements. In this paper we present RTC-Mon, the Real-Time Communications Monitoring framework, which provides an extensible platform for the quick development of high-speed, real-time monitoring applications. While the focus is on VoIP traffic, the framework is general and is capable of monitoring any type of real-time communications traffic. We present testbed performance results for the various components of RTC-Mon, showing that it can monitor a large number of concurrent flows without losing packets. In addition, we implemented a proof-of-concept application that can not only track statistics about a large number of calls and their users, but that consists of only 800 lines of code, showing that the framework is efficient and that it also significantly reduces development time. Borealis is a second-generation distributed stream processing engine that is being developed at Brandeis University, Brown University, and MIT. Borealis inherits core stream processing functionality from Aurora [14] and distribution functionality from Medusa [51]. Borealis modifies and extends both systems in non-trivial and critical ways to provide advanced capabilities that are commonly required by newly-emerging stream processing applications. In this paper, we outline the basic design and functionality of Borealis. Through sample real-world applications, we motivate the need for dynamically revising query results and modifying query specifications. We then describe how Borealis addresses these challenges through an innovative set of features, including revision records, time travel, and control lines. Finally, we present a highly flexible and scalable QoS-based optimization model that operates across server and sensor networks and a new fault-tolerance model with flexible consistency-availability trade-offs.