We study the problem of filtering duplicate items over physically distributed data streams to provide clean data for real-time monitoring applications. Existing approaches only filter local duplicates within each stream, and their space and time costs are hardly feasible for high-speed data streams. Based on the space/time efficient data structure Bloom filter, we propose a novel local filtering algorithm to efficiently filter local duplicates, and then extend it to global duplicates filtering which is never addressed before. To adapt to different additional communication overhead in global duplicates filtering, we present eager and lazy approaches for Bloom filter sharing. Theoretical and experimental results show that our solution can efficiently filter duplicates locally and globally, while the errors are small enough when the arguments are set properly.
In this overview paper we motivate the need for and research issues arising from a new model of data processing. In this model, data does not take the form of persistent relations, but rather arrives in multiple, continuous, rapid, time-varying data streams. In addition to reviewing past work relevant to data stream systems and current projects in the area, the paper explores topics in stream query languages, new requirements and challenges in query processing, and algorithmic issues.
Berzelius failed to make use of Faraday's electrochemical laws in his laborious determination of equivalent weights.