Several alternatives for more efficient spectrum management have been proposed over the last decade, resulting in new techniques for automatic wideband spectrum sensing. However, while spectrum sensing technology is important, understanding, using and taking actions on this data for better spectrum and network resource management is at least equally important. In this paper, we propose a system that is able to automatically detect wireless spectrum events from streaming spectrum sensing data, and enables the consumption of the events as they are produced, as a statistical report or on a per-query basis. The proposed system is referred to as spectrum streamer and is wireless technology agnostic, scalable, able to deliver actionable information to humans and machines and also enables application development by custom querying of the detected events.
SpecInsight is a component of the Microsoft Spectrum Observatory http: observatory.microsoftspectrum.com . Other spectrum monitoring projects are the Google spectrum https: www.google.com get spectrumdatabase for measurements on TV white-spaces; the IBM Horizon https: bluehorizon.network documentation sdr-radio-spectrum-analysis project, and, more recently, Electrosense https: electrosense.org . The Electrosense architecture @cite_7 is designed for processing large volumes of batch and streaming data using popular crowdsourcing projects such as Kafka for Apache Spark for realizing real-time data pipelines, Apache Spark for real-time and batch data processing and Casandra for storage. For the time being, Electronsense does not have specialized modules implemented on top of Apache Spark to perform event detection. Such modules, if developed in the future, could be compared in detection performance and scalability to the spectrum streamer, especially considering the fact that the architecture of QMiner's stream processing functionality, on which the proposed spectrum streamer is based, is closer to Twitter Heron @cite_11 than to Apache Spark.
@cite_7 @cite_11
1804051827 2172986903 2498111289 2187472121
Spectrum sensing has been an active research area for the past two decades. Nonetheless, current spectrum sensing systems provide only coarse occupancy data. They lack information about the detailed signal patterns in each band and can easily miss fleeting signals like radar. This paper presents SpecInsight, a system for acquiring a detailed view of 4 GHz of spectrum in realtime. SpecInsight's design addresses the intrinsic conflict between the need to quickly scan a wide spectrum and the desire to obtain very detailed information about each band. Its key enabler is a learned database of signal patterns and a new scheduling algorithm that leverages these patterns to identify when to sample each band to maximize the probability of sensing active signals. SpecInsight is implemented using off-the-shelf USRP radios with only tens of MHz of instantaneous bandwidth, but it is able to sense 4 GHz of spectrum, and capture very low duty-cycle signals in the radar band. Using SpecInsight, we perform a large-scale study of the spectrum in 7 locations in the US that span major cities and suburban areas, and build a first-of-its-kind database of spectrum usage patterns. A new compound operator is proposed for heterogeneous periocular recognition.The new operator outperforms three basic operators and three state-of-the-art compound operators.NIR, SWIR, MWIR and LWIR spectra at different standoff distances are considered.A metric is introduced to measure image quality and explain its impact on performance.The new operator does not require training and can be applied to various datasets. Cross-spectral matching of active and passive infrared (IR) periocular images to a visible light periocular image gallery is a challenging research problem. This scenario is motivated by a number of surveillance applications such as recognition of subjects at night or in harsh environmental conditions. This problem becomes even more challenging with a varying standoff distance. To address this problem a new compound operator named GWLH that fuses three local descriptors - Histogram of Gradients (HOG), Local Binary Patterns (LBP) and Weber Local Descriptors (WLD) - applied to the outputs of Gabor filters is proposed. The local operators encode both magnitude and phase information. When applied to periocular regions, GWLH outperforms other compound operators that recently appeared in the literature. During performance evaluation LBP, Gabor filters, HOG, and a fusion of HOG and LBP establish a baseline for the performance comparison, while other compound operators such as Gabor followed by HOG and LBP as well as Gabor followed by WLD, LBP and GLBP present the state-of-the-art. The active IR band is presented by short-wave infrared (SWIR) and near-infrared (NIR) and passive IR is presented by mid-wave infrared (MWIR) and long-wave infrared (LWIR). In addition to varying spectrum, we also vary the standoff distance of SWIR and NIR probes. In all but one case of the combination of spectrum and range, GWLH outperforms all the other operators. A sharpness metric is introduced to measure the quality of heterogeneous periocular images and to emphasize the need in development of image enhancement approaches for heterogeneous periocular biometrics. Based on the statistics of the sharpness metric, the performance difference between compound and single operators is increasing proportionally with increasing sharpness metric values. Big Data analytics has recently gained increasing popularity as a tool to process large amounts of data on-demand. Spark and Flink are two Apache-hosted data analytics frameworks that facilitate the development of multi-step data pipelines using directly acyclic graph patterns. Making the most out of these frameworks is challenging because efficient executions strongly rely on complex parameter configurations and on an in-depth understanding of the underlyingarchitectural choices. Although extensive research has been devoted to improving and evaluating the performance of such analytics frameworks, most of them benchmarkthe platforms against Hadoop, as a baseline, a rather unfair comparison consideringthe fundamentally different design principles. This paper aims to bring some justice in this respect, by directly evaluating the performance of Sparkand Flink. Our goal is to identify and explain the impact of the different architecturalchoices and the parameter configurations on the perceived end-to-end performance. To this end, we develop a methodology for correlating the parameter settings and the operators execution plan with the resource usage. We use this methodologyto dissect the performance of Spark and Flink with several representative batchand iterative workloads on up to 100 nodes. Our key finding is that there none of the two framework outperforms the other for all data types, sizes and job patterns. This paper performs a fine characterization of the cases when each framework is superior, and we highlight how this performance correlates to operators, to resource usage and to the specifics of the internal framework design. This thesis analyses the adequacy and the performance of the Apache Spark Streaming engine in the context of anomaly detection in water distribution networks (WDN). It builds on an already proposed scenario in which sensors are extensively deployed in a WDN allowing for distributed, near-real-time anomaly detection. For this purpose, it uses several variations of LISA statistics and according statistical tests. In order to show that Spark Streaming is applicable to such a setting, algorithms computing these statistics in Spark Streaming were developed. Subsequently, the resulting prototype was tested in a simulated WDN setting. Finally, the performance of the different algorithms as well as the impact of several network characteristics were measured. The results show that the calculation of LISA statistics can be achieved with reasonable performance by using Spark Streaming. Furthermore, they reveal certain characteristics and limitations of Spark Streaming.