Several alternatives for more efficient spectrum management have been proposed over the last decade, resulting in new techniques for automatic wideband spectrum sensing. However, while spectrum sensing technology is important, understanding, using and taking actions on this data for better spectrum and network resource management is at least equally important. In this paper, we propose a system that is able to automatically detect wireless spectrum events from streaming spectrum sensing data, and enables the consumption of the events as they are produced, as a statistical report or on a per-query basis. The proposed system is referred to as spectrum streamer and is wireless technology agnostic, scalable, able to deliver actionable information to humans and machines and also enables application development by custom querying of the detected events.
While radio spectrum allocation is well regulated, there is little knowledge about its actual utilization over time and space. This limitation hinders taking effective actions in various applications including cognitive radios, electrosmog monitoring, and law enforcement. We introduce Electrosense, an initiative that seeks a more efficient, safe and reliable monitoring of the electromagnetic space by improving the accessibility of spectrum data for the general public. A collaborative spectrum monitoring network is designed that monitors the spectrum at large scale with low-cost spectrum sensing nodes. The large set of data is stored and processed in a big data architecture and provided back to the community with an open spectrum data as a service model, that allows users to build diverse and novel applications with different requirements. We illustrate useful usage scenarios of the Electrosense data. Storm has long served as the main platform for real-time analytics at Twitter. However, as the scale of data being processed in real-time at Twitter has increased, along with an increase in the diversity and the number of use cases, many limitations of Storm have become apparent. We need a system that scales better, has better debug-ability, has better performance, and is easier to manage -- all while working in a shared cluster infrastructure. We considered various alternatives to meet these needs, and in the end concluded that we needed to build a new real-time stream data processing system. This paper presents the design and implementation of this new system, called Heron. Heron is now the de facto stream data processing engine inside Twitter, and in this paper we also share our experiences from running Heron in production. In this paper, we also provide empirical evidence demonstrating the efficiency and scalability of Heron.