With the emergence of cloud computing and sensor technologies, Big Data analytics for the Internet of Things (IoT) has become the main force behind many innovative solutions for our society's problems. This paper provides practical explanations for the question "why is the number of Big Data applications that succeed and have an effect on our daily life so limited, compared with all of the solutions proposed and tested in the literature?", with examples taken from Smart Grids. We argue that "noninvariants" are the most challenging issues in IoT applications, which can be easily revealed if we use the term "invariant" to replace the more common terms such as "information", "knowledge", or "insight" in any Big Data for IoT research. From our experience with developing Smart Grid applications, we produced a list of "noninvariants", which we believe to be the main causes of the gaps between Big Data in a laboratory and in practice in IoT applications. This paper also proposes Graph of Virtual Actors (GOVA) as a Big Data analytics architecture for IoT applications, which not only can solve the noninvariants issues, but can also quickly scale horizontally in terms of computation, data storage, caching requirements, and programmability of the system.
has been proposed as a suitable data modeling tool for applications where relations between data are as important as the data itself @cite_29 . In Big Data analytics for IoT, subgraph matching, graph traversal, or graph analysis are commonly needed. These graph-based queries are expensive in the traditional database due to the need of recursive JOINs. A native graph database engine such as Neo4j @cite_29 , GraphX @cite_22 , or Trinity @cite_25 can easily outperform an RDBMS in those tasks. Moreover, graph database provides a more flexible data model, where adding a new type of entity or relationship does not necessarily require a change in the database schema. Furthermore, data is stored so that it semantically represents its structure. All of these properties make graph database a useful data modeling tool for IoT. Figure shows a small typical graph in a smart distribution system grid.
@cite_29 @cite_25 @cite_22
776871969 2160459668 1982003698
Graph databases (GDB) are now a viable alternative to Relational Database Systems (RDBMS). Chemistry, biology, semantic web, social networking and recommendation engines are all examples of applications that can be represented in a much more natural form. Comparisons will be drawn between relational database systems (Oracle, MySQL) and graph databases (Neo4J) focusing on aspects such as data structures, data model features and query facilities. Additionally, several of the inherent and contemporary limitations of current offerings comparing and contrasting graph vs. relational database implementations will be explored. Computations performed by graph algorithms are data driven, and require a high degree of random data access. Despite the great progresses made in disk technology, it still cannot provide the level of efficient random access required by graph computation. On the other hand, memory-based approaches usually do not scale due to the capacity limit of single machines. In this paper, we introduce Trinity, a general purpose graph engine over a distributed memory cloud. Through optimized memory management and network communication, Trinity supports fast graph exploration as well as efficient parallel computing. In particular, Trinity leverages graph access patterns in both online and offline computation to optimize memory and communication for best performance. These enable Trinity to support efficient online query processing and offline analytics on large graphs with just a few commodity machines. Furthermore, Trinity provides a high level specification language called TSL for users to declare data schema and communication protocols, which brings great ease-of-use for general purpose graph management and computing. Our experiments show Trinity's performance in both low latency graph queries as well as high throughput graph analytics on web-scale, billion-node graphs. From social networks to targeted advertising, big graphs capture the structure in data and are central to recent advances in machine learning and data mining. Unfortunately, directly applying existing data-parallel tools to graph computation tasks can be cumbersome and inefficient. The need for intuitive, scalable tools for graph computation has lead to the development of new graph-parallel systems (e.g., Pregel, PowerGraph) which are designed to efficiently execute graph algorithms. Unfortunately, these new graph-parallel systems do not address the challenges of graph construction and transformation which are often just as problematic as the subsequent computation. Furthermore, existing graph-parallel systems provide limited fault-tolerance and support for interactive data mining. We introduce GraphX, which combines the advantages of both data-parallel and graph-parallel systems by efficiently expressing graph computation within the Spark data-parallel framework. We leverage new ideas in distributed graph representation to efficiently distribute graphs as tabular data-structures. Similarly, we leverage advances in data-flow systems to exploit in-memory computation and fault-tolerance. We provide powerful new operations to simplify graph construction and transformation. Using these primitives we implement the PowerGraph and Pregel abstractions in less than 20 lines of code. Finally, by exploiting the Scala foundation of Spark, we enable users to interactively load, transform, and compute on massive graphs.