Spatial Big Data and Moving Objects: A Comprehensive Survey <s> 2) APPLICATION AWARE BIG DATA ROUTING IN SDN <s> Traffic information system may produce massive and complex traffic data with the process of collecting real-time original GPS (Global Positioning System) data, matching positions to a map and generating traffic flow information, which brings great markets for even-worse traffic condition in China. Howerver, several issues would occur when we reuse these massive traffic data for history data mining applying on-hand database management tools or traditional data processing apporaches, such as massive storage, high performance processing, open interface. "Big Data" system usually includes data sets with sizes beyond the ability of commonly-used software tools to capture, manage, and process the data within a tolerable elapsed time. With this difficulty and the advantage of "Big Data", we schemed RTIC-C system to handle sensemaking over large quantities of traffic data based on cloud computing technique. RTIC-C designs a distributed data management service to support large scale of data storage, a parallel distributed computing framework for diverse kinds of mining applications based on Map-Reduce mechanism, a restful Web services interface to support third-party mining applications. Experiments on a massive traffic data sets showed that RTIC-C achieves considerable performance comparing with traditional traffic data mining applications. <s> BIB001 </s> Spatial Big Data and Moving Objects: A Comprehensive Survey <s> 2) APPLICATION AWARE BIG DATA ROUTING IN SDN <s> Big-Data applications need to shuffle their data over a network in a very large amount. The ability to efficiently allocate network resources to quickly finish the shuffle greatly affects the overall processing speed of Big-Data applications. The recent emergence of software-defined networking (SDN) technology makes it easier for the network to integrate with a Big-Data application. The network can allocate network resources based on an application's run-time requirements to accelerate the speed of Big-Data delivery. In this paper, we used Hadoop as a Big-Data application platform and designed an application-aware SDN routing scheme for Hadoop to accelerate its MapReduce data shuffling over a network. Our results show that our method can outperform the ECMP-RR scheme by up to 20% and outperform the Spanning Tree method and the Floodlight OpenFlow controller by up to 70%. <s> BIB002 </s> Spatial Big Data and Moving Objects: A Comprehensive Survey <s> 2) APPLICATION AWARE BIG DATA ROUTING IN SDN <s> This paper proposes a scheme to reduce the number of flow entries permanently stored in an OpenFlow switch and the number of configuration messages from a controller in a software-defined network (SDN). In an SDN, a flow table in an OpenFlow switch is used to instruct packets. The flow table consists of flow entries decided by the controller. A flow request is sent from the OpenFlow switch to the controller if the incoming packet does not match any flow entry in the flow table. The controller’s central processing unit may be overloaded to handle user requests, since the user requests for different data types have been rapidly increasing. As a result, flow configuration in switches is delayed. Moreover, the control plane may be flooded by configuration messages of those requests. A scheme to permanently keep the flow entries in the switch can reduce the number of requests. However, a large number of permanent flow entries is required. Other switch features may be degraded, since there is not enough memory in the flow table to implement those features. In the proposed scheme, switches in the network are divided into multiple regions. In order to guide packets from sources to destinations, the flow table incorporating the concept of two multiprotocol label switching tags is re-designed. One tag directs a packet from a source switch to an edge switch in the destination region. The other tag directs the packet from that edge switch to another switch in the same region. A mathematical model for the proposed scheme is formulated as an integer linear programming to determine a set of switches in each region so that the total number of permanent flow entries in the network can be minimized. The performance of the proposed scheme is analyzed. Moreover, the proposed scheme is implemented and demonstrated via Japanese Science Information Network 5. <s> BIB003 </s> Spatial Big Data and Moving Objects: A Comprehensive Survey <s> 2) APPLICATION AWARE BIG DATA ROUTING IN SDN <s> While Hadoop ecosystems become increasingly important for practitioners of large-scale data analysis, they also incur tremendous energy cost. This trend is driving up the need for designing energy-efficient Hadoop clusters in order to reduce the operational costs and the carbon emission associated with its energy consumption. However, despite extensive studies of the problem, existing approaches for energy efficiency have not fully considered the heterogeneity of both workload and machine hardware found in production environments. In this paper, we find that heterogeneity-oblivious task assignment approaches are detrimental to both performance and energy efficiency of Hadoop clusters. Our observation shows that even heterogeneity-aware techniques that aim to reduce the job completion time do not guarantee a reduction in energy consumption of heterogeneous machines. We propose a heterogeneity-aware task assignment approach, E-Ant, that aims to improve the overall energy consumption in a heterogeneous Hadoop cluster without sacrificing job performance. It adaptively schedules heterogeneous workloads on energy-efficient machines, without a priori knowledge of the workload properties. E-Ant employs an ant colony optimization approach that generates task assignment solutions based on the feedback of each task’s energy consumption reported by Hadoop TaskTrackers in an agile way. Furthermore, we integrate DVFS technique with E-Ant to further improve the energy efficiency of heterogeneous Hadoop clusters. It relies on a DVFS controller to dynamically scale the CPU frequency of each slave machine in response to time-varying resource demands. Experimental results on a heterogeneous cluster with varying hardware capabilities show that E-Ant with DVFS improves the overall energy savings for a synthetic workload from Microsoft by 23 and 17 percent compared to Fair Scheduler and Tarazu, respectively. <s> BIB004 </s> Spatial Big Data and Moving Objects: A Comprehensive Survey <s> 2) APPLICATION AWARE BIG DATA ROUTING IN SDN <s> This paper proposes a novel routing protocol for OppNets called MLProph, which uses machine learning (ML) algorithms, namely decision tree and neural networks, to determine the probability of successful deliveries. The ML model is trained by using various factors such as the predictability value inherited from the PROPHET routing scheme, node popularity, node's power consumption, speed, and location. Simulation results show that MLProph outperforms PROPHET+, a probabilistic-based routing protocol for OppNets, in terms of number of successful deliveries, dropped messages, overhead, and hop count, at the cost of small increases in buffer time and buffer occupancy values. <s> BIB005 </s> Spatial Big Data and Moving Objects: A Comprehensive Survey <s> 2) APPLICATION AWARE BIG DATA ROUTING IN SDN <s> The amount of digital data produced worldwide is exponentially growing. While the source of this data, collectively known as Big Data, varies from among mobile services to cyber physical systems and beyond, the invariant is their increasingly rapid growth for the foreseeable future. Immense incentives exist, from marketing campaigns to forensics and to research in social sciences, that motivate processing increasingly bigger data so as to extract information and knowledge for the betterment of processes and benefits. Consequently, the need for more efficient computing systems tailored to such big data applications is increasingly intensified. Such custom architectures would expectedly embrace heterogeneity to better match each phase of the computation. In this paper we review state of the art as well as envisioned future large-scale computing architectures customized for batch processing of big data applications in the MapReduce paradigm. We also provide our view of current important trends relevant to such systems, and their impacts on future architectures and architectural features expected to address the needs of tomorrow big data processing in this paradigm. <s> BIB006
SDN has shown great emergence in the recent past. Basically, in SDN, a centralized entity (a software defined centralized controller) or a platform controls the flow of all the network traffic BIB003 . The centralized controller adds flexibility in routing by intelligently controlling the flow of network traffic, thus making the integration of SDN with BD applications a feasible approach. The work proposed in BIB002 utilizes the ''application awareness'' advantage of SDN to perform realtime routing. Basically, the authors analyze the traffic patterns of MapReduce BIB006 based on the information provided by Hadoop (a well-known BD platform) BIB004 . An entity named as TaskTracker is allowed to store the information about traffic size and volume, which is later communicated to the SDN controller. This novelty was missing in BIB001 where the job of TaskTracker was only to send empty hello/heartbeat messages. Based on the information provided by Hadoop (via TaskTracker), the SDN controller performs routing of BD traffic by assigning priorities to one-on-one flows which are remotely located. Flows having all-to-all connectivity are given lower priorities since they have higher freedom in accessing the available bandwidth. For experiments, real-world traffic patterns are generated using sixteen different Hadoop servers with a bandwidth of 10 Mbps and a delay of 1 ms, respectively. Most prominent feature of the results is the comparison of proposed SDN aware routing with existing solutions not utilizing the SDN as a part of their architecture (such as the famous spanning tree algorithm BIB005 ). As expected, all results show the proposed approach significantly reduces the overall traffic shuffling time due to the flexibility added by the SDN controller. Nevertheless, there is no discussion nor any graphical illustration on other important parameters such as the system complexity, number of messages sent and received between Hadoop and SDN controller, and the effect of increased message size on global system performance.