Energy big data: A survey <s> 2) MAIN TOOLS <s> MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper. ::: ::: Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program's execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system. ::: ::: Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: hundreds of MapReduce programs have been implemented and upwards of one thousand MapReduce jobs are executed on Google's clusters every day. <s> BIB001 </s> Energy big data: A survey <s> 2) MAIN TOOLS <s> The Hadoop Distributed File System (HDFS) is designed to store very large data sets reliably, and to stream those data sets at high bandwidth to user applications. In a large cluster, thousands of servers both host directly attached storage and execute user application tasks. By distributing storage and computation across many servers, the resource can grow with demand while remaining economical at every size. We describe the architecture of HDFS and report on experience using HDFS to manage 25 petabytes of enterprise data at Yahoo!. <s> BIB002 </s> Energy big data: A survey <s> 2) MAIN TOOLS <s> Apache's Hadoop is an open source framework for large scale data analysis and storage. It is an open source implementation of Google's Map/Reduce framework. It enables distributed, data intensive and parallel applications by decomposing a massive job into smaller tasks and a massive data set into smaller partitions such that each task processes a different partition in parallel. Hadoop uses Hadoop distributed File System (HDFS) which is an open source implementation of the Google File System (GFS) for storing data. Map/Reduce application mainly uses HDFS for storing data. HDFS is a very large distributed file system that assumes commodity hardware and provides high throughput and fault tolerance. HDFS stores files as a series of blocks and are replicated for fault tolerance. The default block placement strategy doesn't consider the data characteristics and places the data blocks randomly. Customized strategies can improve the performance of HDFS to a great extend. Applications using HDFS require streaming access to the files and if the related files are placed in the same set of data nodes, the performance can be increased. This paper is discussing about a method for clustering streaming data to the same set of data nodes using the technique of Locality Sensitive Hashing. The method utilizes the compact bitwise representation of document vectors called fingerprints created using the concept of Locality Sensitive Hashing to increase the data processing speed and performance. The process will be done without affecting the default fault tolerant properties of Hadoop and requires only minimal changes to the Hadoop framework. <s> BIB003 </s> Energy big data: A survey <s> 2) MAIN TOOLS <s> Big Data processing with Hadoop has been emerging recently, both on the computing cloud and enterprise deployment. However, wide-spread security exploits may hurt the reputation of public clouds. If Hadoop on the cloud is not an option, an organization has to build its own Hadoop clusters. But having a data center is not worth for a small organization both in terms of building and operating costs. Another viable solution is to build a cluster with low-cost ARM system-on-chip boards. This paper presents a study of a Hadoop cluster for processing Big Data built atop 22 ARM boards. The Hadoop's MapReduce was replaced by Spark and experiments on three different hardware configurations were conducted to understand limitations and constraints of the cluster. From the experimental results, it can be concluded that processing Big Data on an ARM cluster is highly feasible. The cluster could process a 34 GB Wikipedia article file in acceptable time, while generally consumed the power 0.061–0.322 kWh for all benchmarks. It has been found that I/O of the hardware is fast enough, but the power of CPUs is inadequate because they are largely spent for the Hadoop's I/O. <s> BIB004 </s> Energy big data: A survey <s> 2) MAIN TOOLS <s> Now a days, we are living with social media data like heartbeat. The exponential growth with data first presented challenges to cutting-edge businesses such as Google, MSN, Flipkart, Microsoft, Facebook, Twitter, LinkedIn etc. Nevertheless, existing big data analytical models for hadoop comply with MapReduce analytical workloads that process a small segment of the whole data set, thus failing to assess the capabilities of the MapReduce model under heavy workloads that process exponentially accumulative data sizes.[1] In all social business and technical research applications, there is a need to process big data of data in efficient manner on normal uses data. In this paper, we have proposed an efficient technique to classify the big data from e-mail using firefly and naive bayes classifier. Proposed technique is comprised into two phase, (i) Map reduce framework for training and (ii) Map reduce framework for testing. Initially, the input twitter data is given to the process to select the suitable feature for data classification. The traditional firefly algorithm is applied and the optimized feature space is adopted for the best fitting results. Once the best feature space is identified through firefly algorithm, the data classification is done using the naive bayes classifier. Here, these two processes are effectively distributed based on the concept given in Map-Reduce framework. The results of the experiment are validated using evaluation metrics namely, computation time, accuracy, specificity and sensitivity. For comparative analysis, proposed big data classification is compared with the existing works of naive bayes and neural network. <s> BIB005 </s> Energy big data: A survey <s> 2) MAIN TOOLS <s> Big Data concerns massive, heterogeneous, autonomous sources with distributed and decentralized control. These characteristics make it an extreme challenge for organizations using traditional data management mechanism to store and process these huge datasets. It is required to define a new paradigm and re-evaluate current system to manage and process Big Data. In this paper, the important characteristics, issues and challenges related to Big Data management has been explored. Various open source Big Data analytics frameworks that deal with Big Data analytics workloads have been discussed. Comparative study between the given frameworks and suitability of the same has been proposed. <s> BIB006 </s> Energy big data: A survey <s> 2) MAIN TOOLS <s> Growth of the internet, Lead to Cloud Computing, Mobile Network and Internet of Things increasing rapidly, big data is becoming a hot-spot in recent years. Data generated from many sources made huge demand for storing, managing, processing and querying on various stream. This gives rise to data processing in our daily life such as mobile devices, RFID and Wireless sensors, which aims at dealing with billions of users interactive data. At real time processing is intently needed integrated system. An entire system is built on one such application Storm, associated with Sql Stream. In this survey use this static data stream and some logs stream and also click stream to show some basic result. To ensure the practical applicability and high efficiency, A Simulation System is established and shown acceptable performance in various expressions using data sheet. It proved that data analysis system for stream and real time processing based on storm can be used in various computing environment. <s> BIB007
Apache Hadoop is one of the most well-known and powerful big data tools written in JAVA BIB003 . Providing platforms and infrastructures for specific big data applications in business and commerce, it plays an important role on big data related companies, such as MapR, Cloudera and Hortonworks BIB004 . Hadoop usually consists of two major components, one is the Hadoop Distributed File System (HDFS) BIB002 and the other is the Hadoop execution engine: MapReduce BIB001 . HDFS is often applied for data storage while MapReduce is used in data processing BIB005 . As for applications of data streaming, operation in the electric power system needs real-time response for data processing. For example, a technology called Storm is designed especially for real-time data stream analysis BIB007 . In order to analyze and process interactively, the data is generated from interactive environments. Users are able to have real-time interaction with computers due to the direct connection to it. Apache Drill is a distributed system used for interactive analysis of the big data BIB006 . The purpose of Drill is to have low latency in response to ad-hoc queries. Therefore, it has the capability to process PetaBytes (PBs) of data and trillions of records within seconds.