Big Data Architecture, Platform, Application and Trend
Big data has become the hottest topic with the cloud computing and massive data generation. Based on Hadoop, MapReduce and their ecosystem, there are number of improved and optimized big data dealing and analysis platforms. Extensive applications in the Internet of Things, social network, transport, medical, education, and etc. have attracted huge interests and attentions of researchers and industries. To form a hostile profile view, it is necessary to explore big data architecture, platform, application and trend. Abstract architecture and workflow graph of big data is presented, some hot platforms are reviewed and illustrated, and big data application and trend are proposed. The main contribution is to provide detailed and systematic survey and discussion to form a whole technology framework and overview of big data. Introduction Big data is generally thought as with the characteristics of 3Vs (volume, velocity, variety) or expanded 5Vs (volume, velocity, variety, veracity, value). It generally means the petabytes or terabytes level data volume. The huge data generates continuously and exceed the ordinary processing ability of general computers or networks, and the traditional analysis algorithms can’t complete big data in accessible limited time of humans. Big data contains implied semantic and domain knowledge that need to combine with domain experts to perform analysis. Big data are generated from the society life and real world, and the implied knowledge extracted from it can reflect and discover the future development trends, which help people make timely decision and improve the operations of business. The patent value and interest propel the development of big data. Big data is in dynamical evolution, and a number of creativity on architecture, algorithms, platform, and application is bombing in growth. Therefore, the whole prospect of big data technology should be summarized to make more progress and take more advantages of it. The paper is organized as follows. Section 2 introduces related works. Section 3 and 4 describes property, classification, architecture of big data; Section 5 lists main platforms of big data; Section 6 illustrates usual application of big data. In section 7, the trend of big data technology is described. Finally, the conclusion is given. Related Work Big data refers to datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze [1]. Massive data from social media, mobile application, the Internet of things, and more has result to the hot research in big data. In fact, big data has widely sources, including telecommunication, web sites, sales, marketing, RFID enabled logistics, utility of water and energy consumption, finance and insurance, health and prescription, environment, transport, biological, archaeological, geological exploration, nuclear physics, molecular biology, astronomy, and etc.[2]. Big data implies technology of platform, tools, techniques to capture, process, ingest, and visualize large datasets at a reasonable time and cost. Big data also means big data science, framework and infrastructure to implement big data analysis and knowledge discovery [3]. Because big data has various data sources, e.g. file systems and database systems and other, it is expected to access by unified interface. The authors in [4] proposed a unified Data Interface All-iN-A-place (DIANA). A DIANA-based cloud storage system is constructed for versatile, long distance and large volume big data accessing operations by encapsulating multi-stream/multi-path engine at the socket level and a new data communication protocol (CloudJet). Big data storage and processing are considered as one of the main applications for cloud computing systems. In order to integrate big data processing with cloud M2M systems, the paper [5] proposed a Machine to Machine (M2M) communications and enabled novel tele-monitoring architectures built on Remote Telemetry Units and Exhaled Cloud View for E-Health application of big data and IoT systems. Exploring and visualizing very large datasets with scalability is a major research challenge. The system [6] developed by Semantic Web community in the context of the Web of Linked Data is a solution. Because of the new characteristics of big data, new platforms and frameworks are required for management and analysis. Cloud computing can deliver on-demand computing and storage resource over the Internet on a pay-for-use mode. Cloud computing can also accommodate a huge volume of data and methodology to deal with big data. The integration of cloud computing and big data is a trend. Recent research focuses on resource allocation, workflow, security, energy efficiency, and etc. Big Data Property and Classification Big Data Property Big data refers to datasets whose size and complexity is beyond the ability of conventional software tools in accessible time and cost. Big data is opposite to small data. The usual size of big data is big, as compared in Fig.1. Figure 1. The Realms of Data Sizes [7]. Big data definition is generally thought as including three V characteristics, and can be gradually expanded to four, five, or six properties from different views, as shown in Fig.2. Figure 2. From 3Vs, 4Vs, 5Vs to 6Vs Big Data Definition [8]. Big Data Classification According to the characteristics and process procedure of big data, big data can be classified by multiple criteria: data source, data type, content format, data stores, analysis, and processing framework [9]. The classification of big data is as shown in Fig.3 [9]. Based on it, we can expand the big data knowledge and visualization dimension after process framework. Figure 3. Big Data Classification. Big Data Architecture Big data system is comprised of big data ingestion, analysis, visualization, deployment and distribution modules [2]. Big data architecture of application system contains design, development, maintenance, decommissioning and integration. Designing methods can be problem analysis, requirements definition, and business data structure, and etc. [2]. Data view is NoSQL, process view can be process-oriented, business model, task-oriented, and function modules. Based on the big data application in social science [10], we summarize an abstract architecture of big data, which comprises data input, data cleaning, data storing, data mining and analysis, data visualization and knowledge output. This can be shown in Fig.4. The abstract architecture represents that the large dataset processing flow is data input, data cleaning, data storing, data mining and analysis, data visualization and knowledge output. Figure 4. Abstract Architecture of Big Data. According to the view of users centered, the architecture of big data can be simplified to a hierarchical structure, which includes distributed storage, parallel computing, and collaborative pre-process, concurrent inquiring. We present the structure as Fig.5. From the viewpoint of application of big data, we propose the architecture of big data application based on [11], which contains scalable DBMS and DMS, application server cluster, load balance and management, application clients. It is shown as Fig.6. Every layer may be distributed on concurrent. Figure 5. The Hierarchical Structure of Big Data System. Figure 6. The Application Architecture of Big Data . Big data constitutes core technologies, components and mechanisms in data ingesting, storage, processing, analysis, visualization, and delivering results to target applications. Big data may involve the following technologies [3]: (1) Cloud based solutions: Cloud computing can help to provide lower storage costs and stronger computing capability by renting, such as Amazon Web Services (AWS). (2) Virtual file systems: Open source and vendor specific products can implement service based approach and distributed storage. (3) New generation products: Hadoop and NoSQL database can compose and analyze data from heterogeneous data source from sensor network, social media, Internet and etc. (4) Sensors: Sensor devices including cameras, telescopes, medical machines, chemical, physics and biological sensors can continually generate data, which is the one of the main data source needed to be processed and mined. (5) Cluster systems: Cluster computer systems with hundreds and thousands of nodes and benefits of scalability, efficiency and reliability can offer efficient storage and computing power to organize, analyze, and output knowledge. (6) Data analysis: Efficient algorithms and models of computing and statistical and optimization can use large collections of data to analyze fast. (7) Data storage technologies: NAS (Network attached storage) and SAN (Storage area network) can be used in big data. As shown in Fig.7, NIST Big Data Reference Architecture [12] has been proposed in September 2015, which considered security and privacy management in the whole architecture, and added message and communication, resource management module. Figure 7. NIST Big Data Reference Architecture [12]. Big Data Platform There are multitudes of tools and platforms which bring in difficult for users to select the right tools to deal with big data. It is necessary to illustrate various big data tools and platforms. There are many service providers of big data platform, such as IBM, Oracle, Microsoft, HP, SAS, EMC, Amazon, Google. The core big data tools and platforms available in the industry can be classified into analytics & reporting tools, data integration & governance framework. They include broad scopes which may be overlaps: Hadoop Ecosystem, NoSQL Database, In Memory Database, Data Warehousing Application, Streaming Event Processing Frameworks, Search Engines, and Berkeley Data Analytics Stack (BDAS) [13]. Hadoop and its improved version i.e. Spark are foundation and popular. The integration services vendors of big data platforms are Hortonworks, Cloudera, MapR, Palantir, Pivota