Big data storage technologies: a survey <s> Introduction <s> The production environment for analytical data management applications is rapidly changing. Many enterprises are shifting away from deploying their analytical databases on high-end proprietary machines, and moving towards cheaper, lower-end, commodity hardware, typically arranged in a shared-nothing MPP architecture, often in a virtualized environment inside public or private "clouds". At the same time, the amount of data that needs to be analyzed is exploding, requiring hundreds to thousands of machines to work in parallel to perform the analysis. ::: ::: There tend to be two schools of thought regarding what technology to use for data analysis in such an environment. Proponents of parallel databases argue that the strong emphasis on performance and efficiency of parallel databases makes them well-suited to perform such analysis. On the other hand, others argue that MapReduce-based systems are better suited due to their superior scalability, fault tolerance, and flexibility to handle unstructured data. In this paper, we explore the feasibility of building a hybrid system that takes the best features from both technologies; the prototype we built approaches parallel databases in performance and efficiency, yet still yields the scalability, fault tolerance, and flexibility of MapReduce-based systems. <s> BIB001 </s> Big data storage technologies: a survey <s> Introduction <s> Relational databases have been around for many decades and are the database technology of choice for most traditional data-intensive storage and retrieval applications. Retrievals are usually accomplished using SQL, a declarative query language. Relational database systems are generally efficient unless the data contains many relationships requiring joins of large tables. Recently there has been much interest in data stores that do not use SQL exclusively, the socalled NoSQL movement. Examples are Google’s BigTable and Facebook’s Cassandra. This paper reports on a comparison of one such NoSQL graph database called Neo4j with a common relational database system, MySQL, for use as the underlying technology in the development of a software system to record and query data provenance information. <s> BIB002 </s> Big data storage technologies: a survey <s> Introduction <s> In the last two decades, the continuous increase of computational power has produced an overwhelming flow of data. Moreover, the recent advances in Web technology has made it easy for any user to provide and consume content of any form. This has called for a paradigm shift in the computing architecture and large scale data processing mechanisms. Cloud computing is associated with a new paradigm for the provision of computing infrastructure. This paradigm shifts the location of this infrastructure to the network to reduce the costs associated with the management of hardware and software resources. This paper gives a comprehensive survey of numerous approaches and mechanisms of deploying data-intensive applications in the cloud which are gaining a lot of momentum in both research and industrial communities. We analyze the various design decisions of each approach and its suitability to support certain classes of applications and end-users. A discussion of some open issues and future challenges pertaining to scalability, consistency, economical processing of large scale data on the cloud is provided. We highlight the characteristics of the best candidate classes of applications that can be deployed in the cloud. <s> BIB003 </s> Big data storage technologies: a survey <s> Introduction <s> Motivated by requirements of Web 2.0 applications, a plethora of non-relational databases raised in recent years. Since it is very difficult to choose a suitable database for a specific use case, this paper evaluates the underlying techniques of NoSQL databases considering their applicability for certain requirements. These systems are compared by their data models, query possibilities, concurrency controls, partitioning and replication opportunities. <s> BIB004 </s> Big data storage technologies: a survey <s> Introduction <s> The way we are doing science is changing: Data analysis and computation modeling became tightly coupled. Divergent technological trends for computer processors, storage and memory and communication systems showed to be a real challenge in performance of current computing systems. In this paper we analyze the trends that influence computer performance, point out the technical challenges and introduce our vision in developing a guideline to an optimum distribution of computer resources addressing primarily data transmission issues. <s> BIB005 </s> Big data storage technologies: a survey <s> Introduction <s> The CAP theorem asserts that any networked shared-data system can have only two of three desirable properties. However, by explicitly handling partitions, designers can optimize consistency and availability, thereby achieving some trade-off of all three. The featured Web extra is a podcast from Software Engineering Radio, in which the host interviews Dwight Merriman about the emerging NoSQL movement, the three types of nonrelational data stores, Brewer's CAP theorem, and much more. <s> BIB006 </s> Big data storage technologies: a survey <s> Introduction <s> It is already true that Big Data has drawn huge attention from researchers in information sciences, policy and decision makers in governments and enterprises. As the speed of information growth exceeds Moore’s Law at the beginning of this new century, excessive data is making great troubles to human beings. However, there are so much potential and highly useful values hidden in the huge volume of data. A new scientific paradigm is born as dataintensive scientific discovery (DISD), also known as Big Data problems. A large number of fields and sectors, ranging from economic and business activities to public administration, from national security to scientific researches in many areas, involve with Big Data problems. On the one hand, Big Data is extremely valuable to produce productivity in businesses and evolutionary breakthroughs in scientific disciplines, which give us a lot of opportunities to make great progresses in many fields. There is no doubt that the future competitions in business productivity and technologies will surely converge into the Big Data explorations. On the other hand, Big Data also arises with many challenges, such as difficulties in data capture, data storage, data analysis and data visualization. This paper is aimed to demonstrate a close-up view about Big Data, including Big Data applications, Big Data opportunities and challenges, as well as the state-of-the-art techniques and technologies we currently adopt to deal with the Big Data problems. We also discuss several underlying methodologies to handle the data deluge, for example, granular computing, cloud computing, bio-inspired computing, and quantum computing. <s> BIB007 </s> Big data storage technologies: a survey <s> Introduction <s> Most frequent and expensive queries in social networks involve multi-user operations such as requesting the latest tweets or news-feeds of friends. The performance of such queries are heavily dependent on the data partitioning and replication methodologies adopted by the underlying systems. Existing solutions for data distribution in these systems involve hashor graph-based approaches that ignore the multi-way relations among data. In this work, we propose a novel data partitioning and selective replication method that utilizes the temporal information in prior workloads to predict future query patterns. Our method utilizes the social network structure and the temporality of the interactions among its users to construct a hypergraph that correctly models multi-user operations. It then performs simultaneous partitioning and replication of this hypergraph to reduce the query span while respecting load balance and I/O load constraints under replication. To test our model, we enhance the Cassandra NoSQL system to support selective replication and we implement a social network application (a Twitter clone) utilizing our enhanced Cassandra. We conduct experiments on a cloud computing environment (Amazon EC2) to test the developed systems. Comparison of the proposed method with hash- and enhanced graph-based schemes indicate that it significantly improves latency and throughput. <s> BIB008 </s> Big data storage technologies: a survey <s> Introduction <s> Exponential data growth from the Internet, low-cost sensors, and high-fidelity instruments have fueled the development of advanced analytics operating on vast data repositories. These analytics bring business benefits ranging from Web content personalization to predictive maintenance of aircraft components. To construct the data repositories underpinning these systems, rapid innovation has occurred in distributed-data-management technologies, employing schemaless data models and relaxing consistency guarantees to satisfy scalability and availability requirements. These big data systems present many challenges to software architects. Distributed-software architecture quality attributes are tightly linked to both the data and deployment architectures. This causes a consolidation of concerns, and designs must be closely harmonized across these three architectures to satisfy quality requirements. <s> BIB009 </s> Big data storage technologies: a survey <s> Introduction <s> We address the problem of building scalable transaction management mechanisms for multi-row transactions on key-value storage systems, which are commonly termed as NoSQL systems. We develop scalable techniques for transaction management utilizing the snapshot isolation (SI) model. Because the SI model can lead to non-serializable transaction executions, we investigate two conflict detection techniques for ensuring serializability. To support scalability, we investigate system architectures and mechanisms in which the transaction management functions are decoupled from the storage system and integrated with the application-level processes. We present two system architectures and demonstrate their scalability under the scale-out model of cloud computing platforms. In the first system architecture all transaction management functions are executed in a fully decentralized manner by the application processes. The second architecture is based on a hybrid approach in which the conflict detection functions are performed by a dedicated service. We perform a comparative evaluation of these architectures using the TPC-C benchmark and demonstrate their scalability. <s> BIB010
Nowadays, big data is the frontier topic for researchers, as it refers to rapidly increasing amounts of data gathered from heterogeneous devices BIB007 . Sensor networks, scientific experiments, websites, and many other applications produce data in various formats BIB001 . The tendency to shift from structured to unstructured data  makes traditional relational databases unsuitable for storage. This inadequacy of relational databases motivates the development of efficient distributed storage mechanisms. Provision of highly scalable, reliable, and efficient storage for dynamically growing data is the main objective in deploying a tool for big data storage BIB005 . Thus, innovative development of storage systems with improved access performance and fault tolerance is required. Big data has influenced research, management, and business perspectives and has captured the attention of data solution providers toward the deployment of satisfactory technologies for big data storage BIB003 . Relational databases have been very efficient for intensive amounts of data in terms of storage and retrieval processes for many decades BIB002 . However, with the advent and accessibility of the Internet, technology to the public has turned the structure of data towards schema-less, interconnected, and rapidly growing. Apart from that, the complexity of data generated by web resources does not allow the use of relational database technologies for analyzing image data  . The exponential growth, lack of structure, and the variety in types bring data storage and analysis challenges for traditional data management systems  . Transformation of big data structures to relational data models, strictly defined relational schemas, and complexity of procedures for simple tasks are the rigid features of relational databases BIB004 , which are not acceptable to big data. NoSQL technologies introduce flexible data models, horizontal scalability, and schema-less data models BIB009 . These databases aim to provide ease in scalability and management of large-volume data BIB010 . NoSQL databases offer a certain level of transaction handling so that they are adequate for social networking, e-mail, and other web-based applications. To improve the accessibility of data to its users, data are distributed and replicated in more than one site. Replication on the same site not only supports data recovery in case of any damage but also contributes in high availability if replicas are created on different geographic locations  BIB008 . Consistency is another aspect of distributed storage systems when data has multiple copies and keeping the data up-to-date on each site becomes more challenging. BIB006 pointed out that preference to either availability or consistency is a common design objective for distributed databases whereas network partitions are rare. To date, NoSQL technologies have been widely deployed and reported as surveys in the literature, yet state of the art does not provide an in-depth investigation into the features and performance of NoSQL technologies. For instance, BIB003 presented a survey highlighting features and challenges of a few NoSQL databases to deploy on the cloud.  surveyed 15 cloud-based NoSQL databases to analyze read/write optimization, durability, and reliability.  described seven NoSQL databases under key-value, column-oriented, and document categories at an abstract level and classified them with the CAP theorem. Similarly, surveyed nine databases under the same three categories as described by  in the storage section of their survey. Another significant contribution in reviewing big data storage systems was made by , who explained the issues related to massive storage, distributed storage, and big data storage. Their review also covers some well-known database technologies under key-value, column-oriented, and graph data models and categorizes them with Brewer's CAP theorem. However, these three studies did not cover a large number of NoSQL databases in key-value, column-oriented, and document categories. Moreover, graph databases are not considered in these studies. In contrast, BIB002 and  have studied Neo4j, which is a graph database, in comparison with the relational database to observe full-text character searches, security, data scalability, and other data provenance operations.  highlighted the challenges (i.e., volume, variety, velocity, value, and complexity) related to storage of big data over distributed file systems in a different perspective. However, this survey did not aim to report the performance of existing NoSQL databases for the described challenges. Many other comparative studies are present in the literature, analyzing the performance of some specific category or limited features of NoSQL databases. However, the state of the art does not report any detailed investigation of a vast set of performance metrics covering a large number of storage technologies for big data. Therefore, in this paper we highlight the features of distributed database technologies available for big data and present a more comprehensive review. We thoroughly study 26 NoSQL databases in this survey and investigate their performance. Moreover, we describe a number of recent widely spread storage technologies for big data under each data model such as key-value, column-oriented, document, and graph along with their licensing. In addition to that, we have strengthened our analysis with a discussion on the existing and recent trends in Brewer's theorem. In accordance with Brewer's recent explanation for distributed system characterization, we have highlighted each NoSQL database as either consistent or highly available. Therefore, the remarkable contribution of our work is to provide awareness for big data analysts to choose a storage option from a vast variety of databases with better tradeoff between consistency and availability. Furthermore, this study helps researchers understand and leverage an optimum storage solution for their future research work. Following are the key objectives of this paper: (1) to investigate storage structures of a wide range of technologies in a big data environment; (2) to highlight distinctive properties of each storage technology; (3) to develop the taxonomy and evaluate big data storage technologies according to the well-known Brewer theorem presented for distributed systems; (4) to identify the challenges and research directions for coping with big data storage in the future. The rest of the paper is organized as follows: Section 2 describes the evolution of big data storage technologies and their distinctive features over relational databases. Contemporary storage technologies for big data are also detailed in Section 2. Section 3 presents the taxonomy and categorization based on adopted data model and licensing. Section 4 describes Brewer's CAP theorem for distributed systems along with its new explanation. Storage technologies are investigated and analyzed to suggest a type based on Brewer's categorization. Section 5 summarizes the discussion and highlights future research challenges. Section 6 concludes the discussion.