Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> We have designed and implemented the Google File System, a scalable distributed file system for large distributed data-intensive applications. It provides fault tolerance while running on inexpensive commodity hardware, and it delivers high aggregate performance to a large number of clients. While sharing many of the same goals as previous distributed file systems, our design has been driven by observations of our application workloads and technological environment, both current and anticipated, that reflect a marked departure from some earlier file system assumptions. This has led us to reexamine traditional choices and explore radically different design points. The file system has successfully met our storage needs. It is widely deployed within Google as the storage platform for the generation and processing of data used by our service as well as research and development efforts that require large data sets. The largest cluster to date provides hundreds of terabytes of storage across thousands of disks on over a thousand machines, and it is concurrently accessed by hundreds of clients. In this paper, we present file system interface extensions designed to support distributed applications, discuss many aspects of our design, and report measurements from both micro-benchmarks and real world use. <s> BIB001 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> In this paper we examine the probl of how to give hands-on assignments in a database syst internals course. We argue that current approaches are inadequate, either because they are not sufficiently comprehensive or because they require using software that has a steep learning curve. We then describe SimpleDB, which is software written expressly for such a course. SimpleDB is a database syst in the spirit of Minibase. Unlike Minibase, however, it supports multiple users and transactions via JDBC, and its code is easy to read and modify. We then describe a course that we teach using SimpleDB, and discuss the educational benefits resulting from it. <s> BIB002 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving). Despite these varied demands, Bigtable has successfully provided a flexible, high-performance solution for all of these Google products. In this paper we describe the simple data model provided by Bigtable, which gives clients dynamic control over data layout and format, and we describe the design and implementation of Bigtable. <s> BIB003 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> We present Scalaris, an Erlang implementation of a distributed key/value store. It uses, on top of a structured overlay network, replication for data availability and majority based distributed transactions for data consistency. In combination, this implements the ACID properties on a scalable structured overlay. By directly mapping the keys to the overlay without hashing, arbitrary key-ranges can be assigned to nodes, thereby allowing a better load-balancing than would be possible with traditional DHTs. Consequently, Scalaris can be tuned for fast data access by taking, e.g. the nodes' geographic location or the regional popularity of certain keys into account. This improves Scalaris' lookup speed in datacenter or cloud computing environments. Scalaris is implemented in Erlang. We describe the Erlang software architecture, including the transactional Java interface to access Scalaris. Additionally, we present a generic design pattern to implement a responsive server in Erlang that serializes update operations on a common state, while concurrently performing fast asynchronous read requests on the same state. As a proof-of-concept we implemented a simplified Wikipedia frontend and attached it to the Scalaris data store backend. Wikipedia is a challenging application. It requires - besides thousands of concurrent read requests per seconds - serialized, consistent write operations. For Wikipedia's category and backlink pages, keys must be consistently changed within transactions. We discuss how these features are implemented in Scalaris and show its performance. <s> BIB004 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> Collaborative web applications such as Facebook, Flickr and Yelp present new challenges for storing and querying large amounts of data. As users and developers are focused more on performance than single copy consistency or the ability to perform ad-hoc queries, there exists an opportunity for a highly-scalable system tailored specifically for relaxed consistency and pre-computed queries. The Web 2.0 development model demands the ability to both rapidly deploy new features and automatically scale with the number of users. There have been many successful distributed key-value stores, but so far none provide as rich a query language as SQL. We propose a new architecture, SCADS, that allows the developer to declaratively state application specific consistency requirements, takes advantage of utility computing to provide cost effective scale-up and scale-down, and will use machine learning models to introspectively anticipate performance problems and predict the resource requirements of new queries before execution. <s> BIB005 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> In CIDR 2009, we presented a collection of requirements for SciDB, a DBMS that would meet the needs of scientific users. These included a nested-array data model, science-specific operations such as regrid, and support for uncertainty, lineage, and named versions. In this paper, we present an overview of SciDB's key features and outline a demonstration of the first version of SciDB on data and operations from one of our lighthouse users, the Large Synoptic Survey Telescope (LSST). <s> BIB006 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> The Hadoop distributed file system (HDFS) is an open source system currently being used in situations where massive amounts of data need to be processed. Based on experience with the largest deployment of HDFS, I provide an analysis of how the amount of RAM of a single namespace server correlates with the storage capacity of Hadoop clusters, outline the advantages of the single-node namespace server architecture for linear performance scaling, and establish practical limits of growth for this architecture. This study may be applicable to issues with other distributed file systems. <s> BIB007 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> How does MongoDB help you manage a huMONGOus amount of data collected through your web application? With this authoritative introduction, you'll learn the many advantages of using document-oriented databases, and discover why MongoDB is a reliable, high-performance system that allows for almost infinite horizontal scalability. Written by engineers from 10gen, the company that develops and supports this open source database, MongoDB: The Definitive Guide provides guidance for database developers, advanced configuration for system administrators, and an overview of the concepts and use cases for other people on your project. Learn how easy it is to handle data as self-contained JSON-style documents, rather than as records in a relational database. Explore ways that document-oriented storage will work for your project Learn how MongoDBs schema-free data model handles documents, collections, and multiple databases Execute basic write operations, and create complex queries to find data with any criteria Use indexes, aggregation tools, and other advanced query techniques Learn about monitoring, security and authentication, backup and repair, and more Set up master-slave and automatic failover replication in MongoDB Use sharding to scale MongoDB horizontally, and learn how it impacts applications Get example applications written in Java, PHP, Python, and Ruby <s> BIB008 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> We present HyperGraphDB, a novel graph database based on generalized hypergraphs where hyperedges can contain other hyperedges. This generalization automatically reifies every entity expressed in the database thus removing many of the usual difficulties in dealing with higher-order relationships. An open two-layered architecture of the data organization yields a highly customizable system where specific domain representations can be optimized while remaining within a uniform conceptual framework. HyperGraphDB is an embedded, transactional database designed as a universal data model for highly complex, large scale knowledge representation applications such as found in artificial intelligence, bioinformatics and natural language processing. <s> BIB009 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> The analysis of the relationship among data entities has lead to model them as graphs. Since the size of the datasets has significantly grown in the recent years, it has become necessary to implement efficient graph databases that can load and manage these huge datasets. ::: ::: In this paper, we evaluate the performance of four of the most scalable native graph database projects (Neo4j, Jena, HypergraphDB and DEX). We implement the full HPC Scalable Graph Analysis Benchmark, and we test the performance of each database for different typical graph operations and graph sizes, showing that in their current development status, DEX and Neo4j are the most efficient graph databases. <s> BIB010 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> We present here a new version of the Arlequin program available under three different forms: a Windows graphical version (Winarl35), a console version of Arlequin (arlecore), and a specific console version to compute summary statistics (arlsumstat). The command-line versions run under both Linux and Windows. The main innovations of the new version include enhanced outputs in XML format, the possibility to embed graphics displaying computation results directly into output files, and the implementation of a new method to detect loci under selection from genome scans. Command-line versions are designed to handle large series of files, and arlsumstat can be used to generate summary statistics from simulated data sets within an Approximate Bayesian Computation framework. <s> BIB011 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> We present new cloud support that employs a single API -- the Datastore API from Google App Engine (GAE) -- to interface to different open source distributed database (DB) technologies. We employ this support to ``plug in'' DB technologies to the API so that they can be used by web applications and services without modification. The system facilitates an empirical evaluation and comparison of these disparate systems by web software developers, and reduces the barrier to entry for their use by automating their configuration and deployment. <s> BIB012 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> What could you do with data if scalability wasn't a problem? With this hands-on guide, you'll learn how Apache Cassandra handles hundreds of terabytes of data while remaining highly available across multiple data centers -- capabilities that have attracted Facebook, Twitter, and other data-intensive companies. Cassandra: The Definitive Guide provides the technical details and practical examples you need to assess this database management system and put it to work in a production environment.Author Eben Hewitt demonstrates the advantages of Cassandra's nonrelational design, and pays special attention to data modeling. If you're a developer, DBA, application architect, or manager looking to solve a database scaling issue or future-proof your application, this guide shows you how to harness Cassandra's speed and flexibility.Understand the tenets of Cassandra's column-oriented structure Learn how to write, update, and read Cassandra data Discover how to add or remove nodes from the cluster as your application requires Examine a working application that translates from a relational model to Cassandra's data model Use examples for writing clients in Java, Python, and C# Use the JMX interface to monitor a cluster's usage, memory patterns, and more Tune memory settings, data storage, and caching for better performance <s> BIB013 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> The Complete Guide to Building Cloud Computing Solutions with Amazon SimpleDB Using SimpleDB, any organization can leverage Amazon Web Services (AWS), Amazons powerful cloud-based computing platformand dramatically reduce the cost and resources associated with application infrastructure. Now, for the first time, theres a complete developers guide to building production solutions with Amazon SimpleDB. Pioneering SimpleDB developer Mocky Habeeb brings together all the hard-to-find information you need to succeed. Mocky tours the SimpleDB platform and APIs, explains their essential characteristics and tradeoffs, and helps you determine whether your applications are appropriate for SimpleDB. Next, he walks you through all aspects of writing, deploying, querying, optimizing, and securing Amazon SimpleDB applicationsfrom the basics through advanced techniques. Throughout, Mocky draws on his unsurpassed experience supporting developers on SimpleDBs official Web forums. He offers practical tips and answers that cant be found anywhere else, and presents extensive working sample codefrom snippets to complete applications. With A Developers Guide to Amazon SimpleDB you will be able to Evaluate whether a project is suited for Amazon SimpleDB Write SimpleDB applications that take full advantage of SimpleDBs availability, scalability, and flexibility Effectively manage the entire SimpleDB application lifecycle Deploy cloud computing applications faster and more easily Work with SELECT and bulk data operations Fine tune queries to optimize performance Integrate SimpleDB security into existing organizational security plans Write and enhance runtime SimpleDB clients Build complete applications using AJAX and SimpleDB Understand low-level issues involved in writing clients and frameworks Solve common SimpleDB usage problems and avoid hidden pitfalls This book will be an indispensable resource for every IT professional evaluating or using SimpleDB to build cloud-computing applications, clients, or frameworks. <s> BIB014 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> This book introduces you to Apache CouchDB, a document-oriented database that offers a different way to model your data. CouchDB is a schema-free database, designed to work with applications that handle document-based information such as contacts, invoices, and receipts. In "CouchDB: The Definitive Guide", three of the core developers gently explain how to work with CouchDB, using clear and practical scenarios. Each chapter showcases key features, such as simple document CRUD (create, read, updated, delete), advanced MapReduce, and deployment tuning for performance and reliability. With this book, you will: understand the basics of document-based storage and manipulation; model data as self-contained JSON documents; manage basic document CRUD; handle evolving data naturally; query and aggregate data in CouchDB, using MapReduce views; replicate data between nodes; and, carry out deployment tuning for performance and reliability. This is the only book available on CouchDB. <s> BIB015 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> If your organization is looking for a storage solution to accommodate a virtually endless amount of data, this book will show you how Apache HBase can fulfill your needs. As the open source implementation of Google's BigTable architecture, HBase scales to billions of rows and millions of columns, while ensuring that write and read performance remain constant. HBase: The Definitive Guide provides the details you require, whether you simply want to evaluate this high-performance, non-relational database, or put it into practice right away. HBase's adoption rate is beginning to climb, and several IT executives are asking pointed questions about this high-capacity database. This is the only book available to give you meaningful answers. * Learn how to distribute large datasets across an inexpensive cluster of commodity servers * Develop HBase clients in many programming languages, including Java, Python, and Ruby * Get details on HBase's primary storage system, HDFS-Hadoop's distributed and replicated filesystem * Learn how HBase's native interface to Hadoop's MapReduce framework enables easy development and execution of batch jobs that can scan entire tables * Discover the integration between HBase and other facets of the Apache Hadoop project <s> BIB016 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> The amount of social data produced by a wide variety of social platforms grows every day. Storing and querying this huge amount of data in almost real time presents a challenge to storage systems in order to scale up to hundreds or thousands of nodes. Also the graph structure and the diverse and changing structure of every single node in social data has to be handled by these systems. In this paper, we describe five storage system types on the basis of eight current open source storage system solutions in order to analyze their application potential. <s> BIB017 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> SummaryMongoDB in Action is a comprehensive guide to MongoDB for application developers. The book begins by explaining what makes MongoDB unique and describing its ideal use cases. A series of tutorials designed for MongoDB mastery then leads into detailed examples for leveraging MongoDB in e-commerce, social networking, analytics, and other common applications. About the TechnologyBig data can mean big headaches. MongoDB is a document-oriented database designed to be flexible, scalable, and very fast, even with big data loads. It's built for high availability, supports rich, dynamic schemas, and lets you easily distribute data across multiple servers. About this BookMongoDB in Action introduces you to MongoDB and the document-oriented database model. This perfectly paced book provides both the big picture you'll need as a developer and enough low-level detail to satisfy a system engineer. Numerous examples will help you develop confidence in the crucial area of data modeling. You'll also love the deep explanations of each feature, including replication, auto-sharding, and deploymentThis book is written for developers. No MongoDB or NoSQL experience required. What's InsideIndexes, queries, and standard DB operations Map-reduce for custom aggregations and reporting Schema design patterns Deploying for scale and high availability==================================================Table of ContentsPART 1 GETTING STARTED A database for the modern web MongoDB through the JavaScript shell Writing programs using MongoDB PART 2 APPLICATION DEVELOPMENT IN MONGODB Document-oriented data Queries and aggregation Updates, atomic operations, and deletes PART 3 MONGODB MASTERY Indexing and query optimization Replication Sharding Deployment and administration <s> BIB018 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> Persistent, user-defined objects present an attractive abstraction for working with non-volatile program state. However, the slow speed of persistent storage (i.e., disk) has restricted their design and limited their performance. Fast, byte-addressable, non-volatile technologies, such as phase change memory, will remove this constraint and allow programmers to build high-performance, persistent data structures in non-volatile storage that is almost as fast as DRAM. Creating these data structures requires a system that is lightweight enough to expose the performance of the underlying memories but also ensures safety in the presence of application and system failures by avoiding familiar bugs such as dangling pointers, multiple free()s, and locking errors. In addition, the system must prevent new types of hard-to-find pointer safety bugs that only arise with persistent objects. These bugs are especially dangerous since any corruption they cause will be permanent. We have implemented a lightweight, high-performance persistent object system called NV-heaps that provides transactional semantics while preventing these errors and providing a model for persistence that is easy to use and reason about. We implement search trees, hash tables, sparse graphs, and arrays using NV-heaps, BerkeleyDB, and Stasis. Our results show that NV-heap performance scales with thread count and that data structures implemented using NV-heaps out-perform BerkeleyDB and Stasis implementations by 32x and 244x, respectively, by avoiding the operating system and minimizing other software overheads. We also quantify the cost of enforcing the safety guarantees that NV-heaps provide and measure the costs of NV-heap primitive operations. <s> BIB019 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> In this paper, we examine a number of SQL and socalled "NoSQL" data stores designed to scale simple OLTP-style application loads over many servers. Originally motivated by Web 2.0 applications, these systems are designed to scale to thousands or millions of users doing updates as well as reads, in contrast to traditional DBMSs and data warehouses. We contrast the new systems on their data model, consistency mechanisms, storage mechanisms, durability guarantees, availability, query support, and other dimensions. These systems typically sacrifice some of these dimensions, e.g. database-wide transaction consistency, in order to achieve others, e.g. higher availability and scalability. <s> BIB020 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> Reliability and scalability of an application is dependent on how its application state is managed. To run applications at massive scale requires one to operate datastores that can scale to operate seamlessly across thousands of servers and can deal with various failure modes such as server failures, datacenter failures and network partitions. The goal of Amazon DynamoDB is to eliminate this complexity and operational overhead for our customers by offering a seamlessly scalable database service. In this talk, I will talk about how developers can build applications on DynamoDB without having to deal with the complexity of operating a large scale database. <s> BIB021 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> Current serving systems lack the ability to bulk load massive immutable data sets without affecting serving performance. The performance degradation is largely due to index creation and modification as CPU and memory resources are shared with request serving. We have extended Project Voldemort, a general-purpose distributed storage and serving system inspired by Amazon's Dynamo, to support bulk loading terabytes of read-only data. This extension constructs the index offline, by leveraging the fault tolerance and parallelism of Hadoop. Compared to MySQL, our compact storage format and data deployment pipeline scales to twice the request throughput while maintaining sub 5 ms median latency. At LinkedIn, the largest professional social network, this system has been running in production for more than 2 years and serves many of the data-intensive social features on the site. <s> BIB022 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> Ubuntu Unleashed is filled with unique and advanced information for everyone who wants to make the most of the Ubuntu Linux operating system. This new edition has been thoroughly revised and updated by a long-time Ubuntu community leader to reflect the exciting new Ubuntu 11.10 (Oneiric Ocelot) and the forthcoming Ubuntu 12.04. Former Ubuntu Forum administrator Matthew Helmke covers all you need to know about Ubuntu 11.10/12.04 installation, configuration, productivity, multimedia, development, system administration, server operations, networking, virtualization, security, DevOps, and moreincluding intermediate-to-advanced techniques you wont find in any other book. Helmke presents up-to-the-minute introductions to Ubuntus key productivity and Web development tools, programming languages, hardware support, and more. Youll find brand-new coverage of the new Unity desktop, new NoSQL database support and Android mobile development tools, and many other Ubuntu 11.10/12.04 innovations. Whether youre new to Ubuntu or already a power user, youll turn to this book constantly: for new techniques, new solutions, and new ways to do even more with Ubuntu! Matthew Helmke served from 2006 to 2011 on the Ubuntu Forum Council, providing leadership and oversight of the Ubuntu Forums, and spent two years on the Ubuntu regional membership approval board for Europe, the Middle East, and Africa. He has written about Ubuntu for several magazines and websites, is a lead author of The Official Ubuntu Book. He works for The iPlant Collaborative, which is funded by the National Science Foundation and is building cyberinfrastructure for the biological sciences to support the growing use of massive amounts of data and computationally intensive forms of research. Quickly install Ubuntu, configure it, and get your hardware running right Configure and customize the new Unity desktop (or alternatives such as GNOME) Get started with multimedia and productivity applications, including LibreOffice Manage Linux services, users, and software packages Administer and use Ubuntu from the command line Automate tasks and use shell scripting Provide secure remote access Manage kernels and modules Administer file, print, email, proxy, LDAP, and database services (both SQL and NoSQL) Use both Apache and alternative HTTP servers Support and use virtualization Use Ubuntu in cloud environments Learn the basics about popular programming languages including Python, PHP, and Perl, and how to use Ubuntu to develop in them Learn how to get started developing Android mobile devices Ubuntu 11.10 on DVD DVD includes the full Ubuntu 11.10 distribution for Intel x86 computers as well as the complete LibreOffice office suite and hundreds of additional programs and utilities. Free Upgrade! Purchase this book anytime in 2012 and receive a free Ubuntu 12.04 Upgrade Kit by mail (U.S. or Canada only) after Ubuntu 12.04 is released. See inside back cover for details. <s> BIB023 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> SummaryRedis in Action introduces Redis and walks you through examples that demonstrate how to use it effectively. You'll begin by getting Redis set up properly and then exploring the key-value model. Then, you'll dive into real use cases including simple caching, distributed ad targeting, and more. You'll learn how to scale Redis from small jobs to massive datasets. Experienced developers will appreciate chapters on clustering and internal scripting to make Redis easier to use. About the TechnologyWhen you need near-real-time access to a fast-moving data stream, key-value stores like Redis are the way to go. Redis expands on the key-value pattern by accepting a wide variety of data types, including hashes, strings, lists, and other structures. It provides lightning-fast operations on in-memory datasets, and also makes it easy to persist to disk on the fly. Plus, it's free and open source. About this bookRedis in Action introduces Redis and the key-value model. You'll quickly dive into real use cases including simple caching, distributed ad targeting, and more. You'll learn how to scale Redis from small jobs to massive datasets and discover how to integrate with traditional RDBMS or other NoSQL stores. Experienced developers will appreciate the in-depth chapters on clustering and internal scripting. Written for developers familiar with database concepts. No prior exposure to NoSQL database concepts nor to Redis itself is required. Appropriate for systems administrators comfortable with programming. What's InsideRedis from the ground up Preprocessing real-time data Managing in-memory datasets Pub/sub and configuration Persisting to diskAbout the Author Dr. Josiah L. Carlson is a seasoned database professional and an active contributor to the Redis community. Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications. Table of ContentsPART 1 GETTING STARTED Getting to know Redis Anatomy of a Redis web application PART 2 CORE CONCEPTS Commands in Redis Keeping data safe and ensuring performance Using Redis for application support Application components in Redis Search-based applications Building a simple social network PART 3 NEXT STEPS Reducing memory use Scaling Redis Scripting Redis with Lua <s> BIB024 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> In the past, relational databases were used in a large scope of applications due to their rich set of features, query capabilities and transaction management. However, they are not able to store and process big data effectively and are not very efficient to make transactions and join operations. Recently, emerge a new paradigm, NoSQL databases, to overcome some of these problems, which are more suitable for the usage in web environments. In this paper, we describe NoSQL databases, their characteristics and operational principles. The main focus of this paper is to compare and evaluate two of the most popular NoSQL databases: MongoDB and Cassandra. <s> BIB025 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> A description and discussion of the SciDB database management system focuses on lessons learned, application areas, performance comparisons against other solutions, and additional approaches to managing data and complex analytics. <s> BIB026 </s> Big data storage technologies: a survey <s> Contemporary big data storage technologies <s> Many problems in computational neuroscience, neuroinformatics, pattern/image recognition, signal processing and machine learning generate massive amounts of multidimensional data with multiple aspects and high dimensionality. Tensors (i.e., multi-way arrays) provide often a natural and compact representation for such massive multidimensional data via suitable low-rank approximations. Big data analytics require novel technologies to efficiently process huge datasets within tolerable elapsed times. Such a new emerging technology for multidimensional big data is a multiway analysis via tensor networks (TNs) and tensor decompositions (TDs) which represent tensors by sets of factor (component) matrices and lower-order (core) tensors. Dynamic tensor analysis allows us to discover meaningful hidden structures of complex data and to perform generalizations by capturing multi-linear and multi-aspect relationships. We will discuss some fundamental TN models, their mathematical and graphical descriptions and associated learning algorithms for large-scale TDs and TNs, with many potential applications including: Anomaly detection, feature extraction, classification, cluster analysis, data fusion and integration, pattern recognition, predictive modeling, regression, time series analysis and multiway component analysis. ::: Keywords: Large-scale HOSVD, Tensor decompositions, CPD, Tucker models, Hierarchical Tucker (HT) decomposition, low-rank tensor approximations (LRA), Tensorization/Quantization, tensor train (TT/QTT) - Matrix Product States (MPS), Matrix Product Operator (MPO), DMRG, Strong Kronecker Product (SKP). <s> BIB027
Research outcomes of exploring storage technologies for big data advocate different aspects of designing storage mechanisms. These reliable and highly available mechanisms contribute to improving data access performance. Improved data access performance drives better quality of data analysis. These technologies offer scalable storage solutions for growing big data with enhanced data structures and support fault tolerance. This section provides a brief explanation of storage systems for big data in each category. In accomplishing the design goals, current storage technologies are described to review their feasibility for big data. Their storage structure and outstanding features to support scalable resource provisioning for big data are also described here. The Google File System (GFS) is a proprietary system developed by Google Inc. BIB001 to manage data-intensive applications in a distributed manner. It is designed to satisfy the storage needs for steadily growing data as a significant objective along with other features provided by contemporary techniques. Current and future estimated workloads are analyzed to develop such a distributed file system. To deal with the commodity component failure problem, GFS facilitates continuous monitoring and ensures detecting errors, tolerates component faults, and recovers them automatically. GFS adopts a clustered approach that divides data chunks into 64-KB blocks and stores a 32-bit checksum for each block. As shown in Fig. 1 , these checksums are stored on servers as part of metadata to ensure integrity. Moreover, the chunks are replicated to avoid chunk server faults and for availability and reliability . Such systems are supposed to handle large-volume data where many kilo-byte size files become a challenge for them. However, GFS guarantees support to manage these small files along with appending new data concurrently for large files even when they are read/write-intensive. The Hadoop Distributed File System (HDFS) is developed as an inspiration of GFS. HDFS is a distributed, scalable storage system designed as a core of Apache Hadoop to run on inexpensive commodity hardware, initially designed as infrastructure of Apache Nutch. HDFS is a suitable solution for dataintensive applications, typically at gigabyte to terabyte scales, which require high throughput. HDFS provides quick fault detection and automatic recovery, as it comprises a large number of components. However, there is a probability of block failure and nonfunctioning  . Block replication is offered to avoid node failure and unavailability or loss of data BIB007 . Replication ensures not only the availability but also the reliability of the system and it is automatically handled by HDFS NameNode. Rather than just being a storage layer of Hadoop, HDFS is a standalone distributed file system that helps improve the throughput of the system. HDFS has a namespace-separated architecture. Metadata is stored on the master node, which is called NameNode, whereas block-split files are stored on a number of DataNodes. NameNode performs mapping of data on DataNodes and namespace operations such as open, close, and rename file. DataNodes fulfill read-write requests and create block and replicas. The architecture of HDFS is shown in Fig. 2 . BigTable is another product developed by Google Inc. BIB003 to provide a flexible and high-performance storage for large-scale structured data spread over a large number of commodity servers. BigTable is a highly adaptable, reliable, and applicable storage system to manage petabyte scale data on thousands of machines. BigTable has proven to be a high-performance and available data storage. Therefore, many Google applications and projects that might have throughput or latency requirements are using BigTable with different workloads. Moreover, BigTable provides dynamic control over data placement, representation, indexing, and clustering. HBase is a column-oriented data store developed by Apache (Apache Software Foundation, 2015) . HBase implements a BigTable storage system and is designed to handle storage needs of big data in the Apache project BIB016 . HBase provides a scalable, distributed, fault-tolerant, and random read-write oriented access to big data on top of Hadoop and HDFS (Fig. 3)  . HBase uses the underlying HDFS from Hadoop to store table data. Its master component assigns regions to RegionServers and keeps track of these servers. RegionServers receive region information and manage read/write requests from clients on different regions  . As far as concurrency is concerned, HBase has a remarkable support to read-intensive transactions BIB017 . HBase is more concerned about distribution and uses a clustered approach for data management; that is why it is a potential solution for a remarkably large number of rows of data. Hypertable also implements BigTable (Hypertable, 2015) . Hypertable is a distributed database that provides a very good support to consistency of stored data  . Hypertable is designed to run and is compatible with many distributed file systems such as GFS, HDFS, and CloudStore. Hypertable stores data in the form of tables and splits the tables to achieve distribution and scalability. Fig. 3 describes the relationship between the components of the Hypertable system. HyperSpace works as a lock manager and ensures high availability and consistency through distributed configuration of replicas by using a distributed consensus protocol. There is no effect on client data transfer if the master becomes unresponsive for a short period. This is preferable to have more Masters to achieve high availability requirements  . MongoDB is an open-source NoSQL database designed by Mongo, Inc. as a highly available, scalable, and fault-tolerant document-oriented solution  . MongoDB derives the characteristics of MySQl with the JSON data model BIB017 . Therefore, MongoDB has the same horizontal scalability, ease of agile development with dynamic schema support to all kinds of document data, and efficiency in manageability as MySQL offers. Furthermore, indexing, dynamic and ad hoc queries, aggregation, and dynamic updates are some of the vast MySQL facilities that are adopted by MongoDB with slight modification. MongoDB stores documents as data in binary representation called BSON, which makes it easy to map data from applications to databases BIB008 . Moreover, robustness and availability of data are achieved with a replica set, and multiple replicas are available when the primary server is not responding. This database provides support for storing and  combining multiple structure data, whereas the indexing is still feasible BIB018 . Terrastore is another document-based opensource distributed storage system, developed by Terracotta, Inc.  . Terrastore offers support with high scalability as well as consistency and dynamic clustering during execution. Terrastore is built on top of the in-memory storage, Terracotta. To provide scalability, Terrastore offers not only the deployment of a single cluster but also support to multiple clusters. Furthermore, load balancing for data storage over nodes is automatically performed when nodes connect or disconnect from the clusters  ). Terrastore is optimized for consistency, as it performs consistent hashing for partitioning and does not create replicas. However, data are redistributed when nodes join or leave the cluster. Like DynamoDB, Voldemort, and Cassandra, the Terrastore node nearby a client forwards the client requests to the serving node and it is not necessary for the client to have this routing knowledge. Although the built-in support to data distribution and partitioning makes Terrastore better than Memcached, Terrastore has more distribution overhead than Redis, which lowers its performance (Alex and Ana-Maria, 2009). Consequently, Terrastore is not considered a good approach for large amount of document data  . HyperGraphDB is an implementation of the hypergraph model to develop an open-source graph database for artificial intelligence and web semantic projects BIB009 . This graph database uses Berkeley DB for storage and not only provides storage for arbitrary data but also supports data mapping between host language and storage. In addition, an essential customizable indexing feature is provided by HyperGraphDB so that efficient data retrieval and graph traversal are achieved. HyperGraphDB uses a key-value mechanism to store graph information such as nodes and edges as a key BIB010 . In contrast to HBase, HyperTable, Redis, and other master-slave storage systems and access architectures, HyperGraphDB implements a peer-topeer data distribution mechanism. Each peer works independently, and updates are performed asynchronously and eventually  . InfiniteGraph is a distributed object database designed for graph data that offers high scalability as well as complex search and traversing efficiency  . InfiniteGraph is a graph database where data and computation load are distributed among storage locations and runtime configurable zones. To achieve scalability, InfiniteGraph implements a model-based technique. Moreover, an InfiniteGraph database facilitates the creation of a custom model for frequently accessed elements so that query performance is improved. These frequently accessed elements can further be separated from other elements to handle locks. Along with custom models, the indexing framework of InfiniteGraph also contributes to better query performance  . InfiniteGraph database users know that their growing graph data are stored with some schema and they can easily perform normalization and other presentation operations on their data  . Furthermore, because of deploying its model on cloud, development and testing of databases over InfiniteGraph is without any charge. Rocket U2 is a document-oriented, proprietary database management suite that offers two associated platforms of multivalue databases: UniData and UniVerse . Rocket U2 ensures availability of the system after any disconnection, which makes it reliable and robust to immediately recover from any downtime (RocketSoftware, 2014a). According to a case study  , Rocket U2 provides a scalable and flexible system where changes can be made anytime according to the requirements. This flexibility allows easy updating in applications, and the database does not need to be rebuilt when changes occur. Multivalue products are integrated with many pieces of codes, which reduces the developers' effort of coding. Furthermore, Rocket U2 databases sustain their performance in terms of response time even when the size of data grows. However, support to data partitioning is not available with Rocket U2. Scalaris is a scalable, distributed, and highly available key-value store that is designed to fulfill the intensive read/write requirements of enterprises BIB004 . For concurrent access and to support intensive transaction, it is common to face node failures. However, Scalaris manages to obtain consistency for critical write operations by implementing a distributed transaction protocol and a structured overlay network on Erlang. The structured overlay network is implemented as the first layer of Scalaris architecture and provides support to a range of queries. The architecture of Scalaris also applies replication to make the system highly available and fault-tolerant. Regardless of the challenges of writeintensive transactions, the Paxos consensus protocol helps ensure consistency with reduced overhead (Scalaris, 2015) . To achieve high performance, custom data management, and scalability for applications, Berkeley DB provides an embedded support to key-value databases via libraries. Although Berkeley DB offers an open-source license, this license is constrained to the development of open-source applications. For third-party distribution, Berkeley DB offers commercial licensing facility. Berkeley DB was initially licensed under Sleepycat Software but later owned by Oracle. The products developed using Berkeley DB have simple data access and management procedures. Furthermore, due to customized configuration with no overhead, these products fulfill varying development needs such as local or distributed storage and storage capacity configuration, and many other options are available when Berkeley DB is used (Oracle, 2015b) . As far as other features related to data distribution are concerned, Berkeley DB ensures high throughput for data access rate, non-blocking consistency, reliability, fault tolerance, and availability. Like relational databases, Berkeley DB has a good support to data access efficiency, transactions, and recovery mechanism  . However, provision to these relational databases-like features is packaged in libraries, and developers are given the configuration choice according to their requirements. Regardless of the features provided by Berkeley DB, it is noticeable that Berkeley DB does not support synchronization with other databases. Meanwhile, data partitioning is not facilitated by Berkeley DB (Oracle, 2015c) . DynamoDB is a widely used, fully distributed, schema-free NoSQL database tool by Amazon BIB021 . DynamoDB is predominantly used for storing unstructured, variable, and scalable data  . DynamoDB offers infinite scaling capacity for data storage and access rate. It is applicable when frequent updates on distributed sites, efficient indexing, and adaptive scalability are required. Availability and durability can be achieved through automatic replication over multiple zones in a region  . Regardless of the request size, DynamoDB ensures stable performance and visible resource utilization. Administrative tasks, such as hardware provisioning, setup and configuration, replication, software patching, and scaling, can be offloaded to Amazon Web Service (AWS). DynamoDB is an efficiently managed database; that is why it is suitable for scalable applications with exponentially growing data  . Qualcomm has developed a document-oriented database named Qizx for large-volume, highperformance, and transaction-intensive XML data  . Qizx is well suited for enterprises with fast data access and retrieval requirements. For this purpose, Qizx has automatic full-text and customizable indexing capability, which makes it efficient in query responses. In addition, support to terabyte-scale data and other features related to distribution are taken into consideration while designing this database. Qizx's 'shared-nothing' clustering architecture allows each node or host in cluster to fully replicate the database to achieve high availability  . However, these share-nothing hosts serve client read requests, and one dedicated server is responsible for update operations. Consequently, a consistent, fault-tolerant, and up-to-date replicating database system is achieved. An industry-leading, broadly used, schema-less graph database that is effectively replacing relational databases is  . By analyzing complex data relationships with Neo4j, enterprises gain value benefits. In regard to scalability  , high transaction load, concurrency, and performance for read request workloads, Neo4j not only competes other graph databases but also affects improvements on its older versions  . Support to write-intensive transactions without blocking is achieved with the help of buffering. In spite of its benefits, Neo4j is time consuming in setting up a ready-for-production and reliable database. For real-time web applications, RethinkDB is the first open-source document database that supports complex queries in an efficient way  . RethinkDB reverses traditional storage systems and implements append-only nested data storage structure, which makes it more consistent, robust, and easily replicable  . RethinkDB offers easy scalability and faster response to client's real-time queries. Although RethinkDB also uses the JSON data model, query operation processing is slower than that of MongoDB. However, RethinkDB has a real-time architecture and, due to its appendonly storage structure, RethinkDB incurs less scaling cost as compared to MongoDB. Furthermore, advanced query language and administration are better than with MongoDB. Despite the benefits obtained by RethinkDB, RethinkDB does not perform very well for compute-and write-intensive applications. Aerospike is the first flash-optimized, opensource, key-value data store for real-time data, which offers scalability and reliability with very low cost  . Aerospike has a 'shared-nothing' architecture, which supports petabyte scale volume of data with reliability and linear scalability  . Similar to Cassandra, a column-oriented database, Aerospike is designed to provide scalability, replication, automatic node partition recovery, and high availability. However, Aerospike is better than Cassandra in terms of read and write workload, consistency with jitters, and cost for read and write operations. The first multi-model, open-source, and highly scalable database for document data with an extended, transparently managed graph layer to provide connections between records as relationships is OrientDB  . The embedding graph layer makes OrientDB not only persistent and compact but also fast in data traversal and managing data relationships without increasing the cost. The multi-model platform of OrientDB makes it categorizable as a document-oriented graph database. Regardless of volume of data, OrientDB has a distinct speed in data storage as well as read and write transactions. To ensure performance and scalability, clustering and replication over heterogeneous servers is a significant benefit achieved by OrientDB (SD Times Newswire, 2013). OrientDB has many features similar to Couchbase document store, which is derived from CouchDB. Both have a schema-free model, support to secondary indexes, sharding for data partitioning, and the master-master replication model. However, OrientDB has additional features such as a multi-model approach to support all operating systems, which are not provided by Couchbase. AllegroGraph is a graph database efficiently designed for RDF semantic web applications. AllegroGraph ensures persistency, memory utilization performance, and high scalability over millions of quads (AllegroGraph, 2015) . Furthermore, automatic compute and storage resources management in AllegroGraph are the key performance fundamentals. AllegroGraph offers not only an enterprise and commercial version but also an open-source product. However, the open-source version of AllegroGraph is limited in scalability. Hybrid data representation, which covers both relational model and triple store, optimizes the range query performance in AllegroGraph  . Redis is an alternative open-source cache and key-value store for big data, which provides an efficient data structure for indexing to speed up query operations and response BIB024 . Somehow, the functionality of Redis is similar to that of Scalaris. Redis has considerable support to replication in a master-slave environment. However, providing support to multiple data structure makes it a predominant choice for frequent data access situations. Linux is the recommended operating system for its deployment, as Redis is developed and tested on the Linux platform BIB011 . In the case of more scalability requirements when systems become out of RAM, Aerospike in-memory store is preferable to Redis. Redis is much closer to MemcacheDB in fast data access, as the whole data resides in the memory. However, Redis has some other powerful features such as built-in persistency and support to more datatypes. Unlike MemcacheDB, Redis is persistent as a real database and data will not be lost with restart. Furthermore, support to more data types is a unique property of Redis. Thus, Redis becomes a best choice when high scalability, heterogeneity in platform, servers and application, and in-memory data are required. Voldemort is a general-purpose, distributed data storage solution for large-scale data BIB022 Voldemort, 2015) . The routing module of Voldemort is responsible for performing data clustering and replication and follows the same procedure as DynamoDB. Voldemort provides eventual consistency, as read/write operations can be performed on any node and for a very short period as happened when the view of data is inconsistent BIB012 . Like MemcacheDB, Voldemort also achieves data persistence using Berkeley DB, and the replication factor for distributed hash table entries can be set by the developer. KAI is a distributed, key-value-based data store that is developed as an implementation of Amazon's Dynamo. KAI provides high scalability and availability to websites with support to a variety of data. The name KAI is derived to elaborate its good organization of clusters where data are stored and processed in parallel in a distributed, replicated manner (SourceForge, 2015) . Furthermore, load balancing, configurable scalability of nodes, fast configuration, and reliability through replication are the basic performance gains of KAI. Although KAI offers eventual consistency, the implementation of the quorum protocol helps achieve strong consistency. Cassandra is a decentralized and highly available key-value storage system for structured data that comprises a large number of underlying data centers BIB013  . It serves to provide scalability, instance storage, and improved performance for frequent read/write operation requests. Data consistency is achieved in Cassandra through periodic updates on replicating sites  . In contrast to HBase, Cassandra provides significant support to write-intensive transactions BIB017 . Most significantly, Cassandra persistently deals with component failures in such large-scale systems and achieves overall reliability and scalability. Schema can be built any time, as Cassandra provides flexibility. Clustering, partitioning, and replication while taking care of fault tolerance, reduced latency, and scalability are the prominent features of Cassandra  BIB025 . It is a reasonable choice for continuously growing systems such as Facebook. Facebook has deployed Cassandra, where scalability linearly depends upon the number of users and a write-intense system is needed BIB005 . SimpleDB is an open-source, document-oriented database that is available as an Amazon service BIB002 . Without database administration load, SimpleDB ensures high availability and durability of data with automatic geographic replication. Furthermore, the data model is flexible, and automatic indexing is performed on the data. Thus, automatic provisioning of database administration makes application development simple via SimpleDB. Despite the ease of data management provided by SimpleDB, scalability is limited to 10 GB BIB014 . MemcacheDB is an open-source key-value store  MemcacheDB, 2015) developed for fast and reliable storage and access of objects and data. It implements Berkeley DB to develop a general-purpose database, and many features of Berkeley DB such as transaction and replication are derived. MemcacheDB is a persistent database for dynamic, database-driven web applications BIB019 . Memcached users can access MemcacheDB through any API, as it uses the Memcached protocol. When implementing Memcached API, a cache system for fast access from memory, it becomes easy to provide data persistency as caching is already performed by Memcached BIB023 . It applies a master-slave approach to achieve data persistency where the data read request can be fulfilled by any site but write operation is performed only at the master node. It operates with a single master and multiple replicating nodes. However, it is flexible to allow any node to become a replicating node, but it reveals a security challenge BIB012 . CouchDB is an open-source, distributed, scalable document database developed for performing data operations and administration on the web BIB015  . In case of intensive access requests, CouchDB adopts a sophisticated concurrency mechanism that does not let the system fail. However, such intensive workload cause delays in overall responses. CouchDB has support to dynamic data structure, which makes it possible to define schema as required. Regardless of wide adaptability of CouchDB in web application development, it is suggested to check its feasibility for developer requirements (Wenk and Slater, 2014) . Like MongoDB, CouchDB also implements the JSON data model, which helps CouchDB support semistructured data. Furthermore, CouchDB allows the storage of any kind of data as documents BIB017 . Riak is a key-value database that provides high availability and less cost scalability  . Riak provides simpler data models and data conflict resolution. Riak is an advanced form of key-value databases but somehow it is categorized as a document-based database. Although Riak has more functionalities than other key-value databases, it lacks the features of document-based databases BIB020 . Therefore, it is better to categorize Riak as a key-value database. As compared to Redis, Riak has a less complex key construct, which is a feature of document-based databases BIB017 . This less complexity to key construct makes the data model of Riak flexible, and changes in the data model do not affect the indexes. From an analytics perspective, tensor-based models are widely adopted to represent arbitrary relationships in data entities and to traverse graphs BIB027 . Big data has high dimensionality rather than two-tuple relationships and brings the challenge of analyzing such multi-relational datasets. Therefore, tensor-based models are being used for big data representation. SciDB BIB006 , an implementation of the tensor model, is designed to manage multidimensional data as an array database, which creates chunks of arrays for parallel processing. SciDB supports many data manipulation operations such as linear algebra operators for chunkbased N-dimensional arrays. Attributes and dimensions of data are flexible and can be added or removed any time. SciDB is an append-only array database for scientific data, and the changes or corrections made in data are stored as new versions BIB026 . In this section, we have provided a comprehensive account of current storage technologies for big data. Table 2 provides a brief summary of above discussed storage technologies available for big data. It highlights the design objectives and characteristics of these technologies. It comprises four vertical divisions, providing the name of the storage technology, the vendors, and the design goals. Development of distributed storage mechanism, which partitions the data in a NoSQL database format, provides scalability and efficient and fault-tolerant support to read/write operations on that data. Table 2 provides a summary of big data storage technologies. It can be seen that most of the storage technologies are designed to ensure scalability and consistency in general. However, these storage systems support specific features such as intensive workload, high availability, data access performance, and complex query operations. We provide a comparative table identifying the application areas of each storage technology in Table 3 . The following section highlights these features to categorize the above discussed storage systems. Taxonomy based on the adopted data model of big data storage technologies is presented and detailed in the next section.