Big Data analytics has been approached exclusively from a data-parallel perspective, where data are partitioned to multiple workers ‐ threads or separate servers ‐ and model training is executed concurrently over different partitions, under various synchronization schemes that guarantee speedup and or convergence. The dual ‐ Big Model ‐ problem that, surprisingly, has received no attention in database analytics, is how to manage models with millions if not billions of parameters that do not fit in memory. This distinction in model representation changes fundamentally how in-database analytics tasks are carried out. In this paper, we introduce the first secondary storage array-relation dot-product join operator between a set of sparse arrays and a dense relation targeted. The paramount challenge in designing such an operator is how to optimally schedule access to the dense relation based on the sparse non-contiguous entries in the sparse arrays. We prove that this problem is NP-hard and propose a practical solution characterized by two important technical contributions— dynamic batch processing and array reordering. We execute extensive experiments over synthetic and real data that confirm the minimal overhead the operator incurs when sufficient memory is available and the graceful degradation it suffers as memory resources become scarce. Moreover, dot-product join achieves an order of magnitude reduction in execution time for Big Model analytics over alternative in-database solutions.
As we have already discussed in the paper, the dot-product join operator is a constrained formulation of the standard sparse matrix vector (SpMV) multiplication problem. Specifically, the constraint imposes the update of the vector after each multiplication with a row in the sparse matrix. This makes impossible the direct application of SpMV kernels to Big Model analytics---beyond BGD. Moreover, we consider the case when the vector size goes beyond the available memory. SpMV is an exhaustively studied problem with applications to high performance computing, graph algorithms, and analytics. An extended discussion of the recent work on SpMV is presented in @cite_33 ---on which we draw in our discussion. @cite_34 and @cite_40 propose optimizations for SpMV in multicore architectures, while @cite_10 and @cite_30 optimize distributed SpMV for large scale-free graphs with 2D partitioning to reduce communication between machines. Array databases such as SciDB @cite_58 and Rasdaman @cite_35 support SpMV as calls to optimized linear algebra libraries such as Intel MKL and Trilinos. There has also been preliminary research on accelerating matrix multiplication with GPUs @cite_21 and SSDs @cite_33 , showing that speedups are limited by I O and setup costs. None of these works store the vector in secondary storage.
@cite_30 @cite_35 @cite_33 @cite_21 @cite_58 @cite_40 @cite_34 @cite_10
2124007994 2128853364 1987840949 2101511474
The massive parallelism of graphics processing units (GPUs) oers tremendous performance in many high-performance computing applications. While dense linear algebra readily maps to such platforms, harnessing this potential for sparse matrix computations presents additional challenges. Given its role in iterative methods for solving sparse linear systems and eigenvalue problems, sparse matrix-vector multiplication (SpMV) is of singular importance in sparse linear algebra. In this paper we discuss data structures and algorithms for SpMV that are eciently implemented on the CUDA platform for the ne-grained parallel architecture of the GPU. Given the memory-bound nature of SpMV, we emphasize memory bandwidth eciency and compact storage formats. We consider a broad spectrum of sparse matrices, from those that are well-structured and regular to highly irregular matrices with large imbalances in the distribution of nonzeros per matrix row. We develop methods to exploit several common forms of matrix structure while oering alternatives which accommodate greater irregularity. On structured, grid-based matrices we achieve performance of 36 GFLOP s in single precision and 16 GFLOP s in double precision on a GeForce GTX 280 GPU. For unstructured nite-element matrices, we observe performance in excess of 15 GFLOP s and 10 GFLOP s in single and double precision respectively. These results compare favorably to prior state-of-the-art studies of SpMV methods on conventional multicore processors. Our double precision SpMV performance is generally two and a half times that of a Cell BE with 8 SPEs and more than ten times greater than that of a quad-core Intel Clovertown system. Sparse matrix-vector multiplication (SpMV) is of singular importance in sparse linear algebra. In contrast to the uniform regularity of dense linear algebra, sparse operations encounter a broad spectrum of matrices ranging from the regular to the highly irregular. Harnessing the tremendous potential of throughput-oriented processors for sparse operations requires that we expose substantial fine-grained parallelism and impose sufficient regularity on execution paths and memory access patterns. We explore SpMV methods that are well-suited to throughput-oriented architectures like the GPU and which exploit several common sparsity classes. The techniques we propose are efficient, successfully utilizing large percentages of peak bandwidth. Furthermore, they deliver excellent total throughput, averaging 16 GFLOP s and 10 GFLOP s in double precision for structured grid and unstructured mesh matrices, respectively, on a GeForce GTX 285. This is roughly 2.8 times the throughput previously achieved on Cell BE and more than 10 times that of a quad-core Intel Clovertown system. Sparse matrix-vector multiplication (SpMV) is an important kernel in many scientific applications and is known to be memory bandwidth limited. On modern processors with wide SIMD and large numbers of cores, we identify and address several bottlenecks which may limit performance even before memory bandwidth: (a) low SIMD efficiency due to sparsity, (b) overhead due to irregular memory accesses, and (c) load-imbalance due to non-uniform matrix structures. We describe an efficient implementation of SpMV on the IntelR Xeon PhiTM Coprocessor, codenamed Knights Corner (KNC), that addresses the above challenges. Our implementation exploits the salient architectural features of KNC, such as large caches and hardware support for irregular memory accesses. By using a specialized data structure with careful load balancing, we attain performance on average close to 90 of KNC's achievable memory bandwidth on a diverse set of sparse matrices. Furthermore, we demonstrate that our implementation is 3.52x and 1.32x faster, respectively, than the best available implementations on dual IntelR XeonR Processor E5-2680 and the NVIDIA Tesla K20X architecture. Sparse matrix-vector multiplication (spMVM) is the most time-consuming kernel in many numerical algorithms and has been studied extensively on all modern processor and accelerator architectures. However, the optimal sparse matrix data storage format is highly hardware-specific, which could become an obstacle when using heterogeneous systems. Also, it is as yet unclear how the wide single instruction multiple data (SIMD) units in current multi- and many-core processors should be used most efficiently if there is no structure in the sparsity pattern of the matrix. We suggest SELL- @math - @math , a variant of Sliced ELLPACK, as a SIMD-friendly data format which combines long-standing ideas from general-purpose graphics processing units and vector computer programming. We discuss the advantages of SELL- @math - @math compared to established formats like Compressed Row Storage and ELLPACK and show its suitability on a variety of hardware platforms (Intel Sandy Bridge, Intel Xeon Phi, and Nvidia Tesla K20) for a wi...