The increasing number of applications requiring the solution of large scale singular value problems have rekindled interest in iterative methods for the SVD. Some promising recent ad- vances in large scale iterative methods are still plagued by slow convergence and accuracy limitations for computing smallest singular triplets. Furthermore, their current implementations in MATLAB cannot address the required large problems. Recently, we presented a preconditioned, two-stage method to effectively and accurately compute a small number of extreme singular triplets. In this research, we present a high-performance software, PRIMME SVDS, that implements our hybrid method based on the state-of-the-art eigensolver package PRIMME for both largest and smallest singular values. PRIMME SVDS fills a gap in production level software for computing the partial SVD, especially with preconditioning. The numerical experiments demonstrate its superior performance compared to other state-of-the-art software and its good parallel performance under strong and weak scaling.
Despite accuracy limitations, eigenvalue iterative methods based on @math are widely used for computing the largest eigenpairs where the loss of accuracy is limited (see )) and even for low accuracy computations of the smallest singular values. For example, two popular packages in machine learning, scikit-learn http: scikit-learn.org stable modules generated sklearn.decomposition.TruncatedSVD.html and Spark's library MLib https: spark.apache.org docs 1.2.1 mllib-dimensionality-reduction.html , use a wrapper for the popular package ARPACK (implicit restarting Arnoldi method) @cite_13 . Other solvers for standard Hermitian eigenvalue problems can also be used. Table lists the most widely used eigensolver libraries with high-performance computing implementations.
@cite_13
1747539222 2246353276 1973379861 2119759375 2197112176 2964122166 2097897435 2142621404 2574952845 2964047892 1998906317 2083676493 1506690472 2021710646 2170546552 1982666353 1803217602 2094887027 2099119387 2014513111
The computation of a few singular triplets of large, sparse matrices is a challenging task, especially when the smallest magnitude singular values are needed in high accuracy. Most recent efforts try to address this problem through variations of the Lanczos bidiagonalization method, but they are still challenged even for medium matrix sizes due to the difficulty of the problem. We propose a novel SVD approach that can take advantage of preconditioning and of any well-designed eigensolver to compute both largest and smallest singular triplets. Accuracy and efficiency is achieved through a hybrid, two-stage meta-method, PHSVDS. In the first stage, PHSVDS solves the normal equations up to the best achievable accuracy. If further accuracy is required, the method switches automatically to an eigenvalue problem with the augmented matrix. Thus it combines the advantages of the two stages, faster convergence and accuracy, respectively. For the augmented matrix, solving the interior eigenvalue is facilitated by pr... An efficient power flow solution is highly important for power system analyses. Traditionally, LU decomposition-based direct method is dominant in solving linear systems in the power flow analysis. However, with the increasing scale and complexity, direct methods may suffer scalability issues, especially under parallel implementation. Therefore, iterative methods such as Conjugate Gradient (CG) are gaining more attention for their scalability and feasibility for parallelisation. To efficiently apply an iterative solver like CG, a preconditioner is usually required. A polynomial-based Chebyshev preconditioner is discussed in this work. An essential parameter in Chebyshev preconditioning is the maximum eigenvalue of the linear system. However, direct calculation of the eigenvalues is too time-consuming to be practical. Therefore, this work proposes a method to estimate the largest eigenvalue to save the time spent on eigenvalue calculation. Thus, Chebyshev preconditioning will be practical to use in power system analyses. This work first proves that the eigenvalues of power system applications range within (0, 2] after a normalisation step, then demonstrates the eigenvalue estimation accuracy, and finally presents the performance improvement. The test shows an average 98.92 runtime saving for the Chebyshev preconditioner, and a 40.99 runtime saving for precondition and iterative solving process. We present a comparative study of the application of modern eigenvalue algorithms to an eigenvalue problem arising in quantum physics, namely, the computation of a few interior eigenvalues and their associated eigenvectors for the large, sparse, real, symmetric, and indefinite matrices of the Anderson model of localization. We compare the Lanczos algorithm in the 1987 implementation of Cullum and Willoughby with the implicitly restarted Arnoldi method coupled with polynomial and several shift-and-invert convergence accelerators as well as with a sparse hybrid tridiagonalization method. We demonstrate that for our problem the Lanczos implementation is faster and more memory efficient than the other approaches. This seemingly innocuous problem presents a major challenge for all modern eigenvalue algorithms. This paper concerns accurate computation of the singular value decomposition (SVD) of an (m n ) matrix (A ). As is well known, cross-product matrix based SVD algorithms compute large singular values accurately but generally deliver poor small singular values. A new novel cross-product matrix based SVD method is proposed: (a) Use a backward stable algorithm to compute the eigenpairs of (A^ T A ) and take the square roots of the large eigenvalues of it as the large singular values of (A ) ; (b) form the Rayleigh quotient of (A^ T A ) with respect to the matrix consisting of the computed eigenvectors associated with the computed small eigenvalues of (A^ T A ) ; (c) compute the eigenvalues of the Rayleigh quotient and take the square roots of them as the small singular values of (A ). A detailed quantitative error analysis is conducted on the method. It is proved that if small singular values are well separated from the large ones then the method can compute the small ones accurately up to the order of the unit roundoff ( ). An algorithm is developed that is not only cheaper than the standard Golub–Reinsch and Chan SVD algorithms but also can update or downdate a new SVD by adding or deleting a row and compute certain refined Ritz vectors for large matrix eigenproblems at very low cost. Several variants of the algorithm are proposed that compute some or all parts of the SVD. Typical numerical examples confirm the high accuracy of our algorithm. The smallest eigenvalues and the associated eigenvectors (i.e., eigenpairs) of a graph Laplacian matrix have been widely used for spectral clustering and community detection. However, in real-life applications the number of clusters or communities (say, @math ) is generally unknown a-priori. Consequently, the majority of the existing methods either choose @math heuristically or they repeat the clustering method with different choices of @math and accept the best clustering result. The first option, more often, yields suboptimal result, while the second option is computationally expensive. In this work, we propose an incremental method for constructing the eigenspectrum of the graph Laplacian matrix. This method leverages the eigenstructure of graph Laplacian matrix to obtain the @math -th eigenpairs of the Laplacian matrix given a collection of all the @math smallest eigenpairs. Our proposed method adapts the Laplacian matrix such that the batch eigenvalue decomposition problem transforms into an efficient sequential leading eigenpair computation problem. As a practical application, we consider user-guided spectral clustering. Specifically, we demonstrate that users can utilize the proposed incremental method for effective eigenpair computation and determining the desired number of clusters based on multiple clustering metrics. The smallest eigenvalues and the associated eigenvectors (i.e., eigenpairs) of a graph Laplacian matrix have been widely used for spectral clustering and community detection. However, in real-life applications, the number of clusters or communities (say, K) is generally unknown a priori. Consequently, the majority of the existing methods either choose K heuristically or they repeat the clustering method with different choices of K and accept the best clustering result. The first option, more often, yields suboptimal result, while the second option is computationally expensive. In this work, we propose an incremental method for constructing the eigenspectrum of the graph Laplacian matrix. This method leverages the eigenstructure of graph Laplacian matrix to obtain the Kth smallest eigenpair of the Laplacian matrix given a collection of all previously computed (K-1 ) smallest eigenpairs. Our proposed method adapts the Laplacian matrix such that the batch eigenvalue decomposition problem transforms into an efficient sequential leading eigenpair computation problem. As a practical application, we consider user-guided spectral clustering. Specifically, we demonstrate that users can utilize the proposed incremental method for effective eigenpair computation and for determining the desired number of clusters based on multiple clustering metrics. An iterative method is given for solving Ax ffi b and minU Ax b 112, where the matrix A is large and sparse. The method is based on the bidiagonalization procedure of Golub and Kahan. It is analytically equivalent to the standard method of conjugate gradients, but possesses more favorable numerical properties. Reliable stopping criteria are derived, along with estimates of standard errors for x and the condition number of A. These are used in the FORTRAN implementation of the method, subroutine LSQR. Numerical tests are described comparing I QR with several other conjugate-gradient algorithms, indicating that I QR is the most reliable algorithm when A is ill-conditioned. In this paper, two supervised methods for enhancing the classification accuracy of the Nonnegative Matrix Factorization (NMF) algorithm are presented. The idea is to extend the NMF algorithm in order to extract features that enforce not only the spatial locality, but also the separability between classes in a discriminant manner. The first method employs discriminant analysis in the features derived from NMF. In this way, a two-phase discriminant feature extraction procedure is implemented, namely NMF plus Linear Discriminant Analysis (LDA). The second method incorporates the discriminant constraints inside the NMF decomposition. Thus, a decomposition of a face to its discriminant parts is obtained and new update rules for both the weights and the basis images are derived. The introduced methods have been applied to the problem of frontal face verification using the well-known XM2VTS database. Both methods greatly enhance the performance of NMF for frontal face verification In this paper, we propose a novel deep convolutional neural network (CNN)-based algorithm for solving ill-posed inverse problems. Regularized iterative algorithms have emerged as the standard approach to ill-posed inverse problems in the past few decades. These methods produce excellent results, but can be challenging to deploy in practice due to factors including the high computational cost of the forward and adjoint operators and the difficulty of hyperparameter selection. The starting point of this paper is the observation that unrolled iterative methods have the form of a CNN (filtering followed by pointwise non-linearity) when the normal operator ( @math , where @math is the adjoint of the forward imaging operator, @math ) of the forward model is a convolution. Based on this observation, we propose using direct inversion followed by a CNN to solve normal-convolutional inverse problems. The direct inversion encapsulates the physical model of the system, but leads to artifacts when the problem is ill posed; the CNN combines multiresolution decomposition and residual learning in order to learn to remove these artifacts while preserving image structure. We demonstrate the performance of the proposed network in sparse-view reconstruction (down to 50 views) on parallel beam X-ray computed tomography in synthetic phantoms as well as in real experimental sinograms. The proposed network outperforms total variation-regularized iterative reconstruction for the more realistic phantoms and requires less than a second to reconstruct a @math image on the GPU. The standard linear regression (SLR) problem is to recover a vector x0 from noisy linear observations y = Ax0 + w. The approximate message passing (AMP) algorithm recently proposed by Donoho, Maleki, and Montanari is a computationally efficient iterative approach to SLR that has a remarkable property: for large i.i.d. sub-Gaussian matrices A, its periteration behavior is rigorously characterized by a scalar stateevolution whose fixed points, when unique, are Bayes optimal. AMP, however, is fragile in that even small deviations from the i.i.d. sub-Gaussian model can cause the algorithm to diverge. This paper considers a “vector AMP” (VAMP) algorithm and shows that VAMP has a rigorous scalar state-evolution that holds under a much broader class of large random matrices A: those that are right-rotationally invariant. After performing an initial singular value decomposition (SVD) of A, the per-iteration complexity of VAMP is similar to that of AMP. In addition, the fixed points of VAMP's state evolution are consistent with the replica prediction of the minimum mean-squared error recently derived by Tulino, Caire, Verdu, and Shamai. A new numerical algorithm for solving the symmetric eigenvalue problem is presented. The technique deviates fundamentally from the traditional Krylov subspace iteration based techniques (Arnoldi and Lanczos algorithms) or other Davidson-Jacobi techniques, and takes its inspiration from the contour integration and density matrix representation in quantum mechanics. It will be shown that this new algorithm - named FEAST - exhibits high efficiency, robustness, accuracy and scalability on parallel architectures. Examples from electronic structure calculations of Carbon nanotubes (CNT) are presented, and numerical performances and capabilities are discussed. Abstract : The present paper deals with the problem of computing a few of the eigenvalues with largest (or smallest) real parts, of a large sparse nonsymmetric matrix. We present a general acceleration technique based on Chebyshev polynomials and discuss its practical application to Arnoldi's method and the subspace iteration method. The resulting algorithms are compared with the classical ones in a few experiments which exhibit a sharp superiority of the Arnoldi-Chebyshev approach. List of figures List of tables Preface 1. Introduction to ARPACK. Important features Getting started Reverse communication interface Availability Installation Documentation Dependence on LAPACK and BLAS Expected performance P_ARPACK Contributed additions Trouble shooting and problems 2. Getting started with ARPACK. Directory structure and contents Getting started An example for a symmetric Eigenvalue problem 3. General use of ARPACK. Naming conventions, Precisions, and types Shift and invert spectral transformation mode Reverse communication structure for shift-Invert Using the computational modes Computational modes for real symmetric problems Postprocessing for Eigenvectors using dseupd Computational modes for real nonsymmetric problems Postprocessing for Eigenvectors Using dneupd Computational modes for complex problems Postprocessing for Eigenvectors Using zneupd 4. The implicitly restarted Arnoldi method: structure of the Eigenvalue problem Krylov subspaces and projection methods The Arnoldi factorization Restarting the Arnoldi method The generalized Eigenvalue problem Stopping Criterion 5. Computational routines. ARPACK subroutines LAPACK routines used by ARPACK BLAS routines used by ARPACK Appendix A. Templates and driver routines Symmetric drivers Real Nonsymmetric drivers Complex drivers Band drivers The singular value decomposition Appendix B. Tracking the progress of ARPACK. Obtaining trace output Check-pointing ARPACK Appendix C. The XYaupd ARPACK Routines. DSAUPD DNAUPD ZNAUPD Bibliography Index. We discuss a new method for the iterative computation of a portion of the singular values and vectors of a large sparse matrix. Similar to the Jacobi--Davidson method for the eigenvalue problem, we compute in each step a correction by (approximately) solving a correction equation. We give a few variants of this Jacobi--Davidson SVD (JDSVD) method with their theoretical properties. It is shown that the JDSVD can be seen as an accelerated (inexact) Newton scheme. We experimentally compare the method with some other iterative SVD methods. Mulmuley [Mul12a] recently gave an explicit version of Noether’s Normalization lemma for ring of invariants of matrices under simultaneous conjugation, under the conjecture that there are deterministic black-box algorithms for polynomial identity testing (PIT). He argued that this gives evidence that constructing such algorithms for PIT is beyond current techniques. In this work, we show this is not the case. That is, we improve Mulmuley’s reduction and correspondingly weaken the conjecture regarding PIT needed to give explicit Noether Normalization. We then observe that the weaker conjecture has recently been nearly settled by the authors ([FS12]), who gave quasipolynomial size hitting sets for the class of read-once oblivious algebraic branching programs (ROABPs). This gives the desired explicit Noether Normalization unconditionally, up to quasipolynomial factors. As a consequence of our proof we give a deterministic parallel polynomial-time algorithm for deciding if two matrix tuples have intersecting orbit closures, under simultaneous conjugation. We also study the strength of conjectures that Mulmuley requires to obtain similar results as ours. We prove that his conjectures are stronger, in the sense that the computational model he needs PIT algorithms for is equivalent to the well-known algebraic branching program (ABP) model, which is provably stronger than the ROABP model. Finally, we consider the depth-3 diagonal circuit model as defined by Saxena [Sax08], as PIT algorithms for this model also have implications in Mulmuley’s work. Previous work (such as [ASS12] and [FS12]) have given quasipolynomial size hitting sets for this model. In this work, we give a much simpler construction of such hitting sets, using techniques of Shpilka and Volkovich [SV09]. Ng-Jordan-Weiss (NJW) method is one of the most widely used spectral clustering algorithms. For a K clustering problem, this method partitions data using the largest K eigenvectors of the normalized affinity matrix derived from the dataset. It has been demonstrated that the spectral relaxation solution of K-way grouping is located on the subspace of the largest K eigenvectors. However, we find from a lot of experiments that the top K eigenvectors cannot always detect the structure of the data for real pattern recognition problems. So it is necessary to select eigenvectors for spectral clustering. We propose an eigenvector selection method based on entropy ranking for spectral clustering (ESBER). In this method, first all the eigenvectors are ranked according to their importance on clustering, and then a suitable eigenvector combination is obtained from the ranking list. In this paper, we propose two strategies to select eigenvectors in the ranking list of eigenvectors. One is directly adopting the first K eigenvectors in the ranking list. Different to the largest K eigenvectors of NJW method, these K eigenvectors are the most important eigenvectors among all the eigenvectors. The other eigenvector selection strategy is to search a suitable eigenvector combination among the first Km (Km>K) eigenvectors in the ranking list. The eigenvector combination obtained by this strategy can reflect the structure of the original data and lead to a satisfying spectral clustering result. Furthermore, we also present computational complexity reduction strategies for ESBER method to deal with large-scale datasets. We have performed experiments on UCI benchmark datasets, MNIST handwritten digits datasets, and Brodatz texture datasets, adopting NJW method for a baseline comparison. The experimental results show that ESBER method is more robust than NJW method. Especially, ESBER method with the latter eigenvector selection strategy can obtain satisfying clustering results in most cases. Calculating portions of eigenvalues and eigenvectors of matrices or matrix pencils has many applications. An approach to this calculation for Hermitian problems based on a density matrix has been proposed in 2009 and a software package called FEAST has been developed. The density-matrix approach allows FEAST's implementation to exploit a key strength of modern computer architectures, namely, multiple levels of parallelism. Consequently, the software package has been well received and subsequently commercialized. A detailed theoretical analysis of Hermitian FEAST has also been established very recently. This paper generalizes the FEAST algorithm and theory, for the first time, to tackle non-Hermitian problems. Fundamentally, the new algorithm is basic subspace iteration or Bauer bi-iteration, except applied with a novel accelerator based on Cauchy integrals. The resulting algorithm retains the multi-level parallelism of Hermitian FEAST, making it a valuable new tool for large-scale computational science and engineering problems on leading-edge computing platforms. We propose a wavelet multiscale decomposition-based autoregressive approach for the prediction of 1-h ahead load based on historical electricity load data. This approach is based on a multiple resolution decomposition of the signal using the non-decimated or redundant Haar a trous wavelet transform whose advantage is taking into account the asymmetric nature of the time-varying data. There is an additional computational advantage in that there is no need to recompute the wavelet transform (wavelet coefficients) of the full signal if the electricity data (time series) is regularly updated. We assess results produced by this multiscale autoregressive (MAR) method, in both linear and non-linear variants, with single resolution autoregression (AR), multilayer perceptron (MLP), Elman recurrent neural network (ERN) and the general regression neural network (GRNN) models. Results are based on the New South Wales (Australia) electricity load data that is provided by the National Electricity Market Management Company (NEMMCO). We study the performance of a family of randomized parallel coordinate descent methods for minimizing the sum of a nonsmooth and separable convex functions. The problem class includes as a special case L1-regularized L1 regression and the minimization of the exponential loss ("AdaBoost problem"). We assume the input data defining the loss function is contained in a sparse @math matrix @math with at most @math nonzeros in each row. Our methods need @math iterations to find an approximate solution with high probability, where @math is the number of processors and @math for the fastest variant. The notation hides dependence on quantities such as the required accuracy and confidence levels and the distance of the starting iterate from an optimal point. Since @math is a decreasing function of @math , the method needs fewer iterations when more processors are used. Certain variants of our algorithms perform on average only @math arithmetic operations during a single iteration per processor and, because @math decreases when @math does, fewer iterations are needed for sparser problems. Abstract Let the linear system Ax = b be rectangular but solvable. If A is a large sparse matrix, then one possibility to solve the system is to use the iterative method of Kaczmarz. Even in the case when the system is unsolvable, this method is applicable if the relaxation parameters are kept fixed in a certain interval. To get a meaningful result, however, one has to apply strong underrelaxation. We give an interpretation of the limit in terms of a generalized inverse of A and derive bounds with respect to the least-squares solution. Using a result of Buoni and Varga, we estimate the speed of convergence. It is further shown that the eigenvalues of the iteration matrix are nearly real numbers, and we propose to use Chebyshev acceleration. This results in a significant speedup.