Low-rank approximation of a matrix by means of random sampling has been consistently efficient in its empirical studies by many scientists who applied it with various sparse and structured multipliers, but adequate formal support for this empirical phenomenon has been missing so far. Our novel insight into the subject leads to such an elusive formal support and promises significant acceleration of the known algorithms for some fundamental problems of matrix computations and data mining and analysis. Our formal results and our numerical tests are in good accordance with each other. We also outline extensions of low-rank approximation algorithms and of our progress to the Least Squares Regression, the Fast Multipole Method, and the Conjugate Gradient algorithms.
Part (ii) of our Theorem is implied by [Theorem 10.8] HMT11 , but our specific supporting estimates are more compact, cover the case of any @math (whereas @cite_36 assumes that @math ), and we deduce them by using a shorter proof (see Remark ). Our approach, our results listed in Section , some of our techniques, e.g., expan -sion compres -sion in Section , and even the concept of factor-Gaussian matrices are new, and so are our families of multipliers and policies of their generation, combination, and application in Sections and as well. Moreover, our progress can be extended to a variety of important matrix computations. In Section (Conclusions) we outline such novel extensions to the highly popular and much studied computations for Least Squares Regression, the Fast Multipole Method and the Conjugate Gradient Algorithms. Hereafter we use the acronyms LSR" for Least Squares Regression", FMM" for Fast Multipole Method", and CG " for Conjugate Gradient". The extensions provide new insights and new opportunities and should motivate further effort and further progress.
@cite_36
2117756735
Section 1 of the paper contains a general discussion of robustness. In Section 2 the influence function of the Hampel-Rousseeuw least median of squares estimator is derived. Linearly invariant weak metrics are constructed in Section 3. It is shown in Section 4 that @math -estimators satisfy an exact Holder condition of order 1 2 at models with normal errors. In Section 5 the breakdown points of the Hampel-Krasker dispersion and regression functionals are shown to be 0. The exact breakdown point of the Krasker-Welsch dispersion functional is obtained as well as bounds for the corresponding regression functional. Section 6 contains the construction of a linearly equivariant, high breakdown and locally Lipschitz dispersion functional for any design distribution. In Section 7 it is shown that there is no inherent contradiction between efficiency and a high breakdown point. Section 8 contains a linearly equivariant, high breakdown regression functional which is Lipschitz continuous at models with normal errors. Randomized sampling has recently been proven a highly efficient technique for computing approximate factorizations of matrices that have low numerical rank. This paper describes an extension of such techniques to a wider class of matrices that are not themselves rank-deficient but have off-diagonal blocks that are; specifically, the class of so-called hierarchically semiseparable (HSS) matrices. HSS matrices arise frequently in numerical analysis and signal processing, particularly in the construction of fast methods for solving differential and integral equations numerically. The HSS structure admits algebraic operations (matrix-vector multiplications, matrix factorizations, matrix inversion, etc.) to be performed very rapidly, but only once the HSS representation of the matrix has been constructed. How to rapidly compute this representation in the first place is much less well understood. The present paper demonstrates that if an @math matrix can be applied to a vector in @math time, and if individual entries of the matrix can be computed rapidly, then provided that an HSS representation of the matrix exists, it can be constructed in @math operations, where @math is an upper bound for the numerical rank of the off-diagonal blocks. The point is that when legacy codes (based on, e.g., the fast multipole method) can be used for the fast matrix-vector multiply, the proposed algorithm can be used to obtain the HSS representation of the matrix, and then well-established techniques for HSS matrices can be used to invert or factor the matrix. The fast multipole method (FMM) has been implemented to speed up the matrix-vector multiply when an iterative method is used to solve the combined field integral equation (CFIE). FMM reduces the complexity from O(N2) to O(N1.5). With a multilevel fast multipole algorithm (MLFMA), it is further reduced to O(N log N). A 110, 592-unknown problem can be solved within 24 h on a SUN Sparc 10. © 1995 John Wiley & Sons, Inc. In some previous papers the author extended two algorithms proposed by Z. Kovarik for approximate orthogonalization of a finite set of linearly independent vectors from a Hilbert space, to the case when the vectors are rows (not necessary linearly independent) of an arbitrary rectangular matrix. In this paper we describe combinations between these two methods and the classical Kaczmarz’s iteration. We prove that, in the case of a consistent least-squares problem, the new algorithms so obtained converge to any of its solutions (depending on the initial approximation). The numerical experiments described in the last section of the paper on a problem obtained after the discretization of a first kind integral equation ilustrate the fast convergence of the new algorithms. Homological Projective duality (HP-duality) theory, introduced by Kuznetsov [42], is one of the most powerful frameworks in the homological study of algebraic geometry. The main result (HP-duality theorem) of the theory gives complete descriptions of bounded derived categories of coherent sheaves of (dual) linear sections of HP-dual varieties. We show the theorem also holds for more general intersections beyond linear sections. More explicitly, for a given HP-dual pair @math , then analogue of HP-duality theorem holds for their intersections with another HP-dual pair @math , provided that they intersect properly. We also prove a relative version of our main result. Taking @math to be dual linear subspaces (resp. subbundles), our method provides a more direct proof of the original (relative) HP-duality theorem. A new bound (Theorem ) for the duration of the chip-firing game with @math chips on a @math -vertex graph is obtained, by a careful analysis of the pseudo-inverse of the discrete Laplacian matrix of the graph. This new bound is expressed in terms of the entries of the pseudo-inverse. It is shown (Section 5) to be always better than the classic bound due to Bj "o rner, Lov ' a sz and Shor. In some cases the improvement is dramatic. For instance: for strongly regular graphs the classic and the new bounds reduce to @math and @math , respectively. For dense regular graphs - @math - the classic and the new bounds reduce to @math and @math , respectively. This is a snapshot of a work in progress, so further results in this vein are in the works. List of Algorithms Preface 1. Introduction. Brief Overview of the State of the Art Notation Review of Relevant Linear Algebra Part I. Krylov Subspace Approximations. 2. Some Iteration Methods. Simple Iteration Orthomin(1) and Steepest Descent Orthomin(2) and CG Orthodir, MINRES, and GMRES Derivation of MINRES and CG from the Lanczos Algorithm 3. Error Bounds for CG, MINRES, and GMRES. Hermitian Problems-CG and MINRES Non-Hermitian Problems-GMRES 4. Effects of Finite Precision Arithmetic. Some Numerical Examples The Lanczos Algorithm A Hypothetical MINRES CG Implementation A Matrix Completion Problem Orthogonal Polynomials 5. BiCG and Related Methods. The Two-Sided Lanczos Algorithm The Biconjugate Gradient Algorithm The Quasi-Minimal Residual Algorithm Relation Between BiCG and QMR The Conjugate Gradient Squared Algorithm The BiCGSTAB Algorithm Which Method Should I Use? 6. Is There A Short Recurrence for a Near-Optimal Approximation? The Faber and Manteuffel Result Implications 7. Miscellaneous Issues. Symmetrizing the Problem Error Estimation and Stopping Criteria Attainable Accuracy Multiple Right-Hand Sides and Block Methods Computer Implementation Part II. Preconditioners. 8. Overview and Preconditioned Algorithms. 9. Two Example Problems. The Diffusion Equation The Transport Equation 10. Comparison of Preconditioners. Jacobi, Gauss--Seidel, SOR The Perron--Frobenius Theorem Comparison of Regular Splittings Regular Splittings Used with the CG Algorithm Optimal Diagonal and Block Diagonal Preconditioners 11. Incomplete Decompositions. Incomplete Cholesky Decomposition Modified Incomplete Cholesky Decomposition 12. Multigrid and Domain Decomposition Methods. Multigrid Methods Basic Ideas of Domain Decomposition Methods. Matrix functions are used in many areas of linear algebra and arise in numerous applications in science and engineering. The most common matrix function is the matrix inverse; it is not treated specifically in this chapter, but is covered in Section 1.5 and Section 51.3. This chapter is concerned with general matrix functions as well as specific cases such as matrix square roots, trigonometric functions, and the exponential and logarithmic functions. In this article, we combine complex-analytic and arithmetic tools to study the preperiodic points of one-dimensional complex dynamical systems. We show that for any fixed complex numbers a and b, and any integer d at least 2, the set of complex numbers c for which both a and b are preperiodic for z^d+c is infinite if and only if a^d = b^d. This provides an affirmative answer to a question of Zannier, which itself arose from questions of Masser concerning simultaneous torsion sections on families of elliptic curves. Using similar techniques, we prove that if two complex rational functions f and g have infinitely many preperiodic points in common, then they must have the same Julia set. This generalizes a theorem of Mimar, who established the same result assuming that f and g are defined over an algebraic extension of the rationals. The main arithmetic ingredient in the proofs is an adelic equidistribution theorem for preperiodic points over number fields and function fields, with non-archimedean Berkovich spaces playing an essential role. A combination of hierarchical tree-like data structures and data access patterns from fast multipole methods and hierarchical low-rank approximation of linear operators from H-matrix methods appears to form an algorithmic path forward for efficient implementation of many linear algebraic operations of scientific computing at the exascale. The combination provides asymptotically optimal computational and communication complexity and applicability to large classes of operators that commonly arise in scientific computing applications. A convergence of the mathematical theories of the fast multipole and H-matrix methods has been underway for over a decade. We recap this mathematical unification and describe implementation aspects of a hybrid of these two compelling hierarchical algorithms on hierarchical distributed-shared memory architectures, which are likely to be the first to reach the exascale. We present a new communication complexity estimate for fast multipole methods on such architectures. We also show how the data structures and access patterns of H-matrices for low-rank operators map onto those of fast multipole, leading to an algebraically generalized form of fast multipole that compromises none of its architecturally ideal properties. We propose a novel and fast approach to discover structure in western popular music by using a specific type of matrix factorization that adds a convex constrain to obtain a decomposition that can be interpreted as a set of weighted cluster centroids. We show that these centroids capture the different sections of a musical piece (e.g. verse, chorus) in a more consistent and efficient way than classic non-negative matrix factorization. This technique is capable of identifying the boundaries of the sections and then grouping them into different clusters. Additionally, we evaluate this method on two different datasets and show that it is competitive compared to other music segmentation techniques, outperforming other matrix factorization methods. In this paper, I present a simple characterization of the sample selection bias problem that is also applicable to the conceptually distinct econometric problems that arise from truncated samples and from models with limited dependent variables. The problem of sample selection bias is fit within the conventional specification error framework of Griliches and Theil. A simple estimator is discussed that enables analysts to utilize ordinary regression methods to estimate models free of selection bias. The techniques discussed here are applied to re-estimate and test a model of female labor supply developed by the author. (1974). This paper is in three parts. In the first section, selection bias is presented within the specification error framework. In this section, general distributional assumptions are maintained. In section two, specific results are presented for the case of normal regression disturbances. Simple estimators are proposed and discussed. In the third section, empirical results are presented. There is widespread sentiment that it is not possible to effectively utilize fast gradient methods (e.g. Nesterov's acceleration, conjugate gradient, heavy ball) for the purposes of stochastic optimization due to their instability and error accumulation, a notion made precise in d'Aspremont 2008 and Devolder, Glineur, and Nesterov 2014. This work considers these issues for the special case of stochastic approximation for the least squares regression problem, and our main result refutes the conventional wisdom by showing that acceleration can be made robust to statistical errors. In particular, this work introduces an accelerated stochastic gradient method that provably achieves the minimax optimal statistical risk faster than stochastic gradient descent. Critical to the analysis is a sharp characterization of accelerated stochastic gradient descent as a stochastic process. We hope this characterization gives insights towards the broader question of designing simple and effective accelerated stochastic methods for more general convex and non-convex optimization problems. This paper presents a fundamental law for parallel performance: it shows that parallel performance is not only limited by sequential code (as suggested by Amdahl's law) but is also fundamentally limited by synchronization through critical sections. Extending Amdahl's software model to include critical sections, we derive the surprising result that the impact of critical sections on parallel performance can be modeled as a completely sequential part and a completely parallel part. The sequential part is determined by the probability for entering a critical section and the contention probability (i.e., multiple threads wanting to enter the same critical section). This fundamental result reveals at least three important insights for multicore design. (i) Asymmetric multicore processors deliver less performance benefits relative to symmetric processors than suggested by Amdahl's law, and in some cases even worse performance. (ii) Amdahl's law suggests many tiny cores for optimum performance in asymmetric processors, however, we find that fewer but larger small cores can yield substantially better performance. (iii) Executing critical sections on the big core can yield substantial speedups, however, performance is sensitive to the accuracy of the critical section contention predictor. The paper uses empirical process techniques to study the asymptotics of the least-squares estimator (LSE) for the fitting of a nonlinear regression function. By combining and extending ideas of Wu and Van de Geer, it establishes new consistency and central limit theorems that hold under only second moment assumptions on the errors. An application to a delicate example of Wu's illustrates the use of the new theorems, leading to a normal approximation to the LSE with unusual logarithmic rescalings. There exist many classes of block-projections algorithms for approximating solutions of linear least-squares problems. Generally, these methods generate sequences convergent to the minimal norm least-squares solution only for consistent problems. In the inconsistent case, which usually appears in practice because of some approximations or measurements, these sequences do no longer converge to a least-squares solution or they converge to the minimal norm solution of a “perturbed” problem. In the present paper, we overcome this difficulty by constructing extensions for almost all the above classes of block-projections methods. We prove that the sequences generated with these extensions always converge to a least-squares solution and, with a suitable initial approximation, to the minimal norm solution of the problem. Numerical experiments, described in the last section of the paper, confirm the theoretical results obtained. Hardware and software systems will inevitably grow in scale and functionality. Because of this increase in complexity, the likelihood of subtle errors is much greater. Moreover, some of these errors may cause catastrophic loss of money, time, or even human life. A major goal of software engineering is to enable developers to construct systems that operate reliably despite this complexity. One way of achieving this goal is by using formal methods, which are mathematically based languages, techniques, and tools for specifying and verifying such systems. Use of formal methods does not a priori guarantee correctness. However, they can greatly increase our understanding of a system by revealing inconsistencies, ambiguities, and incompleteness that might otherwise go undetected. The first part of this report assesses the state of the art in specification and verification. For verification, we highlight advances in model checking and theorem proving. In the three sections on specification, model checking, and theorem proving, we explain what we mean by the general technique and briefly describe some successful case studies and well-known tools. The second part of this report outlines future directions in fundamental concepts, new methods and tools, integration of methods, and education and technology transfer. We close with summary remarks and pointers to resources for more information. In a recent article, El (Proc Natl Acad Sci 110(36):14557–14562, 2013) study the distribution of robust regression estimators in the regime in which the number of parameters p is of the same order as the number of samples n. Using numerical simulations and ‘highly plausible’ heuristic arguments, they unveil a striking new phenomenon. Namely, the regression coefficients contain an extra Gaussian noise component that is not explained by classical concepts such as the Fisher information matrix. We show here that that this phenomenon can be characterized rigorously using techniques that were developed by the authors for analyzing the Lasso estimator under high-dimensional asymptotics. We introduce an approximate message passing (AMP) algorithm to compute M-estimators and deploy state evolution to evaluate the operating characteristics of AMP and so also M-estimates. Our analysis clarifies that the ‘extra Gaussian noise’ encountered in this problem is fundamentally similar to phenomena already studied for regularized least squares in the setting (n<p ). Preface Acknowledgements 1. Direct solution methods 2. Theory of matrix eigenvalues 3. Positive definite matrices, Schur complements, and generalized eigenvalue problems 4. Reducible and irreducible matrices and the Perron-Frobenius theory for nonnegative matrices 5. Basic iterative methods and their rates of convergence 6. M-matrices, convergent splittings, and the SOR method 7. Incomplete factorization preconditioning methods 8. Approximate matrix inverses and corresponding preconditioning methods 9. Block diagonal and Schur complement preconditionings 10. Estimates of eigenvalues and condition numbers for preconditional matrices 11. Conjugate gradient and Lanczos-type methods 12. Generalized conjugate gradient methods 13. The rate of convergence of the conjugate gradient method Appendices. Abstract There is some controversy regarding the scaling of the fast multipole method (FMM). It has recently been proven by Aluru that the FMM is not a linear scaling method, but an O ( N log 4 N ) method. Aluru's proof cannot be applied to typical computational chemistry calculations where the required precision is smaller than the machine accuracy. In this Letter, we deal with this kind of situation and give a rigorous bound to the scaling and a statistical estimate. We also perform numerical tests. Our results agree with Aluru's proof. The scaling of other methods that use multipoles is also discussed.