We show that for the problem of testing if a matrix @math has rank at most @math , or requires changing an @math -fraction of entries to have rank at most @math , there is a non-adaptive query algorithm making @math queries. Our algorithm works for any field @math . This improves upon the previous @math bound (SODA'03), and bypasses an @math lower bound of (KDD'14) which holds if the algorithm is required to read a submatrix. Our algorithm is the first such algorithm which does not read a submatrix, and instead reads a carefully selected non-adaptive pattern of entries in rows and columns of @math . We complement our algorithm with a matching query complexity lower bound for non-adaptive testers over any field. We also give tight bounds of @math queries in the sensing model for which query access comes in the form of @math ; perhaps surprisingly these bounds do not depend on @math . We next develop a novel property testing framework for testing numerical properties of a real-valued matrix @math more generally, which includes the stable rank, Schatten- @math norms, and SVD entropy. Specifically, we propose a bounded entry model, where @math is required to have entries bounded by @math in absolute value. We give upper and lower bounds for a wide range of problems in this model, and discuss connections to the sensing model above.
Other measures of intrinsic dimensionality include matrix norms, such as the Schatten- @math norm @math , which measures the central tendency of the singular values. Familiar special cases are @math , @math and @math , which have applications in differential privacy @cite_9 and non-convex optimization @cite_61 @cite_18 for @math , and in numerical linear algebra @cite_57 for @math . Matrix norms have been studied extensively in the streaming literature @cite_35 @cite_4 @cite_60 @cite_45 , though their study in property testing models is lacking.
@cite_61 @cite_35 @cite_18 @cite_4 @cite_60 @cite_9 @cite_57 @cite_45
2952822330 2952539829 2951669000 2567701579 2950884216 2951863880 2951936557
This work studies the strong duality of non-convex matrix factorization problems: we show that under certain dual conditions, these problems and its dual have the same optimum. This has been well understood for convex optimization, but little was known for non-convex problems. We propose a novel analytical framework and show that under certain dual conditions, the optimal solution of the matrix factorization program is the same as its bi-dual and thus the global optimality of the non-convex program can be achieved by solving its bi-dual which is convex. These dual conditions are satisfied by a wide class of matrix factorization problems, although matrix factorization problems are hard to solve in full generality. This analytical framework may be of independent interest to non-convex optimization more broadly. We apply our framework to two prototypical matrix factorization problems: matrix completion and robust Principal Component Analysis (PCA). These are examples of efficiently recovering a hidden matrix given limited reliable observations of it. Our framework shows that exact recoverability and strong duality hold with nearly-optimal sample complexity guarantees for matrix completion and robust PCA. The subspace approximation problem Subspace( @math , @math ) asks for a @math -dimensional linear subspace that fits a given set of points optimally, where the error for fitting is a generalization of the least squares fit and uses the @math norm instead. Most of the previous work on subspace approximation has focused on small or constant @math and @math , using coresets and sampling techniques from computational geometry. In this paper, extending another line of work based on convex relaxation and rounding, we give a polynomial time algorithm, , with the approximation guarantee roughly @math , where @math is the @math -th moment of a standard normal random variable N(0,1). We show that the convex relaxation we use has an integrality gap (or "rank gap") of @math , for any constant @math . Finally, we show that assuming the Unique Games Conjecture, the subspace approximation problem is hard to approximate within a factor better than @math , for any constant @math . For any real number @math , we nearly completely characterize the space complexity of estimating @math for @math matrices @math in which each row and each column has @math non-zero entries and whose entries are presented one at a time in a data stream model. Here the @math are the singular values of @math , and when @math , @math is the @math -th power of the Schatten @math -norm. We show that when @math is not an even integer, to obtain a @math -approximation to @math with constant probability, any @math -pass algorithm requires @math bits of space, where @math as @math and @math is a constant independent of @math . However, when @math is an even integer, we give an upper bound of @math bits of space, which holds even in the turnstile data stream model. The latter is optimal up to @math factors. Our results considerably strengthen lower bounds in previous work for arbitrary (not necessarily sparse) matrices @math : the previous best lower bound was @math for @math , @math for @math and @math for @math . We note for @math , while our lower bound for even integers is the same, for other @math in this range our lower bound is @math , which is considerably stronger than the previous @math for small enough constant @math . We obtain similar near-linear lower bounds for Ky-Fan norms, SVD entropy, eigenvalue shrinkers, and M-estimators, many of which could have been solvable in logarithmic space prior to our work. We consider the following oblivious sketching problem: given epsilon in (0,1 3) and n >= d epsilon^2, design a distribution D over R^ k * nd and a function f: R^k * R^ nd -> R , so that for any n * d matrix A, Pr_ S sim D [(1-epsilon) |A|_ op = 2 3, where |A|_ op = sup_ x:|x|_2 = 1 |Ax|_2 is the operator norm of A and S(A) denotes S * A, interpreting A as a vector in R^ nd . We show a tight lower bound of k = Omega(d^2 epsilon^2) for this problem. Previously, Nelson and Nguyen (ICALP, 2014) considered the problem of finding a distribution D over R^ k * n such that for any n * d matrix A, Pr_ S sim D [forall x, (1-epsilon)|Ax|_2 = 2 3, which is called an oblivious subspace embedding (OSE). Our result considerably strengthens theirs, as it (1) applies only to estimating the operator norm, which can be estimated given any OSE, and (2) applies to distributions over general linear operators S which treat A as a vector and compute S(A), rather than the restricted class of linear operators corresponding to matrix multiplication. Our technique also implies the first tight bounds for approximating the Schatten p-norm for even integers p via general linear sketches, improving the previous lower bound from k = Omega(n^ 2-6 p ) [Regev, 2014] to k = Omega(n^ 2-4 p ). Importantly, for sketching the operator norm up to a factor of alpha, where alpha - 1 = (1), we obtain a tight k = Omega(n^2 alpha^4) bound, matching the upper bound of Andoni and Nguyen (SODA, 2013), and improving the previous k = Omega(n^2 ^6) lower bound. Finally, we also obtain the first lower bounds for approximating Ky Fan norms. We present new theoretical results on differentially private data release useful with respect to any target class of counting queries, coupled with experimental results on a variety of real world data sets. Specifically, we study a simple combination of the multiplicative weights approach of [Hardt and Rothblum, 2010] with the exponential mechanism of [McSherry and Talwar, 2007]. The multiplicative weights framework allows us to maintain and improve a distribution approximating a given data set with respect to a set of counting queries. We use the exponential mechanism to select those queries most incorrectly tracked by the current distribution. Combing the two, we quickly approach a distribution that agrees with the data set on the given set of queries up to small error. The resulting algorithm and its analysis is simple, but nevertheless improves upon previous work in terms of both error and running time. We also empirically demonstrate the practicality of our approach on several data sets commonly used in the statistical community for contingency table release. Randomized algorithms for very large matrix problems have received a great deal of attention in recent years. Much of this work was motivated by problems in large-scale data analysis, and this work was performed by individuals from many different research communities. This monograph will provide a detailed overview of recent work on the theory of randomized matrix algorithms as well as the application of those ideas to the solution of practical problems in large-scale data analysis. An emphasis will be placed on a few simple core ideas that underlie not only recent theoretical advances but also the usefulness of these tools in large-scale data applications. Crucial in this context is the connection with the concept of statistical leverage. This concept has long been used in statistical regression diagnostics to identify outliers; and it has recently proved crucial in the development of improved worst-case matrix algorithms that are also amenable to high-quality numerical implementation and that are useful to domain scientists. Randomized methods solve problems such as the linear least-squares problem and the low-rank matrix approximation problem by constructing and operating on a randomized sketch of the input matrix. Depending on the specifics of the situation, when compared with the best previously-existing deterministic algorithms, the resulting randomized algorithms have worst-case running time that is asymptotically faster; their numerical implementations are faster in terms of clock-time; or they can be implemented in parallel computing environments where existing numerical algorithms fail to run at all. Numerous examples illustrating these observations will be described in detail. Given an @math matrix @math , its Schatten- @math norm, @math , is defined as @math , where @math is the @math -th largest singular value of @math . These norms have been studied in functional analysis in the context of non-commutative @math -spaces, and recently in data stream and linear sketching models of computation. Basic questions on the relations between these norms, such as their embeddability, are still open. Specifically, given a set of matrices @math , suppose we want to construct a linear map @math such that @math for each @math , where @math and @math , and further, @math for a given approximation factor @math and real number @math . Then how large do @math and @math need to be as a function of @math ? We nearly resolve this question for every @math , for the case where @math can be expressed as @math , where @math and @math are arbitrary matrices that are allowed to depend on @math , that is, @math can be implemented by left and right matrix multiplication. Namely, for every @math , we provide nearly matching upper and lower bounds on the size of @math and @math as a function of @math . Importantly, our upper bounds are oblivious , meaning that @math and @math do not depend on the @math , while our lower bounds hold even if @math and @math depend on the @math . As an application of our upper bounds, we answer a recent open question of about space-approximation trade-offs for the Schatten @math -norm, showing in a data stream it is possible to estimate the Schatten- @math norm up to a factor of @math using @math space.