Given a subset K of the unit Euclidean sphere, we estimate the minimal number m = m(K) of hyperplanes that generate a uniform tessellation of K, in the sense that the fraction of the hyperplanes separating any pair x, y in K is nearly proportional to the Euclidean distance between x and y. Random hyperplanes prove to be almost ideal for this problem; they achieve the almost optimal bound m = O(w(K)^2) where w(K) is the Gaussian mean width of K. Using the map that sends x in K to the sign vector with respect to the hyperplanes, we conclude that every bounded subset K of R^n embeds into the Hamming cube -1, 1 ^m with a small distortion in the Gromov-Haussdorf metric. Since for many sets K one has m = m(K) << n, this yields a new discrete mechanism of dimension reduction for sets in Euclidean spaces.
Embeddings of subsets @math into normed spaces were studied in geometric functional analysis @cite_10 @cite_14 . In particular, Klartag and Mendelson @cite_10 were concerned with embeddings into @math . They showed that for @math there exists a linear map @math such that @math One can choose @math to be an @math random matrix with Gaussian entries as in Theorem , or with sub-gaussian entries. Schechtman @cite_15 gave a simpler argument for a Gaussian matrix, which also works for embeddings into general normed spaces @math . In the specific case of @math , Schechtman's result states that for @math one has @math This result also follows from Lemma below.
@cite_15 @cite_14 @cite_10
2070944235 2113385666
This paper is concerned with the asymptotic empirical eigenvalue distribution of a non linear random matrix ensemble. More precisely we consider @math with @math where @math and @math are random rectangular matrices with i.i.d. centered entries. The function @math is applied pointwise and can be seen as an activation function in (random) neural networks. We compute the asymptotic empirical distribution of this ensemble in the case where @math and @math have sub-Gaussian tails and @math is real analytic. This extends a previous result where the case of Gaussian matrices @math and @math is considered. We also investigate the same questions in the multi-layer case, regarding neural network applications. Given an @math matrix @math , its Schatten- @math norm, @math , is defined as @math , where @math is the @math -th largest singular value of @math . These norms have been studied in functional analysis in the context of non-commutative @math -spaces, and recently in data stream and linear sketching models of computation. Basic questions on the relations between these norms, such as their embeddability, are still open. Specifically, given a set of matrices @math , suppose we want to construct a linear map @math such that @math for each @math , where @math and @math , and further, @math for a given approximation factor @math and real number @math . Then how large do @math and @math need to be as a function of @math ? We nearly resolve this question for every @math , for the case where @math can be expressed as @math , where @math and @math are arbitrary matrices that are allowed to depend on @math , that is, @math can be implemented by left and right matrix multiplication. Namely, for every @math , we provide nearly matching upper and lower bounds on the size of @math and @math as a function of @math . Importantly, our upper bounds are oblivious , meaning that @math and @math do not depend on the @math , while our lower bounds hold even if @math and @math depend on the @math . As an application of our upper bounds, we answer a recent open question of about space-approximation trade-offs for the Schatten @math -norm, showing in a data stream it is possible to estimate the Schatten- @math norm up to a factor of @math using @math space. Consider an @math matrix @math with the restricted isometry property of order k and level @math ; that is, the norm of any k-sparse vector in @math is preserved to within a multiplicative factor of @math under application of @math . We show that by randomizing the column signs of such a matrix @math , the resulting map with high probability embeds any fixed set of @math points in @math into @math without distorting the norm of any point in the set by more than a factor of @math . Consequently, matrices with the restricted isometry property and with randomized column signs provide optimal Johnson–Lindenstrauss embeddings up to logarithmic factors in N. In particular, our results improve the best known bounds on the necessary embedding dimension m for a wide class of structured random matrices; for partial Fourier and partial Hadamard matrices, we improve the recent bound @math given by Ailon and Liberty to $m... The following problem is considered: given a matrix @math in @math , ( @math rows and @math columns), a vector @math in @math , and @math , compute a vector @math satisfying @math if such exists, such that @math has the fewest number of non-zero entries over all such vectors. It is shown that the problem is NP-hard, but that the well-known greedy heuristic is good in that it computes a solution with at most @math non-zero entries, where @math is the optimum number of nonzero entries at error @math , @math is the matrix obtained by normalizing each column of @math with respect to the @math norm, and @math is its pseudo-inverse. Compressed sensing is a signal processing technique in which data is acquired directly in a compressed form. There are two modeling approaches that can be considered: the worst-case (Hamming) approach and a statistical mechanism, in which the signals are modeled as random processes rather than as individual sequences. In this paper, the second approach is studied. In particular, we consider a model of the form @math , where each comportment of @math is given by @math , where @math are i.i.d. Gaussian random variables, and @math are binary random variables independent of @math , and not necessarily independent and identically distributed (i.i.d.), @math is a random matrix with i.i.d. entries, and @math is white Gaussian noise. Using a direct relationship between optimum estimation and certain partition functions, and by invoking methods from statistical mechanics and from random matrix theory (RMT), we derive an asymptotic formula for the minimum mean-square error (MMSE) of estimating the input vector @math given @math and @math , as @math , keeping the measurement rate, @math , fixed. In contrast to previous derivations, which are based on the replica method, the analysis carried out in this paper is rigorous. This text is about spiked models of non Hermitian random matrices. More specifically, we consider matrices of the type @math , where the rank of @math stays bounded as the dimension goes to infinity and where the matrix @math is a non Hermitian random matrix, satisfying an isotropy hypothesis: its distribution is invariant under the left and right actions of the unitary group. The macroscopic eigenvalue distribution of such matrices is governed by the so called Single Ring Theorem, due to Guionnet, Krishnapur and Zeitouni. We first prove that if @math has some eigenvalues out of the maximal circle of the single ring, then @math has some eigenvalues (called outliers) in the neighborhood of those of @math , which is not the case for the eigenvalues of @math in the inner cycle of the single ring. Then, we study the fluctuations of the outliers of @math around the eigenvalues of @math and prove that they are distributed as the eigenvalues of some finite dimensional random matrices. Such facts had already been noticed for Hermitian models. More surprising facts are that outliers can here have very various rates of convergence to their limits (depending on the Jordan Canonical Form of @math ) and that some correlations can appear between outliers at a macroscopic distance from each other (a fact already noticed by Knowles and Yin in the Hermitian case, but only in the case of non Gaussian models, whereas spiked Gaussian matrices belong to our model and can have such correlated outliers). Our first result generalizes a previous result by Tao for matrices with i.i.d. entries, whereas the second one (about the fluctuations) is new. Let @math be an @math real random matrix whose elements are independent and identically distributed standard normal random variables, and let @math be the 2-norm condition number of @math . We prove that, for any @math , @math , and @math , @math satisfies @math where @math and @math @math are universal positive constants independent of @math , @math , and @math . Moreover, for any @math and @math , @math A similar pair of results for complex Gaussian random matrices is also established. This paper is concerned with the problem min @math where X is a convex subset of a linear space H, and f is a smooth real-valued function on H. We propose the class of methods @math , where P denotes projection on X with respect to a Hilbert space norm @math , @math denotes the Frechet derivative of f at @math with respect to another Hilbert space norm @math on H, and @math is a positive scalar stepsize. We thus remove an important restriction in the original proposal of Goldstein [1] and Levitin and Poljak [2], where the norms @math and @math must be the same. It is therefore possible to match the norm @math with the structure of X so that the projection operation is simplified while at the same time reserving the option to choose @math on the basis of approximations to the Hessian of f so as to attain a typically superlinear rate of convergence. The resulting methods are particularly attrac... We consider the problem of estimating an unknown signal @math from noisy linear observations @math . In many practical instances, @math has a certain structure that can be captured by a structure inducing convex function @math . For example, @math norm can be used to encourage a sparse solution. To estimate @math with the aid of @math , we consider the well-known LASSO method and provide sharp characterization of its performance. We assume the entries of the measurement matrix @math and the noise vector @math have zero-mean normal distributions with variances @math and @math respectively. For the LASSO estimator @math , we attempt to calculate the Normalized Square Error (NSE) defined as @math as a function of the noise level @math , the number of observations @math and the structure of the signal. We show that, the structure of the signal @math and choice of the function @math enter the error formulae through the summary parameters @math and @math , which are defined as the Gaussian squared-distances to the subdifferential cone and to the @math -scaled subdifferential, respectively. The first LASSO estimator assumes a-priori knowledge of @math and is given by @math . We prove that its worst case NSE is achieved when @math and concentrates around @math . Secondly, we consider @math , for some @math . This time the NSE formula depends on the choice of @math and is given by @math . We then establish a mapping between this and the third estimator @math . Finally, for a number of important structured signal classes, we translate our abstract formulae to closed-form upper bounds on the NSE. Recovering a low-rank tensor from incomplete information is a recurring problem in signal processing and machine learning. The most popular convex relaxation of this problem minimizes the sum of the nuclear norms of the unfoldings of the tensor. We show that this approach can be substantially suboptimal: reliably recovering a @math -way tensor of length @math and Tucker rank @math from Gaussian measurements requires @math observations. In contrast, a certain (intractable) nonconvex formulation needs only @math observations. We introduce a very simple, new convex relaxation, which partially bridges this gap. Our new formulation succeeds with @math observations. While these results pertain to Gaussian measurements, simulations strongly suggest that the new norm also outperforms the sum of nuclear norms for tensor completion from a random subset of entries. Our lower bound for the sum-of-nuclear-norms model follows from a new result on recovering signals with multiple sparse structures (e.g. sparse, low rank), which perhaps surprisingly demonstrates the significant suboptimality of the commonly used recovery approach via minimizing the sum of individual sparsity inducing norms (e.g. @math , nuclear norm). Our new formulation for low-rank tensor recovery however opens the possibility in reducing the sample complexity by exploiting several structures jointly. We give efficient algorithms for volume sampling, i.e., for picking @math -subsets of the rows of any given matrix with probabilities proportional to the squared volumes of the simplices defined by them and the origin (or the squared volumes of the parallelepipeds defined by these subsets of rows). In other words, we can efficiently sample @math -subsets of @math with probabilities proportional to the corresponding @math by @math principal minors of any given @math by @math positive semi definite matrix. This solves an open problem from the monograph on spectral algorithms by Kannan and Vempala (see Section @math of KV , also implicit in BDM, DRVW ). Our first algorithm for volume sampling @math -subsets of rows from an @math -by- @math matrix runs in @math arithmetic operations (where @math is the exponent of matrix multiplication) and a second variant of it for @math -approximate volume sampling runs in @math arithmetic operations, which is almost linear in the size of the input (i.e., the number of entries) for small @math . Our efficient volume sampling algorithms imply the following results for low-rank matrix approximation: (1) Given @math , in @math arithmetic operations we can find @math of its rows such that projecting onto their span gives a @math -approximation to the matrix of rank @math closest to @math under the Frobenius norm. This improves the @math -approximation of Boutsidis, Drineas and Mahoney BDM and matches the lower bound shown in DRVW . The method of conditional expectations gives a algorithm with the same complexity. The running time can be improved to @math at the cost of losing an extra @math in the approximation factor. (2) The same rows and projection as in the previous point give a @math -approximation to the matrix of rank @math closest to @math under the spectral norm. In this paper, we show an almost matching lower bound of @math , even for @math . We investigate the minimization of Newton's functional for the problem of the body of minimal resistance with maximal height @math cite butt in the class of convex developable functions defined in a disc. This class is a natural candidate to find a (non-radial) minimizer in accordance with the results of cite lrp2. We prove that the minimizer in this class has a minimal set in the form of a regular polygon with @math sides centered in the disc, and numerical experiments indicate that the natural number @math is a non-decreasing function of @math . The corresponding functions all achieve a lower value of the functional than the optimal radially symmetric function with the same height @math . This paper makes two contributions to the areas of anti-concentration and non-asymptotic random matrix theory. First, we study the counting problem in inverse Littlewood-Offord theory for general random variables: for random variables @math which are i.i.d. copies of a random variable @math (satisfying some mild hypotheses), how many integer vectors @math in a prescribed box have large @math ? Building on recent work of Ferber, Jain, Luh, and Samotij (who treated the case when @math is a Rademacher random variable), we provide significantly better bounds for this problem than those obtained using the inverse Littlewood-Offord theorems of Tao and Vu, and Nguyen and Vu. Next, we study the non-asymptotic behavior of the least singular value @math of a random @math square matrix @math with i.i.d. entries, which are only assumed to be centered with variance @math . As an application of our counting theorem, and utilizing and developing a recent work by the author, we show that for all @math , @math , where @math is an absolute constant, thereby providing a considerably simpler proof of an approximate version of an essentially optimal result due to Rebrova and Tikhomirov. Motivated by applications in which the data may be formulated as a matrix, we consider algorithms for several common linear algebra problems. These algorithms make more efficient use of computational resources, such as the computation time, random access memory (RAM), and the number of passes over the data, than do previously known algorithms for these problems. In this paper, we devise two algorithms for the matrix multiplication problem. Suppose @math and @math (which are @math and @math , respectively) are the two input matrices. In our main algorithm, we perform @math independent trials, where in each trial we randomly sample an element of @math with an appropriate probability distribution @math on @math . We form an @math matrix @math consisting of the sampled columns of @math , each scaled appropriately, and we form a @math matrix @math using the corresponding rows of @math , again scaled appropriately. The choice of @math and the column and row scaling are crucial features of the algorithm. When these are chosen judiciously, we show that @math is a good approximation to @math . More precisely, we show that @math where @math denotes the Frobenius norm, i.e., @math . This algorithm can be implemented without storing the matrices @math and @math in RAM, provided it can make two passes over the matrices stored in external memory and use @math additional RAM to construct @math and @math . We then present a second matrix multiplication algorithm which is similar in spirit to our main algorithm. In addition, we present a model (the pass-efficient model) in which the efficiency of these and other approximate matrix algorithms may be studied and which we argue is well suited to many applications involving massive data sets. In this model, the scarce computational resources are the number of passes over the data and the additional space and time required by the algorithm. The input matrices may be presented in any order of the entries (and not just row or column order), as is the case in many applications where, e.g., the data has been written in by multiple agents. In addition, the input matrices may be presented in a sparse representation, where only the nonzero entries are written. For any real number @math , we nearly completely characterize the space complexity of estimating @math for @math matrices @math in which each row and each column has @math non-zero entries and whose entries are presented one at a time in a data stream model. Here the @math are the singular values of @math , and when @math , @math is the @math -th power of the Schatten @math -norm. We show that when @math is not an even integer, to obtain a @math -approximation to @math with constant probability, any @math -pass algorithm requires @math bits of space, where @math as @math and @math is a constant independent of @math . However, when @math is an even integer, we give an upper bound of @math bits of space, which holds even in the turnstile data stream model. The latter is optimal up to @math factors. Our results considerably strengthen lower bounds in previous work for arbitrary (not necessarily sparse) matrices @math : the previous best lower bound was @math for @math , @math for @math and @math for @math . We note for @math , while our lower bound for even integers is the same, for other @math in this range our lower bound is @math , which is considerably stronger than the previous @math for small enough constant @math . We obtain similar near-linear lower bounds for Ky-Fan norms, SVD entropy, eigenvalue shrinkers, and M-estimators, many of which could have been solvable in logarithmic space prior to our work. Interpolators---estimators that achieve zero training error---have attracted growing attention in machine learning, mainly because state-of-the art neural networks appear to be models of this type. In this paper, we study minimum @math norm ( ridgeless') interpolation in high-dimensional least squares regression. We consider two different models for the feature distribution: a linear model, where the feature vectors @math are obtained by applying a linear transform to a vector of i.i.d. entries, @math (with @math ); and a nonlinear model, where the feature vectors are obtained by passing the input through a random one-layer neural network, @math (with @math , @math a matrix of i.i.d. entries, and @math an activation function acting componentwise on @math ). We recover---in a precise quantitative way---several phenomena that have been observed in large-scale neural networks and kernel machines, including the double descent' behavior of the prediction risk, and the potential benefits of overparametrization. Low-rank approximation is a common tool used to accelerate kernel methods: the @math kernel matrix @math is approximated via a rank- @math matrix @math which can be stored in much less space and processed more quickly. In this work we study the limits of computationally efficient low-rank kernel approximation. We show that for a broad class of kernels, including the popular Gaussian and polynomial kernels, computing a relative error @math -rank approximation to @math is at least as difficult as multiplying the input data matrix @math by an arbitrary matrix @math . Barring a breakthrough in fast matrix multiplication, when @math is not too large, this requires @math time where @math is the number of non-zeros in @math . This lower bound matches, in many parameter regimes, recent work on subquadratic time algorithms for low-rank approximation of general kernels [MM16,MW17], demonstrating that these algorithms are unlikely to be significantly improved, in particular to @math input sparsity runtimes. At the same time there is hope: we show for the first time that @math time approximation is possible for general radial basis function kernels (e.g., the Gaussian kernel) for the closely related problem of low-rank approximation of the kernelized dataset. For any integers @math and @math , we show the existence of a set of @math vectors @math such that any embedding @math satisfying @math must have @math This lower bound matches the upper bound given by the Johnson-Lindenstrauss lemma [JL84]. Furthermore, our lower bound holds for nearly the full range of @math of interest, since there is always an isometric embedding into dimension @math (either the identity map, or projection onto @math ). Previously such a lower bound was only known to hold against linear maps @math , and not for such a wide range of parameters @math [LN16]. The best previously known lower bound for general @math was @math [Wel74, Lev83, Alo03], which is suboptimal for any @math . Consider the recovery of an unknown signal @math from quantized linear measurements. In the one-bit compressive sensing setting, one typically assumes that @math is sparse, and that the measurements are of the form @math . Since such measurements give no information on the norm of @math , recovery methods typically assume that @math . We show that if one allows more generally for quantized affine measurements of the form @math , and if the vectors @math are random, an appropriate choice of the affine shifts @math allows norm recovery to be easily incorporated into existing methods for one-bit compressive sensing. In addition, we show that for arbitrary fixed @math in the annulus @math , one may estimate the norm @math up to additive error @math from @math such binary measurements through a single evaluation of the inverse Gaussian error function. Finally, all of our recovery guarantees can be made universal over sparse vectors in the sense that with high probability, one set of measurements and thresholds can successfully estimate all sparse vectors @math in a Euclidean ball of known radius. We consider the problem of selecting the best subset of exactly @math columns from an @math matrix @math . We present and analyze a novel two-stage algorithm that runs in @math time and returns as output an @math matrix @math consisting of exactly @math columns of @math . In the first (randomized) stage, the algorithm randomly selects @math columns according to a judiciously-chosen probability distribution that depends on information in the top- @math right singular subspace of @math . In the second (deterministic) stage, the algorithm applies a deterministic column-selection procedure to select and return exactly @math columns from the set of columns selected in the first stage. Let @math be the @math matrix containing those @math columns, let @math denote the projection matrix onto the span of those columns, and let @math denote the best rank- @math approximation to the matrix @math . Then, we prove that, with probability at least 0.8, @math This Frobenius norm bound is only a factor of @math worse than the best previously existing existential result and is roughly @math better than the best previous algorithmic result for the Frobenius norm version of this Column Subset Selection Problem (CSSP). We also prove that, with probability at least 0.8, @math This spectral norm bound is not directly comparable to the best previously existing bounds for the spectral norm version of this CSSP. Our bound depends on @math , whereas previous results depend on @math ; if these two quantities are comparable, then our bound is asymptotically worse by a @math factor.