Consider an Erd "os-Renyi random graph in which each edge is present independently with probability 1 2, except for a subset @math of the vertices that form a clique (a completely connected subgraph). We consider the problem of identifying the clique, given a realization of such a random graph. The best known algorithm provably finds the clique in linear time with high probability, provided @math dekel2011finding . Spectral methods can be shown to fail on cliques smaller than @math . In this paper we describe a nearly linear time algorithm that succeeds with high probability for @math for any @math . This is the first algorithm that provably improves over spectral methods. We further generalize the hidden clique problem to other background graphs (the standard case corresponding to the complete graph on @math vertices). For large girth regular graphs of degree @math we prove that local' algorithms succeed if @math and fail if @math .
A rich line of research in statistics addresses the problem of identifying the non-zero entries in a sparse vector (or matrix) @math from observations @math where @math has typically i.i.d. standard Gaussian entries. In particular @cite_38 @cite_2 @cite_31 @cite_0 study cases in which the sparsity pattern of @math is structured'. For instance, we can take @math a matrix with @math if @math and @math otherwise. This fits the framework studied in this paper, for @math the complete graph and @math , @math . This literature however disregards computational considerations. Greedy search methods were developed in several papers, see e.g. @cite_5 @cite_22 .
@cite_38 @cite_22 @cite_0 @cite_2 @cite_5 @cite_31
1984238247 2952259914 2107307520 2072157164
We investigate the minimization of Newton's functional for the problem of the body of minimal resistance with maximal height @math cite butt in the class of convex developable functions defined in a disc. This class is a natural candidate to find a (non-radial) minimizer in accordance with the results of cite lrp2. We prove that the minimizer in this class has a minimal set in the form of a regular polygon with @math sides centered in the disc, and numerical experiments indicate that the natural number @math is a non-decreasing function of @math . The corresponding functions all achieve a lower value of the functional than the optimal radially symmetric function with the same height @math . The following problem is considered: given a matrix @math in @math , ( @math rows and @math columns), a vector @math in @math , and @math , compute a vector @math satisfying @math if such exists, such that @math has the fewest number of non-zero entries over all such vectors. It is shown that the problem is NP-hard, but that the well-known greedy heuristic is good in that it computes a solution with at most @math non-zero entries, where @math is the optimum number of nonzero entries at error @math , @math is the matrix obtained by normalizing each column of @math with respect to the @math norm, and @math is its pseudo-inverse. In sparse principal component analysis we are given noisy observations of a low-rank matrix of dimension @math and seek to reconstruct it under additional sparsity assumptions. In particular, we assume here each of the principal components @math has at most @math non-zero entries. We are particularly interested in the high dimensional regime wherein @math is comparable to, or even much larger than @math . In an influential paper, johnstone2004sparse introduced a simple algorithm that estimates the support of the principal vectors @math by the largest entries in the diagonal of the empirical covariance. This method can be shown to identify the correct support with high probability if @math , and to fail with high probability if @math for two constants @math . Despite a considerable amount of work over the last ten years, no practical algorithm exists with provably better support recovery guarantees. Here we analyze a covariance thresholding algorithm that was recently proposed by KrauthgamerSPCA . On the basis of numerical simulations (for the rank-one case), these authors conjectured that covariance thresholding correctly recover the support with high probability for @math (assuming @math of the same order as @math ). We prove this conjecture, and in fact establish a more general guarantee including higher-rank as well as @math much smaller than @math . Recent lower bounds berthet2013computational, ma2015sum suggest that no polynomial time algorithm can do significantly better. The key technical component of our analysis develops new bounds on the norm of kernel random matrices, in regimes that were not considered before. We consider the problem of estimating an unknown signal @math from noisy linear observations @math . In many practical instances, @math has a certain structure that can be captured by a structure inducing convex function @math . For example, @math norm can be used to encourage a sparse solution. To estimate @math with the aid of @math , we consider the well-known LASSO method and provide sharp characterization of its performance. We assume the entries of the measurement matrix @math and the noise vector @math have zero-mean normal distributions with variances @math and @math respectively. For the LASSO estimator @math , we attempt to calculate the Normalized Square Error (NSE) defined as @math as a function of the noise level @math , the number of observations @math and the structure of the signal. We show that, the structure of the signal @math and choice of the function @math enter the error formulae through the summary parameters @math and @math , which are defined as the Gaussian squared-distances to the subdifferential cone and to the @math -scaled subdifferential, respectively. The first LASSO estimator assumes a-priori knowledge of @math and is given by @math . We prove that its worst case NSE is achieved when @math and concentrates around @math . Secondly, we consider @math , for some @math . This time the NSE formula depends on the choice of @math and is given by @math . We then establish a mapping between this and the third estimator @math . Finally, for a number of important structured signal classes, we translate our abstract formulae to closed-form upper bounds on the NSE. We introduce the framework of blind regression motivated by matrix completion for recommendation systems: given @math users, @math movies, and a subset of user-movie ratings, the goal is to predict the unobserved user-movie ratings given the data, i.e., to complete the partially observed matrix. Following the framework of non-parametric statistics, we posit that user @math and movie @math have features @math and @math respectively, and their corresponding rating @math is a noisy measurement of @math for some unknown function @math . In contrast with classical regression, the features @math are not observed, making it challenging to apply standard regression methods to predict the unobserved ratings. Inspired by the classical Taylor's expansion for differentiable functions, we provide a prediction algorithm that is consistent for all Lipschitz functions. In fact, the analysis through our framework naturally leads to a variant of collaborative filtering, shedding insight into the widespread success of collaborative filtering in practice. Assuming each entry is sampled independently with probability at least @math with @math , we prove that the expected fraction of our estimates with error greater than @math is less than @math plus a polynomially decaying term, where @math is the variance of the additive entry-wise noise term. Experiments with the MovieLens and Netflix datasets suggest that our algorithm provides principled improvements over basic collaborative filtering and is competitive with matrix factorization methods. In sparse recovery we are given a matrix @math (the dictionary) and a vector of the form @math where @math is sparse, and the goal is to recover @math . This is a central notion in signal processing, statistics and machine learning. But in applications such as sparse coding, edge detection, compression and super resolution, the dictionary @math is unknown and has to be learned from random examples of the form @math where @math is drawn from an appropriate distribution --- this is the dictionary learning problem. In most settings, @math is overcomplete: it has more columns than rows. This paper presents a polynomial-time algorithm for learning overcomplete dictionaries; the only previously known algorithm with provable guarantees is the recent work of Spielman, Wang and Wright who gave an algorithm for the full-rank case, which is rarely the case in applications. Our algorithm applies to incoherent dictionaries which have been a central object of study since they were introduced in seminal work of Donoho and Huo. In particular, a dictionary is @math -incoherent if each pair of columns has inner product at most @math . The algorithm makes natural stochastic assumptions about the unknown sparse vector @math , which can contain @math non-zero entries (for any @math ). This is close to the best @math allowable by the best sparse recovery algorithms even if one knows the dictionary @math exactly. Moreover, both the running time and sample complexity depend on @math , where @math is the target accuracy, and so our algorithms converge very quickly to the true dictionary. Our algorithm can also tolerate substantial amounts of noise provided it is incoherent with respect to the dictionary (e.g., Gaussian). In the noisy setting, our running time and sample complexity depend polynomially on @math , and this is necessary. Compressed sensing is a signal processing technique in which data is acquired directly in a compressed form. There are two modeling approaches that can be considered: the worst-case (Hamming) approach and a statistical mechanism, in which the signals are modeled as random processes rather than as individual sequences. In this paper, the second approach is studied. In particular, we consider a model of the form @math , where each comportment of @math is given by @math , where @math are i.i.d. Gaussian random variables, and @math are binary random variables independent of @math , and not necessarily independent and identically distributed (i.i.d.), @math is a random matrix with i.i.d. entries, and @math is white Gaussian noise. Using a direct relationship between optimum estimation and certain partition functions, and by invoking methods from statistical mechanics and from random matrix theory (RMT), we derive an asymptotic formula for the minimum mean-square error (MMSE) of estimating the input vector @math given @math and @math , as @math , keeping the measurement rate, @math , fixed. In contrast to previous derivations, which are based on the replica method, the analysis carried out in this paper is rigorous. This paper makes two contributions to the areas of anti-concentration and non-asymptotic random matrix theory. First, we study the counting problem in inverse Littlewood-Offord theory for general random variables: for random variables @math which are i.i.d. copies of a random variable @math (satisfying some mild hypotheses), how many integer vectors @math in a prescribed box have large @math ? Building on recent work of Ferber, Jain, Luh, and Samotij (who treated the case when @math is a Rademacher random variable), we provide significantly better bounds for this problem than those obtained using the inverse Littlewood-Offord theorems of Tao and Vu, and Nguyen and Vu. Next, we study the non-asymptotic behavior of the least singular value @math of a random @math square matrix @math with i.i.d. entries, which are only assumed to be centered with variance @math . As an application of our counting theorem, and utilizing and developing a recent work by the author, we show that for all @math , @math , where @math is an absolute constant, thereby providing a considerably simpler proof of an approximate version of an essentially optimal result due to Rebrova and Tikhomirov. We study low rank matrix and tensor completion and propose novel algorithms that employ adaptive sampling schemes to obtain strong performance guarantees. Our algorithms exploit adaptivity to identify entries that are highly informative for learning the column space of the matrix (tensor) and consequently, our results hold even when the row space is highly coherent, in contrast with previous analyses. In the absence of noise, we show that one can exactly recover a @math matrix of rank @math from merely @math matrix entries. We also show that one can recover an order @math tensor using @math entries. For noisy recovery, our algorithm consistently estimates a low rank matrix corrupted with noise using @math entries. We complement our study with simulations that verify our theory and demonstrate the scalability of our algorithms. We study the recovery of sparse vectors from subsampled random convolutions via @math -minimization. We consider the setup in which both the subsampling locations as well as the generating vector are chosen at random. For a subgaussian generator with independent entries, we improve previously known estimates: if the sparsity @math is small enough, i.e., @math , we show that @math measurements are sufficient to recover @math -sparse vectors in dimension @math with high probability, matching the well-known condition for recovery from standard Gaussian measurements. If @math is larger, then essentially @math measurements are sufficient, again improving over previous estimates. Our results are shown via the so-called robust null space property which is weaker than the standard restricted isometry property. Our method of proof involves a novel combination of small ball estimates with chaining techniques which should be of independent interest. This paper is concerned with the asymptotic empirical eigenvalue distribution of a non linear random matrix ensemble. More precisely we consider @math with @math where @math and @math are random rectangular matrices with i.i.d. centered entries. The function @math is applied pointwise and can be seen as an activation function in (random) neural networks. We compute the asymptotic empirical distribution of this ensemble in the case where @math and @math have sub-Gaussian tails and @math is real analytic. This extends a previous result where the case of Gaussian matrices @math and @math is considered. We also investigate the same questions in the multi-layer case, regarding neural network applications. Interpolators---estimators that achieve zero training error---have attracted growing attention in machine learning, mainly because state-of-the art neural networks appear to be models of this type. In this paper, we study minimum @math norm ( ridgeless') interpolation in high-dimensional least squares regression. We consider two different models for the feature distribution: a linear model, where the feature vectors @math are obtained by applying a linear transform to a vector of i.i.d. entries, @math (with @math ); and a nonlinear model, where the feature vectors are obtained by passing the input through a random one-layer neural network, @math (with @math , @math a matrix of i.i.d. entries, and @math an activation function acting componentwise on @math ). We recover---in a precise quantitative way---several phenomena that have been observed in large-scale neural networks and kernel machines, including the double descent' behavior of the prediction risk, and the potential benefits of overparametrization. We consider linear regression in the high-dimensional regime where the number of observations @math is smaller than the number of parameters @math . A very successful approach in this setting uses @math -penalized least squares (a.k.a. the Lasso) to search for a subset of @math parameters that best explain the data, while setting the other parameters to zero. Considerable amount of work has been devoted to characterizing the estimation and model selection problems within this approach. In this paper we consider instead the fundamental, but far less understood, question of . More precisely, we address the problem of computing p-values for single regression coefficients. On one hand, we develop a general upper bound on the minimax power of tests with a given significance level. On the other, we prove that this upper bound is (nearly) achievable through a practical procedure in the case of random design matrices with independent entries. Our approach is based on a debiasing of the Lasso estimator. The analysis builds on a rigorous characterization of the asymptotic distribution of the Lasso estimator and its debiased version. Our result holds for optimal sample size, i.e., when @math is at least on the order of @math . We generalize our approach to random design matrices with i.i.d. Gaussian rows @math . In this case we prove that a similar distributional characterization (termed standard distributional limit') holds for @math much larger than @math . Finally, we show that for optimal sample size, @math being at least of order @math , the standard distributional limit for general Gaussian designs can be derived from the replica heuristics in statistical physics. This paper considers the problem of designing maximum distance separable (MDS) codes over small fields with constraints on the support of their generator matrices. For any given @math binary matrix @math , the GM-MDS conjecture, proposed by , states that if @math satisfies the so-called MDS condition, then for any field @math of size @math , there exists an @math MDS code whose generator matrix @math , with entries in @math , fits the matrix @math (i.e., @math is the support matrix of @math ). Despite all the attempts by the coding theory community, this conjecture remains still open in general. It was shown, independently by and , that the GM-MDS conjecture holds if the following conjecture, referred to as the TM-MDS conjecture, holds: if @math satisfies the MDS condition, then the determinant of a transform matrix @math , such that @math fits @math , is not identically zero, where @math is a Vandermonde matrix with distinct parameters. In this work, we first reformulate the TM-MDS conjecture in terms of the Wronskian determinant, and then present an algebraic-combinatorial approach based on polynomial-degree reduction for proving this conjecture. Our proof technique's strength is based primarily on reducing inherent combinatorics in the proof. We demonstrate the strength of our technique by proving the TM-MDS conjecture for the cases where the number of rows ( @math ) of @math is upper bounded by @math . For this class of special cases of @math where the only additional constraint is on @math , only cases with @math were previously proven theoretically, and the previously used proof techniques are not applicable to cases with @math . Given a large data matrix @math , we consider the problem of determining whether its entries are i.i.d. with some known marginal distribution @math , or instead @math contains a principal submatrix @math whose entries have marginal distribution @math . As a special case, the hidden (or planted) clique problem requires to find a planted clique in an otherwise uniformly random graph. Assuming unbounded computational resources, this hypothesis testing problem is statistically solvable provided @math for a suitable constant @math . However, despite substantial effort, no polynomial time algorithm is known that succeeds with high probability when @math . Recently Meka and Wigderson meka2013association , proposed a method to establish lower bounds within the Sum of Squares (SOS) semidefinite hierarchy. Here we consider the degree- @math SOS relaxation, and study the construction of meka2013association to prove that SOS fails unless @math . An argument presented by Barak implies that this lower bound cannot be substantially improved unless the witness construction is changed in the proof. Our proof uses the moments method to bound the spectrum of a certain random association scheme, i.e. a symmetric random matrix whose rows and columns are indexed by the edges of an Erd "os-Renyi random graph. We develop several efficient algorithms for the classical problem, which is used in many diverse areas, from preconditioning linear systems to approximation of the permanent. On an input @math matrix @math , this problem asks to find diagonal (scaling) matrices @math and @math (if they exist), so that @math @math -approximates a doubly stochastic, or more generally a matrix with prescribed row and column sums. We address the general scaling problem as well as some important special cases. In particular, if @math has @math nonzero entries, and if there exist @math and @math with polynomially large entries such that @math is doubly stochastic, then we can solve the problem in total complexity @math . This greatly improves on the best known previous results, which were either @math or @math . Our algorithms are based on tailor-made first and second order techniques, combined with other recent advances in continuous optimization, which may be of independent interest for solving similar problems. We use randomness to exploit the potential sparsity of the Boolean matrix product in order to speed up the computation of the product. Our new fast output-sensitive algorithm for Boolean matrix product and its witnesses is randomized and provides the Boolean product and its witnesses almost certainly. Its worst-case time performance is expressed in terms of the input size and the number of non-zero entries of the product matrix. It runs in time @math , where the input matrices have size n×n, the number of non-zero entries in the product matrix is at most s, ? is the exponent of the fast matrix multiplication and @math denotes O(f(n)log? d n) for some constant d. By the currently best bound on ?, its running time can be also expressed as @math . Our algorithm is substantially faster than the output-sensitive column-row method for Boolean matrix product for s larger than n 1.232 and it is never slower than the fast @math -time algorithm for this problem. By applying the fast rectangular matrix multiplication, we can refine our upper bound further to the form @math , where ?(p,q,t) is the exponent of the fast multiplication of an n p ×n q matrix by an n q ×n t matrix. We also present a partial derandomization of our algorithm as well as its generalization to include the Boolean product of rectangular Boolean matrices. Finally, we show several applications of our output-sensitive algorithms. Motivated by applications in which the data may be formulated as a matrix, we consider algorithms for several common linear algebra problems. These algorithms make more efficient use of computational resources, such as the computation time, random access memory (RAM), and the number of passes over the data, than do previously known algorithms for these problems. In this paper, we devise two algorithms for the matrix multiplication problem. Suppose @math and @math (which are @math and @math , respectively) are the two input matrices. In our main algorithm, we perform @math independent trials, where in each trial we randomly sample an element of @math with an appropriate probability distribution @math on @math . We form an @math matrix @math consisting of the sampled columns of @math , each scaled appropriately, and we form a @math matrix @math using the corresponding rows of @math , again scaled appropriately. The choice of @math and the column and row scaling are crucial features of the algorithm. When these are chosen judiciously, we show that @math is a good approximation to @math . More precisely, we show that @math where @math denotes the Frobenius norm, i.e., @math . This algorithm can be implemented without storing the matrices @math and @math in RAM, provided it can make two passes over the matrices stored in external memory and use @math additional RAM to construct @math and @math . We then present a second matrix multiplication algorithm which is similar in spirit to our main algorithm. In addition, we present a model (the pass-efficient model) in which the efficiency of these and other approximate matrix algorithms may be studied and which we argue is well suited to many applications involving massive data sets. In this model, the scarce computational resources are the number of passes over the data and the additional space and time required by the algorithm. The input matrices may be presented in any order of the entries (and not just row or column order), as is the case in many applications where, e.g., the data has been written in by multiple agents. In addition, the input matrices may be presented in a sparse representation, where only the nonzero entries are written. Abstract Irregular sampling and “stable sampling” of band-limited functions have been studied by H.J. Landau [H.J. Landau, Necessary density conditions for sampling and interpolation of certain entire functions, Acta Math. 117 (1967) 37–52]. We prove that quasicrystals are sets of stable sampling. To cite this article: B. Matei, Y. Meyer, C. R. Acad. Sci. Paris, Ser. I 346 (2008). A discrete uncoupling process for finite spaces is introduced, called the Markov Cluster Process or the MCL process. The process is the engine for the graph clustering algorithm called the MCL algorithm. The MCL process takes a stochastic matrix as input, and then alternates expansion and inflation, each step defining a stochastic matrix in terms of the previous one. Expansion corresponds with taking the @math th power of a stochastic matrix, where @math . Inflation corresponds with a parametrized operator @math , @math , that maps the set of (column) stochastic matrices onto itself. The image @math is obtained by raising each entry in @math to the @math th power and rescaling each column to have sum 1 again. In practice the process converges very fast towards a limit that is invariant under both matrix multiplication and inflation, with quadratic convergence around the limit points. The heuristic behind the process is its expected behavior for (Markov) graphs possessing cluster structure. The process is typically applied to the matrix of random walks on a given graph @math , and the connected components of (the graph associated with) the process limit generically allow a clustering interpretation of @math . The limit is in general extremely sparse and iterands are sparse in a weighted sense, implying that the MCL algorithm is very fast and highly scalable. Several mathematical properties of the MCL process are established. Most notably, the process (and algorithm) iterands posses structural properties generalizing the mapping from process limits onto clusterings. The inflation operator @math maps the class of matrices that are diagonally similar to a symmetric matrix onto itself. The phrase diagonally positive semi-definite (dpsd) is used for matrices that are diagonally similar to a positive semi-definite matrix. For @math and for @math a stochastic dpsd matrix, the image @math is again dpsd. Determinantal inequalities satisfied by a dpsd matrix @math imply a natural ordering among the diagonal elements of @math , generalizing the mapping of process limits onto clusterings. The spectrum of @math is of the form @math , where @math is the number of endclasses of the ordering associated with @math , and @math is the dimension of @math . This attests to the uncoupling effect of the inflation operator.