We introduce normalized nonnegative models (NNM) for explorative data analysis. NNMs are partial convexifications of models from probability theory. We demonstrate their value at the example of item recommendation. We show that NNM-based recommender systems satisfy three criteria that all recommender systems should ideally satisfy: high predictive power, computational tractability, and expressive representations of users and items. Expressive user and item representations are important in practice to succinctly summarize the pool of customers and the pool of items. In NNMs, user representations are expressive because each user's preference can be regarded as normalized mixture of preferences of stereotypical users. The interpretability of item and user representations allow us to arrange properties of items (e.g., genres of movies or topics of documents) or users (e.g., personality traits) hierarchically.
In recommendation we are interested in the co-occurence of pairs @math of users @math and items @math . Depending on the application, @math may be interpreted as , , etc. In , @cite_3 @cite_9 (and similarly in their extension @cite_14 ) we model @math as random variable with distribution P [u,i] = k=1 ^K ; P [u|k] ; P [i|k]. Hence, @math is expressed as a inner product of two vectors: @math and @math . This is reminiscent of where we expressed the probability of the event in terms of the inner product between @math and @math . Therefore, one half of the inner product agrees with the identity as both @math and @math are probability distributions. However, aspect models and NNMs disagree on the other half (i.e., @math ) because @math is constrained through the existence of @math such that @math . More importantly however, the difference between aspect models and NNMs lies in the different interpretations of @math and @math on the one hand and @math and @math on the other hand.
@cite_9 @cite_14 @cite_3
2107743791 1880262756
Space-economical estimation of the pth frequency moments, defined as , for p> 0, are of interest in estimating all-pairs distances in a large data matrix [14], machine learning, and in data stream computation. Random sketches formed by the inner product of the frequency vector f 1 , ..., f n with a suitably chosen random vector were pioneered by Alon, Matias and Szegedy [1], and have since played a central role in estimating F p and for data stream computations in general. The concept of p-stable sketches formed by the inner product of the frequency vector with a random vector whose components are drawn from a p-stable distribution, was proposed by Indyk for estimating F p , for 0 < p< 2, and has been further studied in Li [13]. In this paper, we consider the problem of estimating F p , for 0 p , for 0 sketch [7] and the structure [5]. Our algorithms require space @math to estimate F p to within 1 ±i¾?factors and requires expected time @math to process each update. Thus, our technique trades an @math factor in space for much more efficient processing of stream updates. We also present a stand-alone iterative estimator for F 1 . Let @math be a cyclic @math -algebra of dimension @math with finite dimensional cohomology only in dimension one and two. By transfer theorem there exists a cyclic @math -algebra structure on the cohomology @math . The inner product plus the higher products of the cyclic @math -algebra defines a superpotential function @math on @math . We associate with an analytic Milnor fiber for the formal function @math and define the Euler characteristic of @math is to be the Euler characteristic of the 'et ale cohomology of the analytic Milnor fiber. In this paper we prove a Thom-Sebastiani type formula for the Euler characteristic of cyclic @math -algebras. As applications we prove the Joyce-Song formulas about the Behrend function identities for semi-Schur objects in the derived category of coherent sheaves over Calabi-Yau threefolds. A motivic Thom-Sebastiani type formula and a conjectural motivic Joyce-Song formulas for the motivic Milnor fiber of cyclic @math -algebras are also discussed. In sparse recovery we are given a matrix @math (the dictionary) and a vector of the form @math where @math is sparse, and the goal is to recover @math . This is a central notion in signal processing, statistics and machine learning. But in applications such as sparse coding, edge detection, compression and super resolution, the dictionary @math is unknown and has to be learned from random examples of the form @math where @math is drawn from an appropriate distribution --- this is the dictionary learning problem. In most settings, @math is overcomplete: it has more columns than rows. This paper presents a polynomial-time algorithm for learning overcomplete dictionaries; the only previously known algorithm with provable guarantees is the recent work of Spielman, Wang and Wright who gave an algorithm for the full-rank case, which is rarely the case in applications. Our algorithm applies to incoherent dictionaries which have been a central object of study since they were introduced in seminal work of Donoho and Huo. In particular, a dictionary is @math -incoherent if each pair of columns has inner product at most @math . The algorithm makes natural stochastic assumptions about the unknown sparse vector @math , which can contain @math non-zero entries (for any @math ). This is close to the best @math allowable by the best sparse recovery algorithms even if one knows the dictionary @math exactly. Moreover, both the running time and sample complexity depend on @math , where @math is the target accuracy, and so our algorithms converge very quickly to the true dictionary. Our algorithm can also tolerate substantial amounts of noise provided it is incoherent with respect to the dictionary (e.g., Gaussian). In the noisy setting, our running time and sample complexity depend polynomially on @math , and this is necessary. Let @math denote a random symmetric @math matrix whose upper diagonal entries are independent and identically distributed Bernoulli random variables (which take values @math and @math with probability @math each). It is widely conjectured that @math is singular with probability at most @math . On the other hand, the best known upper bound on the singularity probability of @math , due to Vershynin (2011), is @math , for some unspecified small constant @math . This improves on a polynomial singularity bound due to Costello, Tao, and Vu (2005), and a bound of Nguyen (2011) showing that the singularity probability decays faster than any polynomial. In this paper, improving on all previous results, we show that the probability of singularity of @math is at most @math for all sufficiently large @math . The proof utilizes and extends a novel combinatorial approach to discrete random matrix theory, which has been recently introduced by the authors together with Luh and Samotij. We prove that for any decision tree calculating a boolean function @math , [ [f] i=1 ^n (f), ] where @math is the probability that the @math th input variable is read and @math is the influence of the @math th variable on @math . The variance, influence and probability are taken with respect to an arbitrary product measure on @math . It follows that the minimum depth of a decision tree calculating a given balanced function is at least the reciprocal of the largest influence of any input variable. Likewise, any balanced boolean function with a decision tree of depth @math has a variable with influence at least @math . The only previous nontrivial lower bound known was @math . Our inequality has many generalizations, allowing us to prove influence lower bounds for randomized decision trees, decision trees on arbitrary product probability spaces, and decision trees with non-boolean outputs. As an application of our results we give a very easy proof that the randomized query complexity of nontrivial monotone graph properties is at least @math , where @math is the number of vertices and @math is the critical threshold probability. This supersedes the milestone @math bound of Hajnal and is sometimes superior to the best known lower bounds of Chakrabarti-Khot and Friedgut-Kahn-Wigderson. We consider the scenario where the executions of different business processes are traced into a log, where each trace describes a process instance as a sequence of low-level events representing basic kinds of operations. In this context, we address a novel problem: given a description of the processes' behaviors in terms of high-level activities instead of low-level events, and in the presence of uncertainty in the mapping between events and activities, find all the interpretations of each trace @math . Specifically, an interpretation is a pair @math that provides a two-level "explanation" for @math : @math is a sequence of activities that may have triggered the events in @math , and W is a process whose model admits @math . To solve this problem, we propose a probabilistic framework representing "consistent" @math 's interpretations, where each interpretation is associated with a probability score. We consider @math Hermitian matrices with i.i.d. entries @math whose tail probabilities @math behave like @math for some @math and @math . We establish a large deviation principle for the empirical spectral measure of @math with speed @math with a good rate function @math that is finite only if @math is of the form @math for some probability measure @math on @math , where @math denotes the free convolution and @math is Wigner's semicircle law. We obtain explicit expressions for @math in terms of the @math th moment of @math . The proof is based on the analysis of large deviations for the empirical distribution of very sparse random rooted networks. Many data analysis applications deal with large matrices and involve approximating the matrix using a small number of “components.” Typically, these components are linear combinations of the rows and columns of the matrix, and are thus difficult to interpret in terms of the original features of the input data. In this paper, we propose and study matrix approximations that are explicitly expressed in terms of a small number of columns and or rows of the data matrix, and thereby more amenable to interpretation in terms of the original data. Our main algorithmic results are two randomized algorithms which take as input an @math matrix @math and a rank parameter @math . In our first algorithm, @math is chosen, and we let @math , where @math is the Moore-Penrose generalized inverse of @math . In our second algorithm @math , @math , @math are chosen, and we let @math . ( @math and @math are matrices that consist of actual columns and rows, respectively, of @math , and @math is a generalized inverse of their intersection.) For each algorithm, we show that with probability at least @math , @math , where @math is the “best” rank- @math approximation provided by truncating the SVD of @math , and where @math is the Frobenius norm of the matrix @math . The number of columns of @math and rows of @math is a low-degree polynomial in @math , @math , and @math . Both the Numerical Linear Algebra community and the Theoretical Computer Science community have studied variants of these matrix decompositions over the last ten years. However, our two algorithms are the first polynomial time algorithms for such low-rank matrix approximations that come with relative-error guarantees; previously, in some cases, it was not even known whether such matrix decompositions exist. Both of our algorithms are simple and they take time of the order needed to approximately compute the top @math singular vectors of @math . The technical crux of our analysis is a novel, intuitive sampling method we introduce in this paper called “subspace sampling.” In subspace sampling, the sampling probabilities depend on the Euclidean norms of the rows of the top singular vectors. This allows us to obtain provable relative-error guarantees by deconvoluting “subspace” information and “size-of- @math ” information in the input matrix. This technique is likely to be useful for other matrix approximation and data analysis problems. Estimation of functions of @math variables is considered using ridge combinations of the form @math where the activation function @math is a function with bounded value and derivative. These include single-hidden layer neural networks, polynomials, and sinusoidal models. From a sample of size @math of possibly noisy values at random sites @math , the minimax mean square error is examined for functions in the closure of the @math hull of ridge functions with activation @math . It is shown to be of order @math to a fractional power (when @math is of smaller order than @math ), and to be of order @math to a fractional power (when @math is of larger order than @math ). Dependence on constraints @math and @math on the @math norms of inner parameter @math and outer parameter @math , respectively, is also examined. Also, lower and upper bounds on the fractional power are given. The heart of the analysis is development of information-theoretic packing numbers for these classes of functions. We establish @math and @math error bounds for functions of many variables that are approximated by linear combinations of ReLU (rectified linear unit) and squared ReLU ridge functions with @math and @math controls on their inner and outer parameters. With the squared ReLU ridge function, we show that the @math approximation error is inversely proportional to the inner layer @math sparsity and it need only be sublinear in the outer layer @math sparsity. Our constructions are obtained using a variant of the Jones-Barron probabilistic method, which can be interpreted as either stratified sampling with proportionate allocation or two-stage cluster sampling. We also provide companion error lower bounds that reveal near optimality of our constructions. Despite the sparsity assumptions, we showcase the richness and flexibility of these ridge combinations by defining a large family of functions, in terms of certain spectral conditions, that are particularly well approximated by them. We consider the problem of private information retrieval (PIR) from @math non-colluding and replicated databases when the user is equipped with a cache that holds an uncoded fraction @math from each of the @math stored messages in the databases. We assume that the databases are unaware of the cache content. We investigate @math the optimal download cost normalized with the message size as a function of @math , @math , @math . For a fixed @math , @math , we develop an inner bound (converse bound) for the @math curve. The inner bound is a piece-wise linear function in @math that consists of @math line segments. For the achievability, we develop explicit schemes that exploit the cached bits as side information to achieve @math non-degenerate corner points. These corner points differ in the number of cached bits that are used to generate one side information equation. We obtain an outer bound (achievability) for any caching ratio by memory-sharing between these corner points. Thus, the outer bound is also a piece-wise linear function in @math that consists of @math line segments. The inner and the outer bounds match in general for the cases of very low caching ratio ( @math ) and very high caching ratio ( @math ). As a corollary, we fully characterize the optimal download cost caching ratio tradeoff for @math . For general @math , @math , and @math , we show that the largest gap between the achievability and the converse bounds is @math . Our results show that the download cost can be reduced beyond memory-sharing if the databases are unaware of the cached content. Let @math be a tree space (or tree network) represented by a weighted tree with @math vertices, and @math be a set of @math stochastic points in @math , each of which has a fixed location with an independent existence probability. We investigate two fundamental problems under such a stochastic setting, the closest-pair problem and the nearest-neighbor search. For the former, we study the computation of the @math -threshold probability and the expectation of the closest-pair distance of a realization of @math . We propose the first algorithm to compute the @math -threshold probability in @math time for any given threshold @math , which immediately results in an @math -time algorithm for computing the expected closest-pair distance. Based on this, we further show that one can compute a @math -approximation for the expected closest-pair distance in @math time, by arguing that the expected closest-pair distance can be approximated via @math threshold probability queries. For the latter, we study the @math most-likely nearest-neighbor search ( @math -LNN) via a notion called @math most-likely Voronoi Diagram ( @math -LVD). We show that the size of the @math -LVD @math of @math on @math is bounded by @math if the existence probabilities of the points in @math are constant-far from 0. Furthermore, we establish an @math average-case upper bound for the size of @math , by regarding the existence probabilities as i.i.d. random variables drawn from some fixed distribution. Our results imply the existence of an LVD data structure which answers @math -LNN queries in @math time using average-case @math space, and worst-case @math space if the existence probabilities are constant-far from 0. Finally, we also give an @math -time algorithm to construct the LVD data structure. Let @math be an @math real random matrix whose elements are independent and identically distributed standard normal random variables, and let @math be the 2-norm condition number of @math . We prove that, for any @math , @math , and @math , @math satisfies @math where @math and @math @math are universal positive constants independent of @math , @math , and @math . Moreover, for any @math and @math , @math A similar pair of results for complex Gaussian random matrices is also established. This article offers a simplified approach to the distribution theory of randomly weighted averages or @math -means @math , for a sequence of i.i.d.random variables @math , and independent random weights @math with @math and @math . The collection of distributions of @math , indexed by distributions of @math , is shown to encode Kingman's partition structure derived from @math . For instance, if @math has Bernoulli @math distribution on @math , the @math th moment of @math is a polynomial function of @math which equals the probability generating function of the number @math of distinct values in a sample of size @math from @math : @math . This elementary identity illustrates a general moment formula for @math -means in terms of the partition structure associated with random samples from @math , first developed by Diaconis and Kemperman (1996) and Kerov (1998) in terms of random permutations. As shown by Tsilevich (1997) if the partition probabilities factorize in a way characteristic of the generalized Ewens sampling formula with two parameters @math , found by Pitman (1992), then the moment formula yields the Cauchy-Stieltjes transform of an @math mean. The analysis of these random means includes the characterization of @math -means, known as Dirichlet means, due to Von Neumann (1941), Watson (1956) and Cifarelli and Regazzini (1990) and generalizations of L 'evy's arcsine law for the time spent positive by a Brownian motion, due to Darling (1949) Lamperti (1958) and Barlow, Pitman and Yor (1989). In this paper we consider a system of quadratic equations @math , where @math is unknown while normal random vectors @math and quadratic measurements @math are known. The system is assumed to be underdetermined, i.e., @math . We prove that if there exists a sparse solution @math , i.e., at most @math components of @math are nonzero, then by solving a convex optimization program, we can solve for @math up to a multiplicative constant with high probability, provided that @math . On the other hand, we prove that @math is necessary for a class of natural convex relaxations to be exact. A number of problems in probability and statistics can be addressed using the multivariate normal (Gaussian) distribution. In the one-dimensional case, computing the probability for a given mean and variance simply requires the evaluation of the corresponding Gaussian density. In the @math -dimensional setting, however, it requires the inversion of an @math covariance matrix, @math , as well as the evaluation of its determinant, @math . In many cases, such as regression using Gaussian processes, the covariance matrix is of the form @math , where @math is computed using a specified covariance kernel which depends on the data and additional parameters (hyperparameters). The matrix @math is typically dense, causing standard direct methods for inversion and determinant evaluation to require @math work. This cost is prohibitive for large-scale modeling. Here, we show that for the most commonly used covariance functions, the matrix @math can be hierarchically factored into a product of block low-rank updates of the identity matrix, yielding an @math algorithm for inversion. More importantly, we show that this factorization enables the evaluation of the determinant @math , permitting the direct calculation of probabilities in high dimensions under fairly broad assumptions on the kernel defining @math . Our fast algorithm brings many problems in marginalization and the adaptation of hyperparameters within practical reach using a single CPU core. The combination of nearly optimal scaling in terms of problem size with high-performance computing resources will permit the modeling of previously intractable problems. We illustrate the performance of the scheme on standard covariance kernels. The multiplicative coalescent @math is a @math -valued Markov process representing coalescence of clusters of mass, where each pair of clusters merges at rate proportional to product of masses. From random graph asymptotics it is known (Aldous (1997)) that there exists a standard version of this process starting with infinitesimally small clusters at time @math . In this paper, stochastic calculus techniques are used to describe all versions @math of the multiplicative coalescent. Roughly, an extreme version is specified by translation and scale parameters, and a vector @math of relative sizes of large clusters at time @math . Such a version may be characterized in three ways: via its @math behavior, via a representation of the marginal distribution @math in terms of excursion-lengths of a Levy-type process, or via a weak limit of processes derived from the standard version via a "coloring" construction. We show a parallel repetition theorem for the entangled value @math of any two-player one-round game @math where the questions @math to Alice and Bob are drawn from a product distribution on @math . We show that for the @math -fold product @math of the game @math (which represents the game @math played in parallel @math times independently), @math , where @math and @math represent the sets from which the answers of Alice and Bob are drawn. The planted partition model (also known as the stochastic blockmodel) is a classical cluster-exhibiting random graph model that has been extensively studied in statistics, physics, and computer science. In its simplest form, the planted partition model is a model for random graphs on @math nodes with two equal-sized clusters, with an between-class edge probability of @math and a within-class edge probability of @math . Although most of the literature on this model has focused on the case of increasing degrees (ie. @math as @math ), the sparse case @math is interesting both from a mathematical and an applied point of view. A striking conjecture of Decelle, Krzkala, Moore and Zdeborov 'a based on deep, non-rigorous ideas from statistical physics gave a precise prediction for the algorithmic threshold of clustering in the sparse planted partition model. In particular, if @math and @math , then conjectured that it is possible to cluster in a way correlated with the true partition if @math , and impossible if @math for some sufficiently large @math . We prove half of their prediction, showing that it is indeed impossible to cluster if @math . Following , our work establishes a rigorous connection between the clustering problem, spin-glass models on the Bethe lattice and the so called reconstruction problem. This connection points to fascinating applications and open problems. In this paper we study noisy sorting without re-sampling. In this problem there is an unknown order @math for all pairs @math , where @math is a constant and @math for all @math and @math . It is assumed that the errors are independent. Given the status of the queries the goal is to find the maximum likelihood order. In other words, the goal is find a permutation @math that minimizes the number of pairs @math where @math . The problem so defined is the feedback arc set problem on distributions of inputs, each of which is a tournament obtained as a noisy perturbations of a linear order. Note that when @math and @math is large, it is impossible to recover the original order @math . It is known that the weighted feedback are set problem on tournaments is NP-hard in general. Here we present an algorithm of running time @math and sampling complexity @math that with high probability solves the noisy sorting without re-sampling problem. We also show that if @math is an optimal solution of the problem then it is close'' to the original order. More formally, with high probability it holds that @math and @math . Our results are of interest in applications to ranking, such as ranking in sports, or ranking of search items based on comparisons by experts.