Many different methods to train deep generative models have been introduced in the past. In this paper, we propose to extend the variational auto-encoder (VAE) framework with a new type of prior which we call "Variational Mixture of Posteriors" prior, or VampPrior for short. The VampPrior consists of a mixture distribution (e.g., a mixture of Gaussians) with components given by variational posteriors conditioned on learnable pseudo-inputs. We further extend this prior to a two layer hierarchical model and show that this architecture with a coupled prior and posterior, learns significantly better models. The model also avoids the usual local optima issues related to useless latent dimensions that plague VAEs. We provide empirical studies on six datasets, namely, static and binary MNIST, OMNIGLOT, Caltech 101 Silhouettes, Frey Faces and Histopathology patches, and show that applying the hierarchical VampPrior delivers state-of-the-art results on all datasets in the unsupervised permutation invariant setting and the best results or comparable to SOTA methods for the approach with convolutional networks.
Very recently, the VampPrior was shown to be a perfect match for the information-theoretic approach to learn latent representation @cite_27 . Additionally, the authors of @cite_27 proposed to use a weighted version of the VampPrior: where @math are trainable parameters such that @math and @math . This allows the VampPrior to learn which components (, pseudo-inputs) are meaningful and may prevent from potential overfitting.
@cite_27
2766227112
Many different methods to train deep generative models have been introduced in the past. In this paper, we propose to extend the variational auto-encoder (VAE) framework with a new type of prior which we call "Variational Mixture of Posteriors" prior, or VampPrior for short. The VampPrior consists of a mixture distribution (e.g., a mixture of Gaussians) with components given by variational posteriors conditioned on learnable pseudo-inputs. We further extend this prior to a two layer hierarchical model and show that this architecture with a coupled prior and posterior, learns significantly better models. The model also avoids the usual local optima issues related to useless latent dimensions that plague VAEs. We provide empirical studies on six datasets, namely, static and binary MNIST, OMNIGLOT, Caltech 101 Silhouettes, Frey Faces and Histopathology patches, and show that applying the hierarchical VampPrior delivers state-of-the-art results on all datasets in the unsupervised permutation invariant setting and the best results or comparable to SOTA methods for the approach with convolutional networks. An arithmetic read-once formula (ROF for short) is a formula (a circuit whose underlying graph is a tree) in which the operations are + ,× and such that every input variable labels at most one leaf. In this paper we study the problems of giving deterministic identity testing and reconstruction algorithms for ROFs. Our main result is an @math time deterministic algorithm for checking whether a black box holding the sum of k n -variate ROFs computes the zero polynomial. In other words, we provide a hitting set of size @math for the sum of k ROFs. This result greatly improves [27] where an @math algorithm was given for the problem. Using our new results we obtain a deterministic reconstruction algorithms for read-once formulas that runs in time @math . In fact, our results also hold for the more general model of preprocessed read-once formulas that we define in this paper. In this model we are allowed to replace each variable x i with a polynomial T i (x i ). Our techniques are very close to the techniques in [27]. The main difference is that we obtain several tighter versions of the tools first used there. In particular we obtain a better version of the hardness of representation approach which was first used in [27]. This technique can be thought of as a very explicit way of transforming (mild) hardness of a very structured polynomial to an identity testing algorithm. We study the flow of information and the evolution of internal representations during deep neural network (DNN) training, aiming to demystify the compression aspect of the information bottleneck theory. The theory suggests that DNN training comprises a rapid fitting phase followed by a slower compression phase, in which the mutual information @math between the input @math and internal representations @math decreases. Several papers observe compression of estimated mutual information on different DNN models, but the true @math over these networks is provably either constant (discrete @math ) or infinite (continuous @math ). This work explains the discrepancy between theory and experiments, and clarifies what was actually measured by these past works. To this end, we introduce an auxiliary (noisy) DNN framework for which @math is a meaningful quantity that depends on the network's parameters. This noisy framework is shown to be a good proxy for the original (deterministic) DNN both in terms of performance and the learned representations. We then develop a rigorous estimator for @math in noisy DNNs and observe compression in various models. By relating @math in the noisy DNN to an information-theoretic communication problem, we show that compression is driven by the progressive clustering of hidden representations of inputs from the same class. Several methods to directly monitor clustering of hidden representations, both in noisy and deterministic DNNs, are used to show that meaningful clusters form in the @math space. Finally, we return to the estimator of @math employed in past works, and demonstrate that while it fails to capture the true (vacuous) mutual information, it does serve as a measure for clustering. This clarifies the past observations of compression and isolates the geometric clustering of hidden representations as the true phenomenon of interest. We investigate the minimization of Newton's functional for the problem of the body of minimal resistance with maximal height @math cite butt in the class of convex developable functions defined in a disc. This class is a natural candidate to find a (non-radial) minimizer in accordance with the results of cite lrp2. We prove that the minimizer in this class has a minimal set in the form of a regular polygon with @math sides centered in the disc, and numerical experiments indicate that the natural number @math is a non-decreasing function of @math . The corresponding functions all achieve a lower value of the functional than the optimal radially symmetric function with the same height @math . A major challenge in training deep neural networks is overfitting, i.e. inferior performance on unseen test examples compared to performance on training examples. To reduce overfitting, stochastic regularization methods have shown superior performance compared to deterministic weight penalties on a number of image recognition tasks. Stochastic methods such as Dropout and Shakeout, in expectation, are equivalent to imposing a ridge and elastic-net penalty on the model parameters, respectively. However, the choice of the norm of weight penalty is problem dependent and is not restricted to @math . Therefore, in this paper we propose the Bridgeout stochastic regularization technique and prove that it is equivalent to an @math penalty on the weights, where the norm @math can be learned as a hyperparameter from data. Experimental results show that Bridgeout results in sparse model weights, improved gradients and superior classification performance compared to Dropout and Shakeout on synthetic and real datasets. We present a probabilistic algorithm that, given a connected graph G (represented by adjacency lists) of average degree d, with edge weights in the set 1,...,w , and given a parameter @math , estimates in time @math the weight of the minimum spanning tree (MST) of G with a relative error of at most @math . Note that the running time does not depend on the number of vertices in G. We also prove a nearly matching lower bound of @math on the probe and time complexity of any approximation algorithm for MST weight. The essential component of our algorithm is a procedure for estimating in time @math the number of connected components of an unweighted graph to within an additive error of @math . (This becomes @math for @math .) The time bound is shown to be tight up to within the @math factor. Our connected-components algorithm picks @math vertices in the graph and then grows "local spanning trees" whose sizes are specified by a stochastic process. From the local information collected in this way, the algorithm is able to infer, with high confidence, an estimate of the number of connected components. We then show how estimates on the number of components in various subgraphs of G can be used to estimate the weight of its MST. We present the first formal mathematical presentation of the generalized Russian cards problem, and provide rigorous security definitions that capture both basic and extended versions of weak and perfect security notions. In the generalized Russian cards problem, three players, Alice, Bob, and Cathy, are dealt a deck of @math cards, each given @math , @math , and @math cards, respectively. The goal is for Alice and Bob to learn each other's hands via public communication, without Cathy learning the fate of any particular card. The basic idea is that Alice announces a set of possible hands she might hold, and Bob, using knowledge of his own hand, should be able to learn Alice's cards from this announcement, but Cathy should not. Using a combinatorial approach, we are able to give a nice characterization of informative strategies (i.e., strategies allowing Bob to learn Alice's hand), having optimal communication complexity, namely the set of possible hands Alice announces must be equivalent to a large set of @math -designs, where @math . We also provide some interesting necessary conditions for certain types of deals to be simultaneously informative and secure. That is, for deals satisfying @math for some @math , where @math and the strategy is assumed to satisfy a strong version of security (namely perfect @math -security), we show that @math and hence @math . We also give a precise characterization of informative and perfectly @math -secure deals of the form @math satisfying @math involving @math -designs. We present a novel approximation algorithm for @math -median that achieves an approximation guarantee of @math , improving upon the decade-old ratio of @math . Our improved approximation ratio is achieved by exploiting the power of pseudo-approximation. More specifically, our approach is based on two components, each of which, we believe, is of independent interest. First, we show that in order to give an @math -approximation algorithm for @math -median, it is sufficient to give a pseudo-approximation algorithm that finds an @math -approximate solution by opening @math facilities. This is a rather surprising result as there exist instances for which opening @math facilities may lead to a significantly smaller cost than that of opening only @math facilities. Second, we give such a pseudo-approximation algorithm with @math . Prior to our work, it was not even known whether opening @math facilities would help improve the approximation ratio. We consider the maximum independent set problem on graphs with maximum degree @math . We show that the integrality gap of the Lov 'asz @math -function based SDP is @math . This improves on the previous best result of @math , and almost matches the integrality gap of @math recently shown for stronger SDPs, namely those obtained using poly- @math levels of the @math semidefinite hierarchy. The improvement comes from an improved Ramsey-theoretic bound on the independence number of @math -free graphs for large values of @math . We also show how to obtain an algorithmic version of the above-mentioned @math -based integrality gap result, via a coloring algorithm of Johansson. The resulting approximation guarantee of @math matches the best unique-games-based hardness result up to lower-order poly- @math factors. There are many classical problems in P whose time complexities have not been improved over the past decades. Recent studies of "Hardness in P" have revealed that, for several of such problems, the current fastest algorithm is the best possible under some complexity assumptions. To bypass this difficulty, (SODA 2017) introduced the concept of fully polynomial FPT algorithms. For a problem with the current best time complexity @math , the goal is to design an algorithm running in @math time for a parameter @math and a constant @math . In this paper, we investigate the complexity of graph problems in P parameterized by tree-depth, a graph parameter related to tree-width. We show that a simple divide-and-conquer method can solve many graph problems, including Weighted Matching, Negative Cycle Detection, Minimum Weight Cycle, Replacement Paths, and 2-hop Cover, in @math time or @math time, where @math is the tree-depth of the input graph. Because any graph of tree-width @math has tree-depth at most @math , our algorithms also run in @math time or @math time. These results match or improve the previous best algorithms parameterized by tree-width. Especially, we solve an open problem of fully polynomial FPT algorithm for Weighted Matching parameterized by tree-width posed by The full' edge isoperimetric inequality for the discrete cube (due to Harper, Bernstein, Lindsay and Hart) specifies the minimum size of the edge boundary @math of a set @math , as a function of @math . A weaker (but more widely-used) lower bound is @math , where equality holds iff @math is a subcube. In 2011, the first author obtained a sharp stability' version of the latter result, proving that if @math , then there exists a subcube @math such that @math . The weak' version of the edge isoperimetric inequality has the following well-known generalization for the @math -biased' measure @math on the discrete cube: if @math , or if @math and @math is monotone increasing, then @math . In this paper, we prove a sharp stability version of the latter result, which generalizes the aforementioned result of the first author. Namely, we prove that if @math , then there exists a subcube @math such that @math , where @math . This result is a central component in recent work of the authors proving sharp stability versions of a number of Erd o s-Ko-Rado type theorems in extremal combinatorics, including the seminal complete intersection theorem' of Ahlswede and Khachatrian. In addition, we prove a biased-measure analogue of the full' edge isoperimetric inequality, for monotone increasing sets, and we observe that such an analogue does not hold for arbitrary sets, hence answering a question of Kalai. We use this result to give a new proof of the full' edge isoperimetric inequality, one relying on the Kruskal-Katona theorem. Friedland's Lower Matching Conjecture asserts that if @math is a @math --regular bipartite graph on @math vertices, and @math denotes the number of matchings of size @math , then @math where @math . When @math , this conjecture reduces to a theorem of Schrijver which says that a @math --regular bipartite graph on @math vertices has at least @math perfect matchings. L. Gurvits proved an asymptotic version of the Lower Matching Conjecture, namely he proved that @math In this paper, we prove the Lower Matching Conjecture. In fact, we will prove a slightly stronger statement which gives an extra @math factor compared to the conjecture if @math is separated away from @math and @math , and is tight up to a constant factor if @math is separated away from @math . We will also give a new proof of Gurvits's and Schrijver's theorems, and we extend these theorems to @math --biregular bipartite graphs. Let @math be a set of elements and d a distance function defined among them. Let NNk(u) be the k elements in @math having the smallest distance to u. The k-nearest neighbor graph (knng) is a weighted directed graph @math such that E= (u,v), v∈NNk(u) . Several knng construction algorithms are known, but they are not suitable to general metric spaces. We present a general methodology to construct knngs that exploits several features of metric spaces. Experiments suggest that it yields costs of the form c1n1.27 distance computations for low and medium dimensional spaces, and c2n1.90 for high dimensional ones. In [Proceedings of the Second Symposium on Switching Circuit Theory and Logical Design (FOCS), 1961, pp. 34-38], Chow proved that every Boolean threshold function is uniquely determined by its degree-0 and degree-1 Fourier coefficients. These numbers became known as the Chow parameters. Providing an algorithmic version of Chow's theorem—i.e., efficiently constructing a representation of a threshold function given its Chow parameters—has remained open ever since. This problem has received significant study in the fields of circuit complexity, game theory and the design of voting systems, and learning theory. In this paper we effectively solve the problem, giving a randomized polynomial-time approximation scheme with the following behavior: Given the Chow parameters of a Boolean threshold function @math over @math bits and any constant @math , the algorithm runs in time @math and with high probability outputs a representation of a threshold function @math which is @math -close to @math . Along the way we prove several new results of independent interest about Boolean threshold functions. In addition to various structural results, these include @math -time learning algorithms for threshold functions under the uniform distribution in the following models: (i) the restricted focus of attention model, answering an open question of ; (ii) an agnostic-type model. This contrasts with recent results of Guruswami and Raghavendra who show NP-hardness for the problem under general distributions; (iii) the PAC model, with constant @math . Our @math -time algorithm substantially improves on the previous best known running time and nearly matches the @math bits of training data that any successful learning algorithm must use. A theorem of A. Schrijver asserts that a d-regular bipartite graph on 2n vertices has at least @math perfect matchings. L. Gurvits gave an extension of Schrijver’s theorem for matchings of density p. In this paper we give a stronger version of Gurvits’s theorem in the case of vertex-transitive bipartite graphs. This stronger version in particular implies that for every positive integer k, there exists a positive constant c(k) such that if a d-regular vertex-transitive bipartite graph on 2n vertices contains a cycle of length at most k, then it has at least @math perfect matchings. We consider the @math -Center problem and some generalizations. For @math -Center a set of @math center vertices needs to be found in a graph @math with edge lengths, such that the distance from any vertex of @math to its nearest center is minimized. This problem naturally occurs in transportation networks, and therefore we model the inputs as graphs with bounded highway dimension, as proposed by [SODA 2010]. We show both approximation and fixed-parameter hardness results, and how to overcome them using fixed-parameter approximations, where the two paradigms are combined. In particular, we prove that for any @math computing a @math -approximation is W[2]-hard for parameter @math and NP-hard for graphs with highway dimension @math . The latter does not rule out fixed-parameter @math -approximations for the highway dimension parameter @math , but implies that such an algorithm must have at least doubly exponential running time in @math if it exists, unless the ETH fails. On the positive side, we show how to get below the approximation factor of @math by combining the parameters @math and @math : we develop a fixed-parameter @math -approximation with running time @math . Additionally we prove that, unless P=NP, our techniques cannot be used to compute fixed-parameter @math -approximations for only the parameter @math . We also provide similar fixed-parameter approximations for the weighted @math -Center and @math -Partition problems, which generalize @math -Center. The global convergence properties of a class of penalty methods for nonlinear programming are analyzed. These methods include successive linear programming approaches and, more specifically, the successive linear-quadratic programming approach presented by [Math. Program., 100 (2004), pp. 27--48]. Every iteration requires the solution of two trust-region subproblems involving piecewise linear and quadratic models, respectively. It is shown that, for a fixed penalty parameter, the sequence of iterates approaches stationarity of the penalty function. A procedure for dynamically adjusting the penalty parameter is described, and global convergence results for it are established. In a document network such as a citation network of scientific documents, web-logs, etc., the content produced by authors exhibits their interest in certain topics. In addition some authors influence other authors' interests. In this work, we propose to model the influence of cited authors along with the interests of citing authors. Moreover, we hypothesize that apart from the citations present in documents, the context surrounding the citation mention provides extra topical information about the cited authors. However, associating terms in the context to the cited authors remains an open problem. We propose novel document generation schemes that incorporate the context while simultaneously modeling the interests of citing authors and influence of the cited authors. Our experiments show significant improvements over baseline models for various evaluation criteria such as link prediction between document and cited author, and quantitatively explaining unseen text. This paper builds upon the basic theory of multirate systems for graph signals developed in the companion paper (Part I) and studies @math -channel polynomial filter banks on graphs. The behavior of such graph filter banks differs from that of classical filter banks in many ways, the precise details depending on the eigenstructure of the adjacency matrix @math . It is shown that graph filter banks represent, (linear and) periodically, shift-variant systems only when @math satisfies the noble identity conditions developed in Part I. It is then shown that perfect reconstruction graph filter banks can always be developed when @math satisfies the eigenvector structure satisfied by @math -block cyclic graphs and has distinct eigenvalues (further restrictions on eigenvalues being unnecessary for this). If @math is actually @math -block cyclic then these PR filter banks indeed become practical, i.e., arbitrary filter polynomial orders are possible, and there are robustness advantages. In this case, the PR condition is identical to PR in classical filter banks—any classical PR example can be converted to a graph PR filter bank on an @math -block cyclic graph. It is shown that for @math -block cyclic graphs with all eigenvalues on the unit circle, the frequency responses of filters have meaningful correspondence with classical filter banks. Polyphase representations are then developed for graph filter banks and utilized to develop alternate conditions for alias cancellation and perfect reconstruction, again for graphs with specific eigenstructures. It is then shown that the eigenvector condition on the graph can be relaxed by using similarity transforms. Unsupervised Image-to-Image Translation achieves spectacularly advanced developments nowadays. However, recent approaches mainly focus on one model with two domains, which may face heavy burdens with large cost of @math training time and model parameters, under such a requirement that @math domains are freely transferred to each other in a general setting. To address this problem, we propose a novel and unified framework named Domain-Bank, which consists of a global shared auto-encoder and @math domain-specific encoders decoders, assuming that a universal shared-latent sapce can be projected. Thus, we yield @math complexity in model parameters along with a huge reduction of the time budgets. Besides the high efficiency, we show the comparable (or even better) image translation results over state-of-the-arts on various challenging unsupervised image translation tasks, including face image translation, fashion-clothes translation and painting style translation. We also apply the proposed framework to domain adaptation and achieve state-of-the-art performance on digit benchmark datasets. Further, thanks to the explicit representation of the domain-specific decoders as well as the universal shared-latent space, it also enables us to conduct incremental learning to add a new domain encoder decoder. Linear combination of different domains' representations is also obtained by fusing the corresponding decoders.