This paper presents the recurrent estimation of distributions (RED) for modeling real-valued data in a semiparametric fashion. RED models make two novel uses of recurrent neural networks (RNNs) for density estimation of general real-valued data. First, RNNs are used to transform input covariates into a latent space to better capture conditional dependencies in inputs. After, an RNN is used to compute the conditional distributions of the latent covariates. The resulting model is efficient to train, compute, and sample from, whilst producing normalized pdfs. The effectiveness of RED is shown via several real-world data experiments. Our results show that RED models achieve a lower held-out negative log-likelihood than other neural network approaches across multiple dataset sizes and dimensionalities. Further context of the efficacy of RED is provided by considering anomaly detection tasks, where we also observe better performance over alternative models.
NICE models assume that data is drawn from a latent independent Gaussian space and transformed @cite_19 . The transformation uses several additive coupling'' shifting transformations on the second half of dimensions, using the first half of dimensions. The full transformation is the result of stacking several of these additive coupling layers together followed by a final rescaling operation. The resulting composition of these transformation yields a simple determinant of the Jacobian.
@cite_19
1583912456
We propose a deep learning framework for modeling complex high-dimensional densities called Non-linear Independent Component Estimation (NICE). It is based on the idea that a good representation is one in which the data has a distribution that is easy to model. For this purpose, a non-linear deterministic transformation of the data is learned that maps it to a latent space so as to make the transformed data conform to a factorized distribution, i.e., resulting in independent latent variables. We parametrize this transformation so that computing the Jacobian determinant and inverse transform is trivial, yet we maintain the ability to learn complex non-linear transformations, via a composition of simple building blocks, each based on a deep neural network. The training criterion is simply the exact log-likelihood, which is tractable. Unbiased ancestral sampling is also easy. We show that this approach yields good generative models on four image datasets and can be used for inpainting. The framework of normalizing flows provides a general strategy for flexible variational inference of posteriors over latent variables. We propose a new type of normalizing flow, inverse autoregressive flow (IAF), that, in contrast to earlier published flows, scales well to high-dimensional latent spaces. The proposed flow consists of a chain of invertible transformations, where each transformation is based on an autoregressive neural network. In experiments, we show that IAF significantly improves upon diagonal Gaussian approximate posteriors. In addition, we demonstrate that a novel type of variational autoencoder, coupled with IAF, is competitive with neural autoregressive models in terms of attained log-likelihood on natural images, while allowing significantly faster synthesis. We propose a powerful video denoising algorithm that exploits temporal and spatial redundancy characterizing natural video sequences. The algorithm implements the paradigm of nonlocal grouping and collaborative filtering, where a higher-dimensional transform-domain representation is leveraged to enforce sparsity and thus regularize the data. The proposed algorithm exploits the mutual similarity between 3-D spatiotemporal volumes constructed by tracking blocks along trajectories defined by the motion vectors. Mutually similar volumes are grouped together by stacking them along an additional fourth dimension, thus producing a 4-D structure, termed group, where different types of data correlation exist along the different dimensions: local correlation along the two dimensions of the blocks, temporal correlation along the motion trajectories, and nonlocal spatial correlation (i.e. self-similarity) along the fourth dimension. Collaborative filtering is realized by transforming each group through a decorrelating 4-D separable transform and then by shrinkage and inverse transformation. In this way, collaborative filtering provides estimates for each volume stacked in the group, which are then returned and adaptively aggregated to their original position in the video. Experimental results demonstrate the effectiveness of the proposed procedure which outperforms the state of the art. A method of calculating representative forms from a given set of forms was developed, in which surface data is modeled by polygons based on landmarks. Inter-individual distances are defined as distortions in FFD control points. By calculating inter-individual distances for all possible pairs of forms, the distribution of the 3D forms in m-dimensional space is obtained using MDS. Each MDS dimension represents an independent shape factor. Forms with specific MDS scores, such as (0.5,0,0,0), (1,0,0,0) in standard deviation units, are calculated as weighted averages of all actual forms. An FFD transformation grid is calculated that represents the systematic form transformation along an MDS dimension. Forms with different scores for the first or second MDS dimensions only and average scores (=0) for the other MDS dimensions are calculated using these transformation grids. Abstract This paper examines the application of linear transformations for speaker and environmental adaptation in an HMM-based speech recognition system. In particular, transformations that are trained in a maximum likelihood sense on adaptation data are investigated. Only model-based linear transforms are considered, since, for linear transforms, they subsume the appropriate feature–space transforms. The paper compares the two possible forms of model-based transforms: (i) unconstrained, where any combination of mean and variance transform may be used, and (ii) constrained, which requires the variance transform to have the same form as the mean transform. Re-estimation formulae for all appropriate cases of transform are given. This includes a new and efficient full variance transform and the extension of the constrained model–space transform from the simple diagonal case to the full or block–diagonal case. The constrained and unconstrained transforms are evaluated in terms of computational cost, recognition time efficiency, and use for speaker adaptive training. The recognition performance of the two model–space transforms on a large vocabulary speech recognition task using incremental adaptation is investigated. In addition, initial experiments using the constrained model–space transform for speaker adaptive training are detailed. In this study, we introduce a new image classification technique using decision fusion. The proposed technique, called Meta-Fuzzified Yield Value (Meta-FYV), is based on two-layer Stacked Generalization (SG) architecture [1]. At the base-layer, the system, receives a set of feature vectors of various dimensions and dynamical ranges and outputs hypotheses through fuzzy transformations. Then, the hypotheses created by the base layer transformations are concatenated for building a regression equation at meta-layer. Experimental evidence indicates that the Meta-FYV is superior compared to one of the most successful Fuzzy SG methods, introduced by Akbas [2]. The Radon transform of a bivariate function, which has application in tomographic imaging, has traditionally been viewed as a parametrized univariate function. In this paper, the Radon transform is instead viewed as a bivariate function and two-dimensional sampling theory is used to address sampling and information content issues. It is Shown that the band region of the Radon transform of a function with a finite space-bandwidth product is a "finite-length bowtie." Because of the special shape of this band region. "Nyquist sampling" of the Radon transform is on a hexagonal grid. This sampling grid requires approximately one-half as many samples as the rectangular grid obtained from the traditional viewpoint. It is also shown that for a nonbandlimited function of finite spatial support, the bandregion of the Radon transform is an "infinite-length bowtie." Consequently, it follows that approximately 2M2 π independent pieces of information about the function can be extracted from M "projections." These results and others follow very naturally from the two-dimensional viewpoint presented. We propose a powerful video filtering algorithm that exploits temporal and spatial redundancy characterizing natural video sequences. The algorithm implements the paradigm of nonlocal grouping and collaborative filtering, where a higher dimensional transform-domain representation of the observations is leveraged to enforce sparsity, and thus regularize the data: 3-D spatiotemporal volumes are constructed by tracking blocks along trajectories defined by the motion vectors. Mutually similar volumes are then grouped together by stacking them along an additional fourth dimension, thus producing a 4-D structure, termed group, where different types of data correlation exist along the different dimensions: local correlation along the two dimensions of the blocks, temporal correlation along the motion trajectories, and nonlocal spatial correlation (i.e., self-similarity) along the fourth dimension of the group. Collaborative filtering is then realized by transforming each group through a decorrelating 4-D separable transform and then by shrinkage and inverse transformation. In this way, the collaborative filtering provides estimates for each volume stacked in the group, which are then returned and adaptively aggregated to their original positions in the video. The proposed filtering procedure addresses several video processing applications, such as denoising, deblocking, and enhancement of both grayscale and color data. Experimental results prove the effectiveness of our method in terms of both subjective and objective visual quality, and show that it outperforms the state of the art in video denoising. A first transistor responsive to a source of signals is coupled to a first winding of a transformer and further coupled to a gate control electrode of a semiconductor device for turning the device off in response to the signals. A second winding of the transformer is coupled to a second transistor which is also coupled to the gate control electrode such that a change from a first to a second conduction state of the first transistor causes a current to be induced in the second winding which changes the conduction state of the second transistor for enabling conduction of the gated semiconductor device. Geometric transformations are most commonly represented as square matrices in computer graphics. Following simple geometric arguments we derive a natural and geometrically meaningful definition of scalar multiples and a commutative addition of transformations based on the matrix representation, given that the matrices have no negative real eigenvalues. Together, these operations allow the linear combination of transformations. This provides the ability to create weighted combination of transformations, interpolate between transformations, and to construct or use arbitrary transformations in a structure similar to a basis of a vector space. These basic techniques are useful for synthesis and analysis of motions or animations. Animations through a set of key transformations are generated using standard techniques such as subdivision curves. For analysis and progressive compression a PCA can be applied to sequences of transformations. We describe an implementation of the techniques that enables an easy-to-use and transparent way of dealing with geometric transformations in graphics software. We compare and relate our approach to other techniques such as matrix decomposition and quaternion interpolation. We introduce the scale axis transform, a new skelet al shape representation for bounded open sets O ⊂ Rd. The scale axis transform induces a family of skeletons that captures the important features of a shape in a scale-adaptive way and yields a hierarchy of successively simplified skeletons. Its definition is based on the medial axis transform and the simplification of the shape under multiplicative scaling: the s-scaled shape Os is the union of the medial balls of O with radii scaled by a factor of s. The s-scale axis transform of O is the medial axis transform of Os, with radii scaled back by a factor of 1 s. We prove topological properties of the scale axis transform and we describe the evolution s → Os by defining the multiplicative distance function to the shape and studying properties of the corresponding steepest ascent flow. All our theoretical results hold for any dimension. In addition, using a discrete approximation, we present several examples of two-dimensional scale axis transforms that illustrate the practical relevance of our new framework. We address an open question, regarding whether a lattice code with lattice decoding (as opposed to maximum-likelihood (ML) decoding) can achieve the additive white Gaussian noise (AWGN) channel capacity. We first demonstrate how minimum mean-square error (MMSE) scaling along with dithering (lattice randomization) techniques can transform the power-constrained AWGN channel into a modulo-lattice additive noise channel, whose effective noise is reduced by a factor of spl radic (1+SNR SNR). For the resulting channel, a uniform input maximizes mutual information, which in the limit of large lattice dimension becomes 1 2 log (1+SNR), i.e., the full capacity of the original power constrained AWGN channel. We then show that capacity may also be achieved using nested lattice codes, the coarse lattice serving for shaping via the modulo-lattice transformation, the fine lattice for channel coding. We show that such pairs exist for any desired nesting ratio, i.e., for any signal-to-noise ratio (SNR). Furthermore, for the modulo-lattice additive noise channel lattice decoding is optimal. Finally, we show that the error exponent of the proposed scheme is lower bounded by the Poltyrev exponent. We consider a wavelet neural network approach for electricity load prediction. The wavelet transform is used to decompose the load into different frequency components that are predicted separately using neural networks. We firstly propose a new approach for signal extension which minimizes the border distortion when decomposing the data, outperforming three standard methods. We also compare the performance of the standard wavelet transform, which is shift variant, with a non-decimated transform, which is shift invariant. Our results show that the use of shift invariant transform considerably improves the prediction accuracy. In addition to wavelet neural network, we also present the results of wavelet linear regression, wavelet model trees and a number of baselines. Our evaluation uses two years of Australian electricity data. This paper presents a new approach for face recognition based on the fusion of tensors of census transform histograms from Local Gaussian features maps. Local Gaussian feature maps encode the most relevant information from Gaussian derivative features. Census Transform (CT) histograms are calculated and concatenated to form a tensor for each class of Gaussian map. Multi-linear Principal Component Analysis (MPCA) is applied to each tensor to reduce the number of dimensions as well as the correlation between neighboring pixels due to the Census Transform. We then train Kernel Discriminative Common Vectors (KDCV) to generate a discriminative vector using the results of the MPCA. Results of recognition using MPCA of tensors-CT histograms from Gaussian features maps with KDCV is shown to compare favorably with competing techniques that use more complex features maps like for example Gabor features maps in the FERET and Yale datasets. Additional experiments were done in the Yale B+ extended Yale B Faces dataset to show the performance of Gaussian features map with hard illumination changes. Instantaneous frequency (IF) is necessary for understanding the detailed mechanisms for nonlinear and nonstationary processes. Historically, IF was computed from analytic signal (AS) through the Hilbert transform. This paper offers an overview of the difficulties involved in using AS, and two new methods to overcome the difficulties for computing IF. The first approach is to compute the quadrature (defined here as a simple 90° shift of phase angle) directly. The second approach is designated as the normalized Hilbert transform (NHT), which consists of applying the Hilbert transform to the empirically determined FM signals. Additionally, we have also introduced alternative methods to compute local frequency, the generalized zero-crossing (GZC), and the teager energy operator (TEO) methods. Through careful comparisons, we found that the NHT and direct quadrature gave the best overall performance. While the TEO method is the most localized, it is limited to data from linear processes, the GZC method is the m... In previous work on "transformed mixtures of Gaussians" and "transformed hidden Markov models", we showed how the EM algorithm in a discrete latent variable model can be used to jointly normalize data (e.g., center images, pitch-normalize spectrograms) and learn a mixture model of the normalized data. The only input to the algorithm is the data, a list of possible transformations, and the number of clusters to find. The main criticism of this work was that the exhaustive computation of the posterior probabilities over transformations would make scaling up to large feature vectors and large sets of transformations intractable. Here, we describe how a tremendous speed-up is acheived through the use of a variational technique for decoupling transformations, and a fast Fourier transform method for computing posterior probabilities. For N × N images, learning C clusters under N rotations, N scales, N x-translations and N y-translations takes only (C + 2 log N)N2 scalar operations per iteration. In contrast, the original algorithm takes CN6 operations to account for these transformations. We give results on learning a 4-component mixture model from a video sequence with frames of size 320×240. The model accounts for 360 rotations and 76,800 translations. Each iteration of EM takes only 10 seconds per frame in MATLAB, which is over 5 million times faster than the original algorithm. Abstract A general analysis of exact nonlinear lumping is presented. This analysis can be applied to the kinetics of any reaction system with n species described by a set of first-order ordinary differential equations d y d t = f ( y ), where y is an n -dimensional vector and f ( y ) is an arbitrary n -dimensional function vector. We consider lumping by means of n ( n ⩽ n )-dimensional arbitrary transformation ŷ = h ( y ). The lumped differential equation system is d y D t = y ( h (ŷ))f( h (ŷ)) , where h y (y) is teh Jacobian matrix of h(y) , h is a generalized inverse transformation of h satisfying the relation h( h ) = I n . Three necessary and sufficient conditions of the existence of exact nonlinear lumping schemes have been determined. The geometric and algebraic interpretations of these conditions are discussed. It is found that a system is exactly lumpable by h only if h(y) = 0 is its invariant manifold. A linear partial differential operator A = Σ n i =1 f i ( y )ϑ ϑ y i corresponding to d y d t = f(y ) is defined. Using the eigenfunctions and the generalized eigenfunctions of A , the operator can be transformed to Jordan or diagonal canonical forms which give the lumped differential equation systems without determination of h . These approaches are illustrated by a simple example. The results of this analysis serve as a theoretical basis for the development of approaches for approximate nonlinear lumping. Abstract Being motivated by the multilayer RECOS (REctified-COrrelations on a Sphere) transform, we develop a data-driven Saak (Subspace approximation with augmented kernels) transform in this work. The Saak transform consists of three steps: (1) building the optimal linear subspace approximation with orthonormal bases using the second-order statistics of input vectors, (2) augmenting each transform kernel with its negative, (3) applying the rectified linear unit (ReLU) to the transform output. The Karhunen-Loeve transform (KLT) is used in the first step. The integration of Steps 2 and 3 is powerful since they resolve the sign confusion problem, remove the rectification loss and allow a straightforward implementation of the inverse Saak transform at the same time. Multiple Saak transforms are cascaded to transform images of a larger size. All Saak transform kernels are derived from the second-order statistics of input random vectors in a one-pass feedforward manner. Neither data labels nor backpropagation is used in kernel determination. Multi-stage Saak transforms offer a family of joint spatial-spectral representations between two extremes; namely, the full spatial-domain representation and the full spectral-domain representation. We select Saak coefficients of higher discriminant power to form a feature vector for pattern recognition, and use the MNIST dataset classification problem as an illustrative example. Fourier PCA is Principal Component Analysis of a matrix obtained from higher order derivatives of the logarithm of the Fourier transform of a distribution.We make this method algorithmic by developing a tensor decomposition method for a pair of tensors sharing the same vectors in rank- @math decompositions. Our main application is the first provably polynomial-time algorithm for underdetermined ICA, i.e., learning an @math matrix @math from observations @math where @math is drawn from an unknown product distribution with arbitrary non-Gaussian components. The number of component distributions @math can be arbitrarily higher than the dimension @math and the columns of @math only need to satisfy a natural and efficiently verifiable nondegeneracy condition. As a second application, we give an alternative algorithm for learning mixtures of spherical Gaussians with linearly independent means. These results also hold in the presence of Gaussian noise. Down-sampling is widely adopted in deep convolutional neural networks (DCNN) for reducing the number of network parameters while preserving the transformation invariance. However, it cannot utilize information effectively because it only adopts a fixed stride strategy, which may result in poor generalization ability and information loss. In this paper, we propose a novel random strategy to alleviate these problems by embedding random shifting in the down-sampling layers during the training process. Random shifting can be universally applied to diverse DCNN models to dynamically adjust receptive fields by shifting kernel centers on feature maps in different directions. Thus, it can generate more robust features in networks and further enhance the transformation invariance of down-sampling operators. In addition, random shifting cannot only be integrated in all down-sampling layers including strided convolutional layers and pooling layers, but also improve performance of DCNN with negligible additional computational cost. We evaluate our method in different tasks (e.g., image classification and segmentation) with various network architectures (i.e., AlexNet, FCN and DFNMR). Experimental results demonstrate the effectiveness of our proposed method.