Similarity metrics are a core component of many information retrieval and machine learning systems. In this work we propose a method capable of learning a similarity metric from data equipped with a binary relation. By considering only the similarity constraints, and initially ignoring the features, we are able to learn target vectors for each instance using one of several appropriately designed loss functions. A regression model can then be constructed that maps novel feature vectors to the same target vector space, resulting in a feature extractor that computes vectors for which a predefined metric is a meaningful measure of similarity. We present results on both multiclass and multi-label classification datasets that demonstrate considerably faster convergence, as well as higher accuracy on the majority of the intrinsic evaluation tasks and all extrinsic evaluation tasks.
There is also an extension of NCA to nonlinear transformations of the input data @cite_16 . This method can be viewed as a probabilistic variant of Siamese networks. The nonlinear transformation models used in the original exposition of this method were stacked Restricted Boltzmann Machines, initialised using unsupervised pretraining and subsequently fine-tuned using the NCA loss function.
@cite_16
1504066675 2949474319 2964055354 2687693326
Nowadays model based techniques play very important role in solving measurement and control problems. Recently for representing nonlinear systems fuzzy and neural network (NN) models became very popular. For evaluating measurement data and for controller design also the inverse models are of considerable interest. In this paper, different observer based techniques to perform fuzzy and neural network model inversion are presented. The methods are based on solving a nonlinear equation derived from the multiple-input single-output (MISO) forward fuzzy model simple by interchanging the role of the output and one of the inputs. The utilization of the inverse model can be either a direct compensation of some measurement nonlinearities or a controller mechanism for nonlinear plants. For discrete-time inputs the technique provides good performance if the iterative inversion is fast enough compared to system variations, i.e., the iteration is convergent within the sampling period applied. The proposed method can be considered also as a simple nonlinear state observer which reconstructs the selected input of the forward (fuzzy or NN) model from its output using an appropriate strategy and a copy of the fuzzy or neural network model itself. Improved performance can be obtained by introducing genetic algorithms in the prediction-correction mechanism. Although, the overall performance of the suggested technique is highly influenced by the nature of the non-linearity and the actual prediction-correction mechanism applied, it can also be shown that using this observer concept completely inverted models can be derived. The inversion can be extended towards anytime modes of operation, as well, providing short response time and flexibility during temporal loss of computational power and or time. The model parameters of convolutional neural networks (CNNs) are determined by backpropagation (BP). In this work, we propose an interpretable feedforward (FF) design without any BP as a reference. The FF design adopts a data-centric approach. It derives network parameters of the current layer based on data statistics from the output of the previous layer in a one-pass manner. To construct convolutional layers, we develop a new signal transform, called the Saab (Subspace Approximation with Adjusted Bias) transform. It is a variant of the principal component analysis (PCA) with an added bias vector to annihilate activation's nonlinearity. Multiple Saab transforms in cascade yield multiple convolutional layers. As to fully-connected (FC) layers, we construct them using a cascade of multi-stage linear least squared regressors (LSRs). The classification and robustness (against adversarial attacks) performances of BP- and FF-designed CNNs applied to the MNIST and the CIFAR-10 datasets are compared. Finally, we comment on the relationship between BP and FF designs. In this paper, we make two contributions to unsupervised domain adaptation (UDA) using the convolutional neural network (CNN). First, our approach transfers knowledge in all the convolutional layers through attention alignment. Most previous methods align high-level representations, e.g., activations of the fully connected (FC) layers. In these methods, however, the convolutional layers which underpin critical low-level domain knowledge cannot be updated directly towards reducing domain discrepancy. Specifically, we assume that the discriminative regions in an image are relatively invariant to image style changes. Based on this assumption, we propose an attention alignment scheme on all the target convolutional layers to uncover the knowledge shared by the source domain. Second, we estimate the posterior label distribution of the unlabeled data for target network training. Previous methods, which iteratively update the pseudo labels by the target network and refine the target network by the updated pseudo labels, are vulnerable to label estimation errors. Instead, our approach uses category distribution to calculate the cross-entropy loss for training, thereby ameliorating the error accumulation of the estimated labels. The two contributions allow our approach to outperform the state-of-the-art methods by +2.6 on the Office-31 dataset. Generative Adversarial Networks (GANs) excel at creating realistic images with complex models for which maximum likelihood is infeasible. However, the convergence of GAN training has still not been proved. We propose a two time-scale update rule (TTUR) for training GANs with stochastic gradient descent on arbitrary GAN loss functions. TTUR has an individual learning rate for both the discriminator and the generator. Using the theory of stochastic approximation, we prove that the TTUR converges under mild assumptions to a stationary local Nash equilibrium. The convergence carries over to the popular Adam optimization, for which we prove that it follows the dynamics of a heavy ball with friction and thus prefers flat minima in the objective landscape. For the evaluation of the performance of GANs at image generation, we introduce the "Frechet Inception Distance" (FID) which captures the similarity of generated images to real ones better than the Inception Score. In experiments, TTUR improves learning for DCGANs and Improved Wasserstein GANs (WGAN-GP) outperforming conventional GAN training on CelebA, CIFAR-10, SVHN, LSUN Bedrooms, and the One Billion Word Benchmark.