Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> Learning to store information over extended time intervals by recurrent backpropagation takes a very long time, mostly because of insufficient, decaying error backflow. We briefly review Hochreiter's (1991) analysis of this problem, then address it by introducing a novel, efficient, gradient based method called long short-term memory (LSTM). Truncating the gradient where this does not do harm, LSTM can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units. Multiplicative gate units learn to open and close access to the constant error flow. LSTM is local in space and time; its computational complexity per time step and weight is O. 1. Our experiments with artificial data involve local, distributed, real-valued, and noisy pattern representations. In comparisons with real-time recurrent learning, back propagation through time, recurrent cascade correlation, Elman nets, and neural sequence chunking, LSTM leads to many more successful runs, and learns much faster. LSTM also solves complex, artificial long-time-lag tasks that have never been solved by previous recurrent network algorithms. <s> BIB001 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> We present a simple image-based method of generating novel visual appearance in which a new image is synthesized by stitching together small patches of existing images. We call this process image quilting. First, we use quilting as a fast and very simple texture synthesis algorithm which produces surprisingly good results for a wide range of textures. Second, we extend the algorithm to perform texture transfer — rendering an object with a texture taken from a different object. More generally, we demonstrate how an image can be re-rendered in the style of a different image. The method works directly on the images and does not require 3D information. <s> BIB002 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> This paper describes a new framework for processing images by example, called “image analogies.” The framework involves two stages: a design phase, in which a pair of images, with one image purported to be a “filtered” version of the other, is presented as “training data”; and an application phase, in which the learned filter is applied to some new target image in order to create an “analogous” filtered result. Image analogies are based on a simple multi-scale autoregression, inspired primarily by recent results in texture synthesis. By choosing different types of source image pairs as input, the framework supports a wide variety of “image filter” effects, including traditional image filters, such as blurring or embossing; improved texture synthesis, in which some textures are synthesized with higher quality than by previous approaches; super-resolution, in which a higher-resolution image is inferred from a low-resolution source; texture transfer, in which images are “texturized” with some arbitrary source texture; artistic filters, in which various drawing and painting styles are synthesized based on scanned real-world examples; and texture-by-numbers, in which realistic scenes, composed of a variety of textures, are created using a simple painting interface. <s> BIB003 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> We propose a new phrase-based translation model and decoding algorithm that enables us to evaluate and compare several, previously proposed phrase-based translation models. Within our framework, we carry out a large number of experiments to understand better and explain why phrase-based models out-perform word-based models. Our empirical results, which hold for all examined language pairs, suggest that the highest levels of performance can be obtained through relatively simple means: heuristic learning of phrase translations from word-based alignments and lexical weighting of phrase translations. Surprisingly, learning phrases longer than three words and learning phrases from high-accuracy word-level alignment models does not have a strong impact on performance. Learning only syntactically motivated phrases degrades the performance of our systems. <s> BIB004 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> Learning visual models of object categories notoriously requires hundreds or thousands of training examples. We show that it is possible to learn much information about a category from just one, or a handful, of images. The key insight is that, rather than learning from scratch, one can take advantage of knowledge coming from previously learned categories, no matter how different these categories might be. We explore a Bayesian implementation of this idea. Object categories are represented by probabilistic models. Prior knowledge is represented as a probability density function on the parameters of these models. The posterior model for an object category is obtained by updating the prior in the light of one or more observations. We test a simple implementation of our algorithm on a database of 101 diverse object categories. We compare category models learned by an implementation of our Bayesian approach to models learned from by maximum likelihood (ML) and maximum a posteriori (MAP) methods. We find that on a database of more than 100 categories, the Bayesian approach produces informative models when the number of training examples is too small for other methods to operate successfully. <s> BIB005 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> We present a system that composes a realistic picture from a simple freehand sketch annotated with text labels. The composed picture is generated by seamlessly stitching several photographs in agreement with the sketch and text labels; these are found by searching the Internet. Although online image search generates many inappropriate results, our system is able to automatically select suitable photographs to generate a high quality composition, using a filtering scheme to exclude undesirable images. We also provide a novel image blending algorithm to allow seamless image composition. Each blending result is given a numeric score, allowing us to find an optimal combination of discovered images. Experimental results show the method is very successful; we also evaluate our system using the results from two user studies. <s> BIB006 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> Learning-based approaches for human action recognition often rely on large training sets. Most of these approaches do not perform well when only a few training samples are available. In this paper, we consider the problem of human action recognition from a single clip per action. Each clip contains at most 25 frames. Using a patch based motion descriptor and matching scheme, we can achieve promising results on three different action datasets with a single clip as the template. Our results are comparable to previously published results using much larger training sets. We also present a method for learning a transferable distance function for these patches. The transferable distance function learning extracts generic knowledge of patch weighting from previous training sets, and can be applied to videos of new actions without further learning. Our experimental results show that the transferable distance function learning not only improves the recognition accuracy of the single clip action recognition, but also significantly enhances the efficiency of the matching scheme. <s> BIB007 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> In this paper, we address the problem of efficient human action detection with only one template. We choose the standard sliding-window approach to scan the template video against test videos, and the template video is represented by patch-based motion features. Using generic knowledge learnt from previous training sets, we weight the patches on the template video, by a transferable distance function. Based on the patch weighting, we propose a cascade structure which can efficiently scan the template video over test videos. Our method is evaluated on a human action dataset with cluttered background, and a ballet video with complex human actions. The experimental results show that our cascade structure not only achieves very reliable detection, but also can significantly improve the efficiency of patch-based human action detection, with an order of magnitude improvement in efficiency. <s> BIB008 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> Learning object categories from small samples is a challenging problem, where machine learning tools can in general provide very few guarantees. Exploiting prior knowledge may be useful to reproduce the human capability of recognizing objects even from only one single view. This paper presents an SVM-based model adaptation algorithm able to select and weight appropriately prior knowledge coming from different categories. The method relies on the solution of a convex optimization problem which ensures to have the minimal leave-one-out error on the training set. Experiments on a subset of the Caltech-256 database show that the proposed method produces better results than both choosing one single prior model, and transferring from all previous experience in a flat uninformative way. <s> BIB009 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> In this paper, we study the problem of transfer learning from text to images in the context of network data in which link based bridges are available to transfer the knowledge between the different domains. The problem of classification of image data is often much more challenging than text data because of the following two reasons: (a) Labeled text data is very widely available for classification purposes. On the other hand, this is often not the case for image data, in which a lot of images are available from many sources, but many of them are often not labeled. (b) The image features are not directly related to semantic concepts inherent in class labels. On the other hand, since text data tends to have natural semantic interpretability (because of their human origins), they are often more directly related to class labels. Therefore, the relationships between the images and text features also provide additional hints for the classification process in terms of the image feature transformations which provide the most effective results. The semantic challenges of image features are glaringly evident, when we attempt to recognize complex abstract concepts, and the visual features often fail to discriminate such concepts. However, the copious availability of bridging relationships between text and images in the context of web and social network data can be used in order to design for effective classifiers for image data. One of our goals in this paper is to develop a mathematical model for the functional relationships between text and image features, so as indirectly transfer semantic knowledge through feature transformations. This feature transformation is accomplished by mapping instances from different domains into a common space of unspecific topics. This is used as a bridge to semantically connect the two heterogeneous spaces. This is also helpful for the cases where little image data is available for the classification process. We evaluate our knowledge transfer techniques on an image classification task with labeled text corpora and show the effectiveness with respect to competing algorithms. <s> BIB010 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> One shot learning of simple visual concepts Brenden M. Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua B. Tenenbaum Department of Brain and Cognitive Sciences Massachusetts Institute of Technology Abstract People can learn visual concepts from just one example, but it remains a mystery how this is accomplished. Many authors have proposed that transferred knowledge from more familiar concepts is a route to one shot learning, but what is the form of this abstract knowledge? One hypothesis is that the shar- ing of parts is core to one shot learning, and we evaluate this idea in the domain of handwritten characters, using a massive new dataset. These simple visual concepts have a rich inter- nal part structure, yet they are particularly tractable for com- putational models. We introduce a generative model of how characters are composed from strokes, where knowledge from previous characters helps to infer the latent strokes in novel characters. The stroke model outperforms a competing state- of-the-art character model on a challenging one shot learning task, and it provides a good fit to human perceptual data. Keywords: category learning; transfer learning; Bayesian modeling; neural networks Figure 1: Test yourself on one shot learning. From the example boxed in red, can you find the others in the array? On the left is a Segway and on the right is the first character of the Bengali alphabet. Answer for the Bengali character: Row 2, Column 3; Row 4, Column 2. A hallmark of human cognition is learning from just a few examples. For instance, a person only needs to see one Seg- way to acquire the concept and be able to discriminate future Segways from other vehicles like scooters and unicycles (Fig. 1 left). Similarly, children can acquire a new word from one encounter (Carey & Bartlett, 1978). How is one shot learning possible? New concepts are almost never learned in a vacuum. Past experience with other concepts in a domain can support the rapid learning of novel concepts, by showing the learner what matters for generalization. Many authors have suggested this as a route to one shot learning: transfer of abstract knowledge from old to new concepts, often called transfer learning, rep- resentation learning, or learning to learn. But what is the nature of the learned abstract knowledge that lets humans ac- quire new object concepts so quickly? The most straightforward proposals invoke attentional learning (Smith, Jones, Landau, Gershkoff-Stowe, & Samuel- son, 2002) or overhypotheses (Kemp, Perfors, & Tenenbaum, 2007; Dewar & Xu, in press), like the shape bias in word learning. Prior experience with concepts that are clearly orga- nized along one dimension (e.g., shape, as opposed to color or material) draws a learner’s attention to that same dimension (Smith et al., 2002) – or increases the prior probability of new concepts concentrating on that same dimension (Kemp et al., 2007). But this approach is limited since it requires that the relevant dimensions of similarity be defined in advance. For many real-world concepts, the relevant dimensions of similarity may be constructed in the course of learning to learn. For instance, when we first see a Segway, we may parse it into a structure of familiar parts arranged in a novel configuration: it has two wheels, connected by a platform, supporting a motor and a central post at the top of which are two handlebars. These parts and their relations comprise a Figure 2: Examples from a new 1600 character database. useful representational basis for many different vehicle and artifact concepts – a representation that is likely learned in the course of learning the concepts that they support. Several papers from the recent machine learning and computer vision literature argue for such an approach: joint learning of many concepts and a high-level part vocabulary that underlies those concepts (e.g., Torralba, Murphy, & Freeman, 2007; Fei-Fei, Fergus, & Perona, 2006). Another recently popular machine learning approach is based on deep learning (Salakhutdinov & Hinton, 2009): unsupervised learning of hierarchies of dis- tributed feature representations in neural-network-style prob- abilistic generative models. These models do not specify ex- plicit parts and structural relations, but they can still construct meaningful representations of what makes two objects deeply similar that go substantially beyond low-level image features. These approaches from machine learning may be com- pelling ways to understand how humans learn so quickly, but there is little experimental evidence that directly supports them. Models that construct parts or features from sensory data (pixels) while learning object concepts have been tested in elegant behavioral experiments with very simple stimuli and a very small number of concepts (Austerweil & Griffiths, 2009; Schyns, Goldstone, & Thibaut, 1998). But there have been few systematic comparisons of multiple state-of-the-art computational approaches to representation learning with hu- <s> BIB011 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> Our objective is transfer training of a discriminatively trained object category detector, in order to reduce the number of training images required. To this end we propose three transfer learning formulations where a template learnt previously for other categories is used to regularize the training of a new category. All the formulations result in convex optimization problems. Experiments (on PASCAL VOC) demonstrate significant performance gains by transfer learning from one class to another (e.g. motorbike to bicycle), including one-shot learning, specialization from class to a subordinate class (e.g. from quadruped to horse) and transfer using multiple components. In the case of multiple training samples it is shown that a detection performance approaching that of the state of the art can be achieved with substantially fewer training samples. <s> BIB012 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> The vast majority of transfer learning methods proposed in the visual recognition domain over the last years addresses the problem of object category detection, assuming a strong control over the priors from which transfer is done. This is a strict condition, as it concretely limits the use of this type of approach in several settings: for instance, it does not allow in general to use off-the-shelf models as priors. Moreover, the lack of a multiclass formulation for most of the existing transfer learning algorithms prevents using them for object categorization problems, where their use might be beneficial, especially when the number of categories grows and it becomes harder to get enough annotated data for training standard learning methods. This paper presents a multiclass transfer learning algorithm that allows to take advantage of priors built over different features and with different learning methods than the one used for learning the new task. We use the priors as experts, and transfer their outputs to the new incoming samples as additional information. We cast the learning problem within the Multi Kernel Learning framework. The resulting formulation solves efficiently a joint optimization problem that determines from where and how much to transfer, with a principled multiclass formulation. Extensive experiments illustrate the value of this approach. <s> BIB013 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> We introduce a class of probabilistic continuous translation models called Recurrent Continuous Translation Models that are purely based on continuous representations for words, phrases and sentences and do not rely on alignments or phrasal translation units. The models have a generation and a conditioning aspect. The generation of the translation is modelled with a target Recurrent Language Model, whereas the conditioning on the source sentence is modelled with a Convolutional Sentence Model. Through various experiments, we show first that our models obtain a perplexity with respect to gold translations that is > 43% lower than that of stateof-the-art alignment-based translation models. Secondly, we show that they are remarkably sensitive to the word order, syntax, and meaning of the source sentence despite lacking alignments. Finally we show that they match a state-of-the-art system when rescoring n-best lists of translations. <s> BIB014 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> We introduce "time hallucination": synthesizing a plausible image at a different time of day from an input image. This challenging task often requires dramatically altering the color appearance of the picture. In this paper, we introduce the first data-driven approach to automatically creating a plausible-looking photo that appears as though it were taken at a different time of day. The time of day is specified by a semantic time label, such as "night". Our approach relies on a database of time-lapse videos of various scenes. These videos provide rich information about the variations in color appearance of a scene throughout the day. Our method transfers the color appearance from videos with a similar scene as the input photo. We propose a locally affine model learned from the video for the transfer, allowing our model to synthesize new color data while retaining image details. We show that this model can hallucinate a wide range of different times of day. The model generates a large sparse linear system, which can be solved by off-the-shelf solvers. We validate our methods by synthesizing transforming photos of various outdoor scenes to four times of interest: daytime, the golden hour, the blue hour, and nighttime. <s> BIB015 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> We consider the transfer learning scenario, where the learner does not have access to the source domain directly, but rather operates on the basis of hypotheses induced from it - the Hypothesis Transfer Learning (HTL) problem. Particularly, we conduct a theoretical analysis of HTL by considering the algorithmic stability of a class of HTL algorithms based on Regularized Least Squares with biased regularization. We show that the relatedness of source and target domains accelerates the convergence of the Leave-One-Out error to the generalization error, thus enabling the use of the Leave-One-Out error to find the optimal transfer parameters, even in the presence of a small training set. In case of unrelated domains we also suggest a theoretically principled way to prevent negative transfer, so that in the limit we recover the performance of the algorithm not using any knowledge from the source domain. <s> BIB016 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT-14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous state of the art. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier. <s> BIB017 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition. <s> BIB018 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> We live in a dynamic visual world where the appearance of scenes changes dramatically from hour to hour or season to season. In this work we study "transient scene attributes" -- high level properties which affect scene appearance, such as "snow", "autumn", "dusk", "fog". We define 40 transient attributes and use crowdsourcing to annotate thousands of images from 101 webcams. We use this "transient attribute database" to train regressors that can predict the presence of attributes in novel images. We demonstrate a photo organization method based on predicted attributes. Finally we propose a high-level image editing method which allows a user to adjust the attributes of a scene, e.g. change a scene to be "snowy" or "sunset". To support attribute manipulation we introduce a novel appearance transfer technique which is simple and fast yet competitive with the state-of-the-art. We show that we can convincingly modify many transient attributes in outdoor scenes. <s> BIB019 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> The transfer learning and domain adaptation problems originate from a distribution mismatch between the source and target data distribution. The causes of such mismatch are traditionally considered different. Thus, transfer learn- ing and domain adaptation algorithms are designed to ad- dress different issues, and cannot be used in both settings unless substantially modified. Still, one might argue that these problems are just different declinations of learning to learn, i.e. the ability to leverage over prior knowledge when attempting to solve a new task. We propose a learning to learn framework able to lever- age over source data regardless of the origin of the distri- bution mismatch. We consider prior models as experts, and use their output confidence value as features. We use them to build the new target model, combined with the features from the target data through a high-level cue integration scheme. This results in a class of algorithms usable in a plug-and-play fashion over any learning to learn scenario, from binary and multi-class transfer learning to single and multiple source domain adaptation settings. Experiments on several public datasets show that our approach consis- tently achieves the state of the art. <s> BIB020 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> Much research on human action recognition has been oriented toward the performance gain on lab-collected datasets. Yet real-world videos are more diverse, with more complicated actions and often only a few of them are precisely labeled. Thus, recognizing actions from these videos is a tough mission. The paucity of labeled real-world videos motivates us to "borrow" strength from other resources. Specifically, considering that many lab datasets are available, we propose to harness lab datasets to facilitate the action recognition in real-world videos given that the lab and real-world datasets are related. As their action categories are usually inconsistent, we design a multi-task learning framework to jointly optimize the classifiers for both sides. The general Schatten $$p$$ p -norm is exerted on the two classifiers to explore the shared knowledge between them. In this way, our framework is able to mine the shared knowledge between two datasets even if the two have different action categories, which is a major virtue of our method. The shared knowledge is further used to improve the action recognition in the real-world videos. Extensive experiments are performed on real-world datasets with promising results. <s> BIB021 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> In this paper we address three different computer vision tasks using a single basic architecture: depth prediction, surface normal estimation, and semantic labeling. We use a multiscale convolutional network that is able to adapt easily to each task using only small modifications, regressing from the input image to the output map directly. Our method progressively refines predictions using a sequence of scales, and captures many image details without any superpixels or low-level segmentation. We achieve state-of-the-art performance on benchmarks for all three tasks. <s> BIB022 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Indeed, since the release of the pix2pix software associated with this paper, a large number of internet users (many of them artists) have posted their own experiments with our system, further demonstrating its wide applicability and ease of adoption without the need for parameter tweaking. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without hand-engineering our loss functions either. <s> BIB023 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> This paper proposes Markovian Generative Adversarial Networks (MGANs), a method for training generative networks for efficient texture synthesis. While deep neural network approaches have recently demonstrated remarkable results in terms of synthesis quality, they still come at considerable computational costs (minutes of run-time for low-res images). Our paper addresses this efficiency issue. Instead of a numerical deconvolution in previous work, we precompute a feed-forward, strided convolutional network that captures the feature statistics of Markovian patches and is able to directly generate outputs of arbitrary dimensions. Such network can directly decode brown noise to realistic texture, or photos to artistic paintings. With adversarial training, we obtain quality comparable to recent neural texture synthesis methods. As no optimization is required at generation time, our run-time performance (0.25 M pixel images at 25 Hz) surpasses previous neural texture synthesizers by a significant margin (at least 500 times faster). We apply this idea to texture synthesis, style transfer, and video stylization. <s> BIB024 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> Current generative frameworks use end-to-end learning and generate images by sampling from uniform noise distribution. However, these approaches ignore the most basic principle of image formation: images are product of: (a) Structure: the underlying 3D model; (b) Style: the texture mapped onto structure. In this paper, we factorize the image generation process and propose Style and Structure Generative Adversarial Network (\({\text {S}^2}\)-GAN). Our \({\text {S}^2}\)-GAN has two components: the Structure-GAN generates a surface normal map; the Style-GAN takes the surface normal map as input and generates the 2D image. Apart from a real vs. generated loss function, we use an additional loss with computed surface normals from generated images. The two GANs are first trained independently, and then merged together via joint learning. We show our \({\text {S}^2}\)-GAN model is interpretable, generates more realistic images and can be used to learn unsupervised RGBD representations. <s> BIB025 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> Rendering the semantic content of an image in different styles is a difficult image processing task. Arguably, a major limiting factor for previous approaches has been the lack of image representations that explicitly represent semantic information and, thus, allow to separate image content from style. Here we use image representations derived from Convolutional Neural Networks optimised for object recognition, which make high level image information explicit. We introduce A Neural Algorithm of Artistic Style that can separate and recombine the image content and style of natural images. The algorithm allows us to produce new images of high perceptual quality that combine the content of an arbitrary photograph with the appearance of numerous wellknown artworks. Our results provide new insights into the deep image representations learned by Convolutional Neural Networks and demonstrate their potential for high level image synthesis and manipulation. <s> BIB026 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> We consider image transformation problems, where an input image is transformed into an output image. Recent methods for such problems typically train feed-forward convolutional neural networks using a per-pixel loss between the output and ground-truth images. Parallel work has shown that high-quality images can be generated by defining and optimizing perceptual loss functions based on high-level features extracted from pretrained networks. We combine the benefits of both approaches, and propose the use of perceptual loss functions for training feed-forward networks for image transformation tasks. We show results on image style transfer, where a feed-forward network is trained to solve the optimization problem proposed by Gatys et al. in real-time. Compared to the optimization-based method, our network gives similar qualitative results but is three orders of magnitude faster. We also experiment with single-image super-resolution, where replacing a per-pixel loss with a perceptual loss gives visually pleasing results. <s> BIB027 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> Learning from a few examples remains a key challenge in machine learning. Despite recent advances in important domains such as vision and language, the standard supervised deep learning paradigm does not offer a satisfactory solution for learning new concepts rapidly from little data. In this work, we employ ideas from metric learning based on deep neural features and from recent advances that augment neural networks with external memories. Our framework learns a network that maps a small labelled support set and an unlabelled example to its label, obviating the need for fine-tuning to adapt to new class types. We then define one-shot learning problems on vision (using Omniglot, ImageNet) and language tasks. Our algorithm improves one-shot accuracy on ImageNet from 87.6% to 93.2% and from 88.0% to 93.8% on Omniglot compared to competing approaches. We also demonstrate the usefulness of the same model on language modeling by introducing a one-shot task on the Penn Treebank. <s> BIB028 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> We develop a new edge detection algorithm that addresses two important issues in this long-standing vision problem: (1) holistic image training and prediction; and (2) multi-scale and multi-level feature learning. Our proposed method, holistically-nested edge detection (HED), performs image-to-image prediction by means of a deep learning model that leverages fully convolutional neural networks and deeply-supervised nets. HED automatically learns rich hierarchical representations (guided by deep supervision on side responses) that are important in order to resolve the challenging ambiguity in edge and object boundary detection. We significantly advance the state-of-the-art on the BSDS500 dataset (ODS F-score of 0.790) and the NYU Depth dataset (ODS F-score of 0.746), and do so with an improved speed (0.4 s per image) that is orders of magnitude faster than some CNN-based edge detection algorithms developed before HED. We also observe encouraging results on other boundary detection benchmark datasets such as Multicue and PASCAL-Context. <s> BIB029 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> Though deep neural networks have shown great success in the large data domain, they generally perform poorly on few-shot learning tasks, where a model has to quickly generalize after seeing very few examples from each class. The general belief is that gradient-based optimization in high capacity models requires many iterative steps over many examples to perform well. Here, we propose an LSTM-based meta-learner model to learn the exact optimization algorithm used to train another learner neural network in the few-shot regime. The parametrization of our model allows it to learn appropriate parameter updates specifically for the scenario where a set amount of updates will be made, while also learning a general initialization of the learner network that allows for quick convergence of training. We demonstrate that this meta-learning model is competitive with deep metric-learning techniques for few-shot learning. <s> BIB030 </s> Transfer Learning for Cross-Dataset Recognition: A Survey <s> Higher-level Representation Criterion. <s> A recent approach to few-shot classification called matching networks has demonstrated the benefits of coupling metric learning with a training procedure that mimics test. This approach relies on a complicated fine-tuning procedure and an attention scheme that forms a distribution over all points in the support set, scaling poorly with its size. We propose a more streamlined approach, prototypical networks, that learns a metric space in which few-shot classification can be performed by computing Euclidean distances to prototype representations of each class, rather than individual points. Our method is competitive with state-of-the-art one-shot classification approaches while being much simpler and more scalable with the size of the support set. We empirically demonstrate the performance of our approach on the Omniglot and mini-ImageNet datasets. We further demonstrate that a similar idea can be used for zero-shot learning, where each class is described by a set of attributes, and achieve state-of-the-art results on the Caltech UCSD bird dataset. <s> BIB031
Feature Representation Transfer. Similar to , also assume that the source domain contains data with multi-modality but the target domain only has one data modality. They name this setting as Missing Modality Problem. They propose to recover the missing modality in the target domain by finding similar data from the source domain. In their method, a latent factor is incorporated to uncover the missing modality based on the low-rank criterion. Zero-padding has also been used for dealing with the heterogeneous feature spaces. For example, the marginalized SDA (mSDA) ] deals with heterogeneous feature spaces by padding all input vectors with zeros to make both domains be of equal dimensionality. Feature Representation Transfer. The co-occurrence data between different feature spaces or modalities have been employed for heterogeneous domain adaptation BIB010 . In Qi's method BIB010 , the co-occurrence data are used for mapping instances from different domains into a common space as a bridge to semantically connect the two heterogeneous spaces. Differently, propose to learn the transferred weights obtained from co-occurrence data, which indicate the relatedness between heterogeneous domains. Specifically, they compute the principal components of instances in each feature space such that co-occurrence data can be represented by these principal components. By using these principal component coefficients, the Markov Chain Monte Carlo method is employed to construct a directed cyclic network where each node is a domain and each edge weight is the conditional dependence from one domain to another domain. A line of research dedicates on the task of translation between different domains. For example, in the context of machine translation between languages, the sentence pairs are presented in the form of a parallel training corpus for learning the translation system. Traditional phrase-based translation system BIB004 ] consists of many small sub-components that are tuned separately. Differently, a newly emerging approach, named Neural machine translation BIB014 BIB017  BIB018 , attempts to build and train a single, large neural network that reads a sentence and outputs a correct translation. The neural machine translation approach typically consists of two components: the first encodes a source sentence, and the second decodes to a target sentence. Similarly, in the vision domain, image-to-image translation BIB023 ] has also been extensively exploited, which aims at converting an image from one representation of a given scene to another (i.e. greyscale to color BIB023 , texture synthesis BIB002 BIB003 BIB024 , sketch to photograph BIB006 BIB023 ], time hallucination BIB015 BIB019 BIB023 ], image to semantic labels BIB022 BIB029 , and style transfer BIB024 BIB025 BIB026 BIB027 ). The key idea for tackling these tasks is to learn a translation model between paired samples from different domains. The recent deep learning based techniques have greatly advanced the image-to-image translation task. For example, the deep convolutional neural networks BIB029 BIB022 BIB026 BIB027 , and the Generative Adversarial Networks (GANs) BIB025 BIB024 BIB023 ] have been extensively exploited for learning the translation model. Though the original purposes of these work on translation between domains may not be cross-dataset recognition, the ideas can be borrowed for cross-modality or cross feature spaces recognition. Since if the source domain data can be translated to the target domain properly, the target task can be boosted by the translated source domain data. assume that the source data are large scale labelled RGB data and the target data are unlabelled RGB and depth image pairs. The source data are used for training multiple layers of rich representations using deep convolutional neural networks. Then the paired target data are used for transfer the source rich parameters to the target networks by constraining the paired samples from different modalities to have similar representations. Feature Representation-based Transfer. BIB007 BIB008 propose to transfer the parameters of distance function from source data to the target. The distance function can help to detect the patch saliency in the target videos. Instance-based Transfer. Qi et al. [2011b] develop a cross-category label propagation algorithm, which directly propagate the inter-category knowledge at instance level between the source and the target categories. Feature Representation-based Transfer. BIB020 consider each source as an expert that judges on the new target samples. The output confidence value of prior models is treated as features and the features are combined with the features from the target samples to build a target classifier. Classifier-based Transfer. Several classifier-based methods are proposed to transfer the parameters of classifiers. BIB005 propose a Bayesian approach, where a generic object model is estimated from some source categories and it is then used as prior to evaluate the target object parameter distribution with a maximum-a-posteriori technique. BIB011 introduce a generative model of how characters are composed from strokes, where knowledge from previous characters helps to infer the latent strokes in novel characters. Instead of using generative model, some discriminative classifier-based transfer methods BIB009 BIB012 BIB021 BIB013 are also proposed to incorporate prior knowledge. For example, BIB009 present an Least Square Support Vector Machine (LS-SVM) based model adaptation algorithm able to select and weight appropriately prior knowledge coming from different categories by assuming the new categories are similar to some of the "prior" categories. Similarly, BIB012 propose three transfer learning formulations, namely Adaptive SVM(A-SVM), Projective Model Transfer SVM (PMT-SVM), Deformable Adaptive SVM (DA-SVM), where a template learnt previously for other categories is used to regularize the training of a new category. They also assume the target category is visually similar to a source category. BIB021 design a multi-task learning framework to explore the shared knowledge between source and target classifiers and jointly optimize the classifiers for both sides. BIB013 propose Multi Kernel Transfer Learning (MKTL) method, which takes advantages of priors built over different features and with different learning methods. They use the priors as experts and transfer their outputs to the target using multiple kernel learning (MKL). The intuition is that knowledge can be transferred between classes that share common visual properties, such as bicycle and motorbike. BIB016 conduct a theoretical analysis (with the focus of algorithmic stability) of the discriminative classifier-based transfer methods BIB009 BIB012 BIB021 BIB013 , which is named as Hypothesis Transfer Learning (HTL) in their paper. In HTL, only source hypotheses trained on a source domain are utilized for transferring the parameters to the target domain with the presence of a small set of target labelled data. They give the generalization bound in terms of the Leave-One-Out (LOO) risk and show that the relatedness of source and target domains accelerates the convergence of the LOO error and generalization error. In the case of unrelated domains, they propose how a hypothetical algorithm could avoid negative transfer. Recently, the deep learning based approaches have been emerged for few-shot learning. BIB028 propose the matching networks, which uses an attention mechanism over a learned embedding of the limited labelled data of target classes, which can be interpreted as a weighted nearest-neighbor classifier applied within an embedding space. BIB030 propose a meta-learning approach to fewshot learning. Their approach involves training an LSTM BIB001 to produce the updates to a classifier, given a few target labelled examples, such that it will generalize well to the target set. BIB031 propose the prototypical networks by learning a non-linear mapping of the input into an embedding space using a neural network and take a class's prototype to be the mean of its support set in the embedding space.