A Survey on the New Generation of Deep Learning in Image Processing <s> 1) FEW-SHOT LEARNING FOR DEEP LEARNING <s> The reinforcement learning paradigm is a popular way to address problems that have only limited environmental feedback, rather than correctly labeled examples, as is common in other machine learning contexts. While significant progress has been made to improve learning in a single task, the idea of transfer learning has only recently been applied to reinforcement learning tasks. The core idea of transfer is that experience gained in learning to perform one task can help improve learning performance in a related, but different, task. In this article we present a framework that classifies transfer learning methods in terms of their capabilities and goals, and then use it to survey the existing literature, as well as to suggest future directions for transfer learning work. <s> BIB001 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> 1) FEW-SHOT LEARNING FOR DEEP LEARNING <s> Convolutional neural networks (CNNs) work well on large datasets. But labelled data is hard to collect, and in some applications larger amounts of data are not available. The problem then is how to use CNNs with small data -- as CNNs overfit quickly. We present an efficient Bayesian CNN, offering better robustness to over-fitting on small data than traditional approaches. This is by placing a probability distribution over the CNN's kernels. We approximate our model's intractable posterior with Bernoulli variational distributions, requiring no additional model parameters. ::: On the theoretical side, we cast dropout network training as approximate inference in Bayesian neural networks. This allows us to implement our model using existing tools in deep learning with no increase in time complexity, while highlighting a negative result in the field. We show a considerable improvement in classification accuracy compared to standard techniques and improve on published state-of-the-art results for CIFAR-10. <s> BIB002 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> 1) FEW-SHOT LEARNING FOR DEEP LEARNING <s> We develop a conceptually simple but powerful approach that can learn novel categories from few annotated examples. In this approach, the experience with already learned categories is used to facilitate the learning of novel classes. Our insight is two-fold: (1) there exists a generic, category agnostic transformation from models learned from few samples to models learned from large enough sample sets, and (2) such a transformation could be effectively learned by high-capacity regressors. In particular, we automatically learn the transformation with a deep model regression network on a large collection of model pairs. Experiments demonstrate that encoding this transformation as prior knowledge greatly facilitates the recognition in the small sample size regime on a broad range of tasks, including domain adaptation, fine-grained recognition, action recognition, and scene classification. <s> BIB003 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> 1) FEW-SHOT LEARNING FOR DEEP LEARNING <s> We introduce deep transform learning — a new tool for deep learning. Deeper representation is learnt by stacking one transform after another. The learning proceeds in a greedy way. The first layer learns the transform and features from the input training samples. Subsequent layers use the features (after activation) from the previous layers as training input. Experiments have been carried out with other deep representation learning tools — deep dictionary learning, stacked denoising autoencoder, deep belief network and PCANet (a version of convolutional neural network). Results show that our proposed technique is better than all the said techniques, at least on the benchmark datasets (MNIST, CIFAR-10 and SVHN) compared on. <s> BIB004 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> 1) FEW-SHOT LEARNING FOR DEEP LEARNING <s> INTRODUCTION Compositionality, generalization, and learning from a few examples are among the hallmarks of human intelligence. CAPTCHAs (Completely Automated Public Turing test to tell Computers and Humans Apart), images used by websites to block automated interactions, are examples of problems that are easy for people but difficult for computers. CAPTCHAs add clutter and crowd letters together to create a chicken-and-egg problem for algorithmic classifiers—the classifiers work well for characters that have been segmented out, but segmenting requires an understanding of the characters, which may be rendered in a combinatorial number of ways. CAPTCHAs also demonstrate human data efficiency: A recent deep-learning approach for parsing one specific CAPTCHA style required millions of labeled examples, whereas humans solve new styles without explicit training. By drawing inspiration from systems neuroscience, we introduce recursive cortical network (RCN), a probabilistic generative model for vision in which message-passing–based inference handles recognition, segmentation, and reasoning in a unified manner. RCN learns with very little training data and fundamentally breaks the defense of modern text-based CAPTCHAs by generatively segmenting characters. In addition, RCN outperforms deep neural networks on a variety of benchmarks while being orders of magnitude more data-efficient. RATIONALE Modern deep neural networks resemble the feed-forward hierarchy of simple and complex cells in the neocortex. Neuroscience has postulated computational roles for lateral and feedback connections, segregated contour and surface representations, and border-ownership coding observed in the visual cortex, yet these features are not commonly used by deep neural nets. We hypothesized that systematically incorporating these findings into a new model could lead to higher data efficiency and generalization. Structured probabilistic models provide a natural framework for incorporating prior knowledge, and belief propagation (BP) is an inference algorithm that can match the cortical computational speed. The representational choices in RCN were determined by investigating the computational underpinnings of neuroscience data under the constraint that accurate inference should be possible using BP. RESULTS RCN was effective in breaking a wide variety of CAPTCHAs with very little training data and without using CAPTCHA-specific heuristics. By comparison, a convolutional neural network required a 50,000-fold larger training set and was less robust to perturbations to the input. Similar results are shown on one- and few-shot MNIST (modified National Institute of Standards and Technology handwritten digit data set) classification, where RCN was significantly more robust to clutter introduced during testing. As a generative model, RCN outperformed neural network models when tested on noisy and cluttered examples and generated realistic samples from one-shot training of handwritten characters. RCN also proved to be effective at an occlusion reasoning task that required identifying the precise relationships between characters at multiple points of overlap. On a standard benchmark for parsing text in natural scenes, RCN outperformed state-of-the-art deep-learning methods while requiring 300-fold less training data. CONCLUSION Our work demonstrates that structured probabilistic models that incorporate inductive biases from neuroscience can lead to robust, generalizable machine learning models that learn with high data efficiency. In addition, our model’s effectiveness in breaking text-based CAPTCHAs with very little training data suggests that websites should seek more robust mechanisms for detecting automated interactions. <s> BIB005 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> 1) FEW-SHOT LEARNING FOR DEEP LEARNING <s> Deep convolutional neural networks have recently proven extremely effective for difficult face recognition problems in uncontrolled settings. To train such networks, very large training sets are needed with millions of labeled images. For some applications, such as near-infrared (NIR) face recognition, such large training datasets are not publicly available and difficult to collect. In this work, we propose a method to generate very large training datasets of synthetic images by compositing real face images in a given dataset. We show that this method enables to learn models from as few as 10,000 training images, which perform on par with models trained from 500,000 images. Using our approach we also obtain state-of-the-art results on the CASIA NIR-VIS2.0 heterogeneous face recognition dataset. <s> BIB006
In general, when the amount of training images with labels is not large enough, the deep networks can effectively avoid over-fitting phenomena using the following standard four methods BIB003 . • The first is data augmentation; we can manually increase the size of the training images; that is, one can produce a batch of new image sets from the available small training images by shift, rotation, noise addition, flipping, color jittering, and random crop. • The second is regularization; over-fitting can be suppressed by adding a regularization term after the loss function. However, the disadvantage is the introduction of a hyper-parameter that requires manual adjustment. • The third is a dropout; in essence, this is also a regularization method, which can be achieved by randomly zeroing the output of some neurons in a hidden layer. • The fourth is adopt the unsupervised layer-wise pretraining and fine-tuning; that is, layer-wise unsupervised pre-training can achieve by using AE or RBM, then adding classification layer to perform supervised endto-end fine tuning BIB004 , BIB006 . Apart from the above four commonly used methods, more optimization designs framework for deep learning have also been proposed and applied to few-shot learning. Below, we introduce some of the most representative network frameworks. • The first is the deep generative model, which is a powerful way of learning any data distribution function using unsupervised learning, which can generate a new image with some variations. Two of the most commonly used and efficient approaches are VAE and GAN. Notably, these two approaches have achieved tremendous success in just a few years. However, an open problem is still worth exploring is how to generate more detailed and high-quality images under the premise of guaranteeing network convergence. • The second is a recursive cortical network (RCN) BIB005 , which is a probabilistic and-or graph model in essence. RCN can acquire the learning ability on small samples by hierarchical and production modeling. Concretely, through the separation modeling of the edges and planes of objects and the hierarchical modeling of complex changes such as textures and scales, the whole model has a strong ability of generalization and robustness to changes in appearance. Naturally, the combination of probabilistic and graph models and deep networks may be a promising research direction. • The third is transfer learning, which focuses on storing knowledge gained while solving one problem and applying it to a different but related problem BIB001 ,  . For example, we have a classification task in one domain of interest, but sufficient training image sets in another domain. If transfer learning can be trained successfully, then significantly improve the performance of learning by avoiding many expensive samples. Generally, transfer learning works in two similar domains, and the performance is excellent. However, the standard of measuring the difference between the two domains has not improved. Nevertheless, this does not seem to affect its combination with deep learning may become the next research hotspot. • The fourth is bayesian CNN BIB002 , which placing a probability distribution over the CNN's kernels, can offer better robustness to over-fitting on small sample learning.