Data augmentation (DA) is fundamental against overfitting in large convolutional neural networks, especially with a limited training dataset. In images, DA is usually based on heuristic transformations, like geometric or color transformations. Instead of using predefined transformations, our work learns data augmentation directly from the training data by learning to transform images with an encoder-decoder architecture combined with a spatial transformer network. The transformed images still belong to the same class, but are new, more complex samples for the classifier. Our experiments show that our approach is better than previous generative data augmentation methods, and comparable to predefined transformation methods when training an image classifier. Convolutional neural networks have shown impressive results in visual recognition tasks. However, for proper training and good performance, they require large labeled datasets. If the amount of training data is small, data augmentation is an effective way to improve the final performance of the network BID6 ; BID9 ). In images, data augmentation (DA) consists of applying predefined transformations such as flip, rotations or color changes BID8 ; BID3 ). This approach provides consistent improvements when training a classifier. However, the required transformations are dataset dependent. For instance, flipping an image horizontally makes sense for natural images, but produces ambiguities on datasets of numbers (e.g. 2 and 5).Several recent studies investigate automatic DA learning as a method to avoid the manual selection of transformations. BID10 define a large set of transformations and learn how to combine them. This approach works well however, as it is based on predefined transformations, it prevents the model from finding other transformations that could be useful for the classifier. Alternatively , BID2 and BID12 generate new samples via a generative adversarial networks model (GAN) from the probability distribution of the data p(X), while BID0 learn the transformations of images, instead of generating images from scratch. These alternative methods show their limits when the number of training samples is low, given the difficulty of training a high-performing generative model with a reduced dataset. BID5 learn the natural transformations in a dataset by aligning pairs of samples from the same class. This approach produces good results on easy datasets like MNIST however, it does not appear to be applicable to more complex datasets.Our work combines the advantages of generative models and transformation learning approaches in a single end-to-end network architecture. Our model is based on a conditional GAN architecture that learns to generate transformations of a given image that are useful for DA. In other words, instead of learning to generate samples from p(X), it learns to generate samples from the conditional distribution p(X|X), withX a reference image. As shown in FIG0 , our approach combines a global transformation defined by an affine matrix with a more localized transformation defined by ensures that the transformed sample G(x i , z) is dissimilar from the input sample x i but similar to a sample x j from the same class. (b) Given an input image x i and a random noise vector z, our generator first performs a global transformation using a spatial transformer network followed by more localized transformations using a convolutional encoder-decoder network.a convolutional encoder-decoder architecture. The global transformations are learned by an adaptation of spatial transformer network (STN) BID7 ) so that the entire architecture is differentiable and can be learned with standard back-propagation. In its normal use, the purpose of STN is to learn how to transform the input data, so that the model becomes invariant to certain transformations. In contrast, our approach uses STN to generate augmented samples in an adversarial way. With the proposed model we show that, for optimal performance, it is important to jointly train the generator of the augmented samples with the classifier in an end-to-end fashion. By doing that, we can also add an adversarial loss between the generator and classifier such that the generated samples are difficult, or adversarial, for the classifier.To summarize, the contributions of this paper are: i) We propose a DA network that can automatically learn to generate augmented samples without expensive searches for the optimal data transformations; ii) Our model trains jointly with a classifier, is fully differentiable, trainable end-to-end, and can significantly improve the performance of any image classifier; iii) In low-data regime it outperforms models trained with strong predefined DA; iv) Finally, we notice that, for optimal performance, it is fundamental to train the model jointly with the image classifier. In this work, we have presented a new approach for improving the learning of a classifier through an automatic generation of augmented samples. The method is fully differentiable and can be learned end-to-end. In our experiments, we have shown several elements contributing to an improved classification performance. First, the generator and the classifier should be trained jointly. Second, the combined use of global transformations with STN and local transformation with U-Net is essential to reach the highest accuracy levels. For future work, we want to include more differentiable transformations such as deformations and color transformations and evaluate how these additional sample augmentations affect the final accuracy.
Automatic Learning of data augmentation using a GAN based architecture to improve an image classifier
{"src": "scitldr-AIC_train", "paper_id": "BylSX4meOV"}