This paper explores the use of self-ensembling for visual domain adaptation problems. Our technique is derived from the mean teacher variant (Tarvainen et. al 2017) of temporal ensembling (Laine et al. 2017), a technique that achieved state of the art results in the area of semi-supervised learning. We introduce a number of modifications to their approach for challenging domain adaptation scenarios and evaluate its effectiveness. Our approach achieves state of the art results in a variety of benchmarks, including our winning entry in the VISDA-2017 visual domain adaptation challenge. In small image benchmarks, our algorithm not only outperforms prior art, but can also achieve accuracy that is close to that of a classifier trained in a supervised fashion. The strong performance of deep learning in computer vision tasks comes at the cost of requiring large datasets with corresponding ground truth labels for training. Such datasets are often expensive to produce, owing to the cost of the human labour required to produce the ground truth labels.Semi-supervised learning is an active area of research that aims to reduce the quantity of ground truth labels required for training. It is aimed at common practical scenarios in which only a small subset of a large dataset has corresponding ground truth labels. Unsupervised domain adaptation is a closely related problem in which one attempts to transfer knowledge gained from a labeled source dataset to a distinct unlabeled target dataset, within the constraint that the objective (e.g.digit classification) must remain the same. Domain adaptation offers the potential to train a model using labeled synthetic data -that is often abundantly available -and unlabeled real data. The scale of the problem can be seen in the VisDA-17 domain adaptation challenge images shown in FIG3 . We will present our winning solution in Section 4.2.Recent work BID28 ) has demonstrated the effectiveness of self-ensembling with random image augmentations to achieve state of the art performance in semi-supervised learning benchmarks.We have developed the approach proposed by BID28 to work in a domain adaptation scenario. We will show that this can achieve excellent results in specific small image domain adaptation benchmarks. More challenging scenarios, notably MNIST → SVHN and the VisDA-17 domain adaptation challenge required further modifications. To this end, we developed confidence thresholding and class balancing that allowed us to achieve state of the art results in a variety of benchmarks, with some of our results coming close to those achieved by traditional supervised learning. Our approach is sufficiently flexble to be applicable to a variety of network architectures, both randomly initialized and pre-trained.Our paper is organised as follows; in Section 2 we will discuss related work that provides context and forms the basis of our technique; our approach is described in Section 3 with our experiments and results in Section 4; and finally we present our conclusions in Section 5. In this section we will cover self-ensembling based semi-supervised methods that form the basis of our approach and domain adaptation techniques to which our work can be compared. We have presented an effective domain adaptation algorithm that has achieved state of the art results in a number of benchmarks and has achieved accuracies that are almost on par with traditional supervised learning on digit recognition benchmarks targeting the MNIST and SVHN datasets. The Table 2 : VisDA-17 performance, presented as mean ± std-dev of 5 independent runs. Full results are presented in TAB6 in Appendix C. resulting networks will exhibit strong performance on samples from both the source and target domains. Our approach is sufficiently flexible to be usable for a variety of network architectures, including those based on randomly initialised and pre-trained networks. stated that the self-ensembling methods presented by Laine & Aila (2017) -on which our algorithm is based -operate by label propagation. This view is supported by our results, in particular our MNIST → SVHN experiment. The latter requires additional intensity augmentation in order to sufficiently align the dataset distributions, after which good quality label predictions are propagated throughout the target dataset. In cases where data augmentation is insufficient to align the dataset distributions, a pre-trained network may be used to bridge the gap, as in our solution to the VisDA-17 challenge. This leads us to conclude that effective domain adaptation can be achieved by first aligning the distributions of the source and target datasets -the focus of much prior art in the field -and then refining their correspondance; a task to which self-ensembling is well suited. The datasets used in this paper are described in Some of the experiments that involved datasets described in TAB3 required additional data preparation in order to match the resolution and format of the input samples and match the classification target. These additional steps will now be described.MNIST ↔ USPS The USPS images were up-scaled using bilinear interpolation from 16 × 16 to 28 × 28 resolution to match that of MNIST.CIFAR-10 ↔ STL CIFAR-10 and STL are both 10-class image datasets. The STL images were down-scaled to 32 × 32 resolution to match that of CIFAR-10. The 'frog' class in CIFAR-10 and the 'monkey' class in STL were removed as they have no equivalent in the other dataset, resulting in a 9-class problem with 10% less samples in each dataset.Syn-Signs → GTSRB GTSRB is composed of images that vary in size and come with annotations that provide region of interest (bounding box around the sign) and ground truth classification. We extracted the region of interest from each image and scaled them to a resolution of 40 × 40 to match those of Syn-Signs.MNIST ↔ SVHN The MNIST images were padded to 32 × 32 resolution and converted to RGB by replicating the greyscale channel into the three RGB channels to match the format of SVHN.B SMALL IMAGE EXPERIMENT TRAINING B.1 TRAINING PROCEDURE Our networks were trained for 300 epochs. We used the Adam BID12 gradient descent algorithm with a learning rate of 0.001. We trained using mini-batches composed of 256 samples, except in the Syn-digits → SVHN and Syn-signs → GTSRB experiments where we used 128 in order to reduce memory usage. The self-ensembling loss was weighted by a factor of 3 and the class balancing loss was weighted by 0.005. Our teacher network weights t i were updated so as to be an exponential moving average of those of the student s i using the formula t i = αt i−1 + (1 − α)s i , with a value of 0.99 for α. A complete pass over the target dataset was considered to be one epoch in all experiments except the MNIST → USPS and CIFAR-10 → STL experiments due to the small size of the target datasets, in which case one epoch was considered to be a pass over the larger soure dataset.We found that using the proportion of samples that passed the confidence threshold can be used to drive early stopping BID18 ). The final score was the target test set performance at the epoch at which the highest confidence threshold pass rate was obtained.C VISDA-17 C.1 HYPER-PARAMETERS Our training procedure was the same as that used in the small image experiments, except that we used 160 × 160 images, a batch size of 56 (reduced from 64 to fit within the memory of an nVidia 1080-Ti), a self-ensembling weight of 10 (instead of 3), a confidence threshold of 0.9 (instead of 0.968) and a class balancing weight of 0.01. We used the Adam BID12 gradient descent algorithm with a learning rate of 10 −5 for the final two randomly initialized layers and 10 DISPLAYFORM0 for the pre-trained layers. The first convolutional layer and the first group of convolutional layers (with 64 feature channels) of the pre-trained ResNet were left unmodified during training.Reduced data augmentation:• scale image so that its smallest dimension is 176 pixels, then randomly crop a 160 × 160 section from the scaled image • No random affine transformations as they increase confusion between the car and truck classes in the validation set • random uniform scaling in the range [0.75, 1.333]• horizontal flipping Competition data augmentation adds the following in addition to the above:• random intensity/brightness scaling in the range [0.75, 1.333]• random rotations, normally distributed with a standard deviation of 0.2π• random desaturation in which the colours in an image are randomly desaturated to greyscale by a factor between 0% and 100% • rotations in colour space, around a randomly chosen axes with a standard deviation of 0.05π• random offset in colour space, after standardisation using parameters specified by 10 × 10 × 192 Dropout, 50%10 × 10 × 192 Conv 3 × 3 × 384, pad 1, batch norm 10 × 10 × 384 Conv 3 × 3 × 384, pad 1, batch norm 10 × 10 × 384 Conv 3 × 3 × 384, pad 1, batch norm 10 × 10 × 384 Max-pool, 2x25 × 5 × 384 Dropout, 50%5 × 5 × 384 Global pooling layer 1 × 1 × 384 Fully connected, 43 units, softmax 43 Table 8 : Syn-signs → GTSRB architecture
Self-ensembling based algorithm for visual domain adaptation, state of the art results, won VisDA-2017 image classification domain adaptation challenge.
{"src": "scitldr-AIC_train", "paper_id": "rkpoTaxA-"}