Unsupervised visual representation learning remains a largely unsolved problem in computer vision research. Among a big body of recently proposed approaches for unsupervised learning of visual representations, a class of self-supervised techniques achieves superior performance on many challenging benchmarks. A large number of the pretext tasks for self-supervised learning have been studied, but other important aspects, such as the choice of convolutional neural networks (CNN), has not received equal attention. Therefore, we revisit numerous previously proposed self-supervised models, conduct a thorough large scale study and, as a result, uncover multiple crucial insights. We challenge a number of common practices in selfsupervised visual representation learning and observe that standard recipes for CNN design do not always translate to self-supervised representation learning. As part of our study, we drastically boost the performance of previously proposed techniques and outperform previously published state-of-the-art results by a large margin.
Finally, many works have tried to combine multiple pretext tasks in one way or another. For instance, Kim al extend the jigsaw puzzle'' task by combining it with colorization and inpainting in @cite_17 . Combining the jigsaw puzzle task with clustering-based pseudo labels as in @cite_39 leads to the method called Jigsaw++ @cite_32 . Doersch and Zisserman @cite_5 implement four different self-supervision methods and make one single neural network learn all of them in a multi-task setting.
@cite_5 @cite_32 @cite_39 @cite_17
2963826423 2624862751 2962824366 2963749571
In this paper, we explore methods of complicating selfsupervised tasks for representation learning. That is, we do severe damage to data and encourage a network to recover them. First, we complicate each of three powerful self-supervised task candidates: jigsaw puzzle, inpainting, and colorization. In addition, we introduce a novel complicated self-supervised task called "Completing damaged jigsaw puzzles" which is puzzles with one piece missing and the other pieces without color. We train a convolutional neural network not only to solve the puzzles, but also generate the missing content and colorize the puzzles. The recovery of the aforementioned damage pushes the network to obtain robust and general-purpose representations. We demonstrate that complicating the self-supervised tasks improves their original versions and that our final task learns more robust and transferable representations compared to the previous methods, as well as the simple combination of our candidate tasks. Our approach achieves state-of-the-art performance in transfer learning on PASCAL classification and semantic segmentation. Jigsaw puzzle algorithm is important as it can be applied to many areas such as biology, image editing, archaeology and incomplete crime-scene reconstruction. But, still, some problems exist in the process of practical application, for example, when there are a large number of similar objects in the puzzle fragments, the error rate will reach 30 –50 . When some fragments are missing, most algorithms fail to restore the images accurately. When the number of fragments of the jigsaw puzzle is large, efficiency is reduced. During the intelligent puzzle, mainly the Sum of Squared Distance Scoring (SSD), Mahalanobis Gradient Compatibility (MGC) and other metrics are used to calculate the similarity between the fragments. On the basis of these two measures, we put forward some new methods: 1. MGC is one of the most effective measures, but using MGC to reassemble the puzzle can cause an error image every 30 or 50 times, so we combine the Jaccard and MGC metric measure to compute the similarity between the image fragments, and reassemble the puzzle with a greedy algorithm. This algorithm not only reduces the error rate, but can also maintain a high accuracy in the case of a large number of fragments of similar objects. 2. For the lack of fragmentation and low efficiency, this paper uses a new method of SSD measurement and mark matrix, it is general in the sense that it can handle puzzles of unknown size, with fragments of unknown orientation, and even puzzles with missing fragments. The algorithm does not require any preset conditions and is more practical in real life. Finally, experiments show that the algorithm proposed in this paper improves not only the accuracy but also the efficiency of the operation. We investigate methods for combining multiple selfsupervised tasks—i.e., supervised tasks where data can be collected without manual labeling—in order to train a single visual representation. First, we provide an apples-toapples comparison of four different self-supervised tasks using the very deep ResNet-101 architecture. We then combine tasks to jointly train a network. We also explore lasso regularization to encourage the network to factorize the information in its representation, and methods for “harmonizing” network inputs in order to learn a more unified representation. We evaluate all methods on ImageNet classification, PASCAL VOC detection, and NYU depth prediction. Our results show that deeper networks work better, and that combining tasks—even via a na¨ýve multihead architecture—always improves performance. Our best joint network nearly matches the PASCAL performance of a model pre-trained on ImageNet classification, and matches the ImageNet network on NYU depth prediction. Learning visual representations with self-supervised learning has become popular in computer vision. The idea is to design auxiliary tasks where labels are free to obtain. Most of these tasks end up providing data to learn specific kinds of invariance useful for recognition. In this paper, we propose to exploit different self-supervised approaches to learn representations invariant to (i) inter-instance variations (two objects in the same class should have similar features) and (ii) intra-instance variations (viewpoint, pose, deformations, illumination, etc.). Instead of combining two approaches with multi-task learning, we argue to organize and reason the data with multiple variations. Specifically, we propose to generate a graph with millions of objects mined from hundreds of thousands of videos. The objects are connected by two types of edges which correspond to two types of invariance: “different instances but a similar viewpoint and category” and “different viewpoints of the same instance”. By applying simple transitivity on the graph with these edges, we can obtain pairs of images exhibiting richer visual invariance. We use this data to train a Triplet-Siamese network with VGG16 as the base architecture and apply the learned representations to different recognition tasks. For object detection, we achieve 63.2 mAP on PASCAL VOC 2007 using Fast R-CNN (compare to 67.3 with ImageNet pre-training). For the challenging COCO dataset, our method is surprisingly close (23.5 ) to the ImageNet-supervised counterpart (24.4 ) using the Faster R-CNN framework. We also show that our network can perform significantly better than the ImageNet network in the surface normal estimation task.