Convolutional neural networks (CNNs) have been generally acknowledged as one of the driving forces for the advancement of computer vision. Despite their promising performances on many tasks, CNNs still face major obstacles on the road to achieving ideal machine intelligence. One is that CNNs are complex and hard to interpret. Another is that standard CNNs require large amounts of annotated data, which is sometimes very hard to obtain, and it is desirable to be able to learn them from few examples. In this work, we address these limitations of CNNs by developing novel, simple, and interpretable models for few-shot learn- ing. Our models are based on the idea of encoding objects in terms of visual concepts, which are interpretable visual cues represented by the feature vectors within CNNs. We first adapt the learning of visual concepts to the few-shot setting, and then uncover two key properties of feature encoding using visual concepts, which we call category sensitivity and spatial pattern. Motivated by these properties, we present two intuitive models for the problem of few-shot learning. Experiments show that our models achieve competitive performances, while being much more flexible and interpretable than alternative state-of-the-art few-shot learning methods. We conclude that using visual concepts helps expose the natural capability of CNNs for few-shot learning. After their debut BID13 BID13 have played an ever increasing role in computer vision, particularly after their triumph BID11 on the ImageNet challenge BID3 . Some researchers have even claimed that CNNs have surpassed human-level performance BID8 , although other work suggests otherwise . Recent studies also show that CNNs are vulnerable to adversarial attacks BID6 . Nevertheless, the successes of CNNs have inspired the computer vision community to develop more sophisticated models BID9 BID21 .But despite the impressive achievements of CNNs we only have limited insights into why CNNs are effective. The ever-increasing depth and complicated structures of CNNs makes them very difficult to interpret while the non-linear nature of CNNs makes it very hard to perform theoretical analysis. In addition, CNNs traditionally require large annotated datasets which is problematic for many real world applications. We argue that the ability to learn from a few examples, or few-shot learning, is a characteristic of human intelligence and is strongly desirable for an ideal machine learning system. The goal of this paper is to develop an approach to few-shot learning which builds on the successes of CNNs but which is simple and easy to interpret. We start from the intuition that objects can be represented in terms of spatial patterns of parts which implies that new objects can be learned from a few examples if they are built from parts that are already known, or which can be learned from a few examples. We recall that previous researchers have argued that object parts are represented by the convolutional layers of CNNs BID29 BID14 provided the CNNs are trained for object detection. More specifically, we will build on recent work BID23 which learns a dictionary of Visual Concepts (VCs) from CNNs representing object parts, see FIG0 . It has been shown that these VCs can be combined to detect semantic parts BID24 and, in work in preparation, can be used to represent objects using VC-Encoding (where In general, these patches roughly correspond to semantic parts of objects, e.g., the cushion of a sofa (a), the side windows of trains (b) and the wheels of bicycles (c). All VCs are referred to by their indices (e.g., VC 139). We stress that VCs are learned in an unsupervised manner and terms like"sofa cushion" are inferred by observing the closest image patches and are used to describe them informally.objects are represented by binary codes of VCs). This suggests that we can use VCs to represent new objects in terms of parts hence enabling few-shot learning.But it is not obvious that VCs, as described in BID24 , can be applied to few-shot learning. Firstly, these VCs were learned independently for each object category (e.g., for cars or for airplanes) using deep network features from CNNs which had already been trained on data which included these categories. Secondly, the VCs were learned using large numbers of examples of the object category, ranging from hundreds to thousands. By contrast, for few-shot learning we have to learn the VCs from a much smaller number of examples (by an order of magnitude or more). Moreover, we can only use deep network features which are trained on datasets which do not include the new object categories which we hope to learn. This means that although we will extract VCs using very similar algorithms to those in BID23 our motivation and problem domain are very different. To summarize, in this paper we use VCs to learn models of new object categories from existing models of other categories, while BID23 uses VCs to help understand CNNs and to perform unsupervised part detection.In Section 3, we will review VCs in detail. Briefly speaking , VCs are extracted by clustering intermediate-level raw features of CNNs, e.g., features produced by the Pool-4 layer of VGG16 BID19 . Serving as the cluster centers in feature space, VCs divide intermediate-level deep network features into a discrete dictionary. We show that VCs can be learned in the few-shot learning setting and they have two desirable properties when used for image encoding, which we call category sensitivity and spatial patterns.More specifically, we develop an approach to few-shot learning which is simple, interpretable, and flexible. We learn a dictionary of VCs as described above which enables us to represent novel objects by their VC-Encoding. Then we propose two intuitive models: (i) nearest neighbor and (ii) a factorizable likelihood model based on the VC-Encoding. The nearest neighbor model uses a similarity measure to capture the difference between two VC-Encodings. The factorizable likelihood model learns a likelihood function of the VC-Encoding which, by assuming spatial independence, can be learned form a few examples. We emphasize that both these models are very flexible, in the sense that they can be applied directly to any few-shot learning scenarios. This differs from other approaches which are trained specifically for scenarios such as 5-way 5-shot (where there are 5 new object categories with 5 examples of each). This flexibility is attractive for real world applications where the numbers of new object categories, and the number of examples of each category, will be variable. Despite their simplicity, these models achieve comparable results to the state-of-theart few-shot learning methods (using only the simplest versions of our approach), such as learning a metric and learning to learn. From a deeper perspective, our results show that CNNs have the potential for few-shot learning on novel categories but to achieve this potential requires studying the internal structures of CNNs to re-express them in simpler and more interpretable terms.Overall, our major contributions are two-fold:(1) We show that VCs can be learned in the few-shot setting using CNNs trained on other object categories. By encoding images using VCs, we observe two desirable properties, i.e., category sensitivity and spatial patterns. (2) Based on these properties, we present two simple, interpretable, and flexible models for fewshot learning. These models yield competitive results compared to the state-of-the-art methods on specific few-shot learning tasks and can also be applied directly, without additional training, to other few-shot scenarios. In this paper we address the challenge of developing simple interpretable models for few-shot learning exploiting the internal representations of CNNs. We are motivated by VCs BID23 which enable us to represent objects in terms of VC-Encodings. We show that VCs can be adapted to the few-shot learning setting where the VCs are extracted from a small set of images of novel object categories using features from CNNs trained on other object categories. We observe two properties of VC-Encoding, namely category sensitivity and spatial pattern, which leads us to propose two novel, but closely related, methods for few-shot learning which are simple, interpretable, and flexible. Our methods show comparable performances to the current state-of-the-art methods which are specialized for specific few-shot learning scenarios. We demonstrate the flexibility of our two models by showing that they can be applied to a range of different few-shot scenarios with minimal re-training. In summary, we show that VCs and VC-Encodings enable ordinary CNNs to perform few-shot learning. We emphasize that in this paper we have concentrated on developing the core ideas of our two few-shot learning models and that we have not explored variants of our ideas which could lead to better performance by exploiting standard performance enhancing tricks, or by specializing to specific few-shot challenges. Future work includes improving the quality of the extracted VCs and extending our approach to few-shot detection.
We enable ordinary CNNs for few-shot learning by exploiting visual concepts which are interpretable visual cues learnt within CNNs.
{"src": "scitldr-AIC_train", "paper_id": "BJ_QxP1AZ"}