Conventionally, convolutional neural networks (CNNs) process different images with the same set of filters. However, the variations in images pose a challenge to this fashion. In this paper, we propose to generate sample-specific filters for convolutional layers in the forward pass. Since the filters are generated on-the-fly, the model becomes more flexible and can better fit the training data compared to traditional CNNs. In order to obtain sample-specific features, we extract the intermediate feature maps from an autoencoder. As filters are usually high dimensional, we propose to learn a set of coefficients instead of a set of filters. These coefficients are used to linearly combine the base filters from a filter repository to generate the final filters for a CNN. The proposed method is evaluated on MNIST, MTFL and CIFAR10 datasets. Experiment results demonstrate that the classification accuracy of the baseline model can be improved by using the proposed filter generation method. Variations exist widely in images. For example, in face images, faces present with different head poses and different illuminations which are challenges to most face recognition models. In the conventional training process of CNNs, filters are optimized to deal with different variations. The number of filters increases if more variations are added to the input data. However, for a test image, only a small number of the neurons in the network are activated which indicates inefficient computation BID13 ).Unlike CNNs with fixed filters, CNNs with dynamically generated sample-specific filters are more flexible since each input image is associated with a unique set of filters. Therefore , it provides possibility for the model to deal with variations without increasing model size.However, there are two challenges for training CNNs with dynamic filter generation. The first challenge is how to learn sample-specific features for filter generation. Intuitively , filter sets should correspond to variations in images. If the factors of variations are restricted to some known factors such as face pose or illumination, we can use the prior knowledge to train a network to represent the variation as a feature vector. The main difficulty is that besides the factors of variations that we have already known, there are also a number of them that we are not aware of. Therefore, it is difficult to enumerate all the factors of variations and learn the mapping in a supervised manner. The second challenge is that how to map a feature vector to a set of new filters. Due to the high dimension of the filters, a direct mapping needs a large number of parameters which can be infeasible in real applications.In response, we propose to use an autoencoder for variation representation leaning. Since the objective of an autoencoder is to reconstruct the input images from internal feature representations, each layer of the encoder contains sufficient information about the input image. Therefore, we extract features from each layer in the encoder as sample-specific features. For the generation of filters , given a sample-specific feature vector, we firstly construct a filter repository. Then we learn a matrix that maps the feature vector to a set of coefficients which will be used to linearly combine the base filters in the repository to generate new filters.Our model has several elements of interest. Firstly, our model bridges the gap between the autoencoder network and the prediction network by mapping the autoencoder features to the filters in the prediction network. Therefore, we embed the knowledge from unsupervised learning to supervised learning. Secondly, instead of generating new filters directly from a feature vector, we facilitate the generation with a filter repository which stores a small number of base filters. Thirdly, we use linear combination of the base filters in the repository to generate new filters. It can be easily implemented as a convolution operation so that the whole pipeline is differentiable with respect to the model parameters. In this paper, we propose to learn to generate filters for convolutional neural networks. The filter generation module transforms features from an autoencoder network to sets of coefficients which are then used to linearly combine base filters in filter repositories. Dynamic filters increase model capacity so that a small model with dynamic filters can also be competitive to a deep model. Evaluation on three tasks show the accuracy improvement brought by our filter generation. In this section, we show the details of the network structures used in our experiments.When we extract sample-specific features, we directly take the convolution feature maps (before LReLU layer) from the autoencoder network as input and feed them to the dimension reduction network. The entire process of sample-specific feature extraction is split into the autoencoder network and the dimension reduction network for the purpose of plain and straightforward illustration.
dynamically generate filters conditioned on the input image for CNNs in each forward pass 
{"src": "scitldr-AIC_train", "paper_id": "rJa90ceAb"}