In this paper, we propose a novel method to incorporate partial evidence in the inference of deep convolutional neural networks. Contrary to the existing, top performing methods, which either iteratively modify the input of the network or exploit external label taxonomy to take the partial evidence into account, we add separate network modules ("Plugin Networks") to the intermediate layers of a pre-trained convolutional network. The goal of these modules is to incorporate additional signal, ie information about known labels, into the inference procedure and adjust the predicted output accordingly. Since the attached plugins have a simple structure, consisting of only fully connected layers, we drastically reduced the computational cost of training and inference. At the same time, the proposed architecture allows to propagate information about known labels directly to the intermediate layers to improve the final representation. Extensive evaluation of the proposed method confirms that our Plugin Networks outperform the state-of-the-art in a variety of tasks, including scene categorization, multi-label image annotation, and semantic segmentation.
The FeedbackProp proposed by Wang al @cite_7 , on the other hand, uses an iterative procedure which is applied at inference time. The idea is to modify network activations to maximize probabilities of labels under the partial evidence. The method does not require to re-train base model. However, due to iterative procedure introduce at inference time it requires more computational effort. In addition, they introduce hyperparameters like a number of iterations and learning rate to inference phase. Finally, in the case of FeedbackProp, the partial evidence labels can only be a subset of labels that base model can recognize. Our method, however, can accept any kind of labels as partial evidence. In addition, our method does not introduce any additional computations nor additional parameters to inference phase. The comparison shows that our method outperforms FeedbackProp while being an significantly faster at inference phase.
@cite_7
2963175631
We propose an inference procedure for deep convolutional neural networks (CNNs) when partial evidence is available. Our method consists of a general feedback-based propagation approach (feedback-prop) that boosts the prediction accuracy for an arbitrary set of unknown target labels when the values for a non-overlapping arbitrary set of target labels are known. We show that existing models trained in a multi-label or multi-task setting can readily take advantage of feedback-prop without any retraining or fine-tuning. Our feedback-prop inference procedure is general, simple, reliable, and works on different challenging visual recognition tasks. We present two variants of feedback-prop based on layer-wise and residual iterative updates. We experiment using several multi-task models and show that feedback-prop is effective in all of them. Our results unveil a previously unreported but interesting dynamic property of deep CNNs. We also present an associated technical approach that takes advantage of this property for inference under partial evidence in general visual recognition tasks.