This paper presents a method to explain the knowledge encoded in a convolutional neural network (CNN) quantitatively and semantically. How to analyze the specific rationale of each prediction made by the CNN presents one of key issues of understanding neural networks, but it is also of significant practical values in certain applications. In this study, we propose to distill knowledge from the CNN into an explainable additive model, so that we can use the explainable model to provide a quantitative explanation for the CNN prediction. We analyze the typical bias-interpreting problem of the explainable model and develop prior losses to guide the learning of the explainable additive model. Experimental results have demonstrated the effectiveness of our method. Convolutional neural networks (CNNs) BID17 BID15 BID10 have achieved superior performance in various tasks, such as object classification and detection. Besides the discrimination power of neural networks, the interpretability of neural networks has received an increasing attention in recent years.In this paper, we focus on a new problem, i.e. explaining the specific rationale of each network prediction semantically and quantitatively. "Semantic explanations" and "quantitative explanations" are two core issues of understanding neural networks. In this paper, we focus on a new task, i.e. explaining the logic of each CNN prediction semantically and quantitatively, which presents considerable challenges in the scope of understanding neural networks. We propose to distill knowledge from a pre-trained performer into an interpretable additive explainer. We can consider that the performer and the explainer encode similar knowledge. The additive explainer decomposes the prediction score of the performer into value components from semantic visual concepts, in order to compute quantitative contributions of different concepts. The strategy of using an explainer for explanation avoids decreasing the discrimination power of the performer. In preliminary experiments, we have applied our method to different benchmark CNN performers to prove the broad applicability.Note that our objective is not to use pre-trained visual concepts to achieve super accuracy in classification/prediction. Instead, the explainer uses these visual concepts to mimic the logic of the performer and produces similar prediction scores as the performer.In particular, over-interpreting is the biggest challenge of using an additive explainer to interpret another neural network. In this study, we design two losses to overcome the bias-interpreting problems. Besides, in experiments, we also measure the amount of the performer knowledge that could not be represented by visual concepts in the explainer. Table 4 : Classification accuracy of the explainer and the performer. We use the the classification accuracy to measure the information loss when using an explainer to interpret the performer. Note that the additional loss for bias-interpreting successfully overcame the bias-interpreting problem, but did not decrease the classification accuracy of the explainer. Another interesting finding of this research is that sometimes, the explainer even outperformed the performer in classification. A similar phenomenon has been reported in BID9 . A possible explanation for this phenomenon is given as follows. When the student network in knowledge distillation had sufficient representation power, the student network might learn better representations than the teacher network, because the distillation process removed abnormal middle-layer features corresponding to irregular samples and maintained common features, so as to boost the robustness of the student network. Table 5 : Relative deviations of the explainer. The additional loss for bias-interpreting successfully overcame the bias-interpreting problem and just increased a bit (ignorable) relative deviation of the explainer. BID40 ) used a tree structure to summarize the inaccurate rationale of each CNN prediction into generic decision-making models for a number of samples. This method assumed the significance of a feature to be proportional to the Jacobian w.r.t. the feature, which is quite problematic. This assumption is acceptable for BID40 , because the objective of BID40 ) is to learn a generic explanation for a group of samples, and the inaccuracy in the explanation for each specific sample does not significantly affect the accuracy of the generic explanation. In comparisons, our method focuses on the quantitative explanation for each specific sample, so we design an additive model to obtain more convincing explanations. Baseline Our method Figure 4 : We compared the contribution distribution of different visual concepts (filters) that was estimated by our method and the distribution that was estimated by the baseline. The baseline usually used very few visual concepts to make predictions, which was a typical case of bias-interpreting. In comparisons, our method provided a much more reasonable contribution distribution of visual concepts. Legs & feet Tail Figure 9 : Quantitative explanations for object classification. We assigned contributions of filters to their corresponding object parts, so that we obtained contributions of different object parts. According to top figures, we found that different images had similar explanations, i.e. the CNN used similar object parts to classify objects. Therefore, we showed the grad-CAM visualization of feature maps BID24 on the bottom, which proved this finding. We visualized interpretable filters in the top conv-layer of a CNN, which were learned based on BID38 . We projected activation regions on the feature map of the filter onto the image plane for visualization. Each filter represented a specific object part through different images. BID38 ) learned a CNN, where each filter in the top conv-layer represented a specific object part. Thus, we annotated the name of the object part that corresponded to each filter based on visualization results (see FIG4 for examples). We simply annotate each filter of the top conv-layer in a performer once, so the total annotation cost was O(N ), where N is the filter number.
This paper presents a method to explain the knowledge encoded in a convolutional neural network (CNN) quantitatively and semantically.
{"src": "scitldr-AIC_train", "paper_id": "SJfWKsC5K7"}