Previous visual explanation works mainly focus on the interpretation of deep neural network for classification [1]}, [2]}, [3]}, [4]}. Guided back propagation (guided BP) [1]} has been used for explanation by generating the gradient from prediction to input, which shows how much the output will change with a little change in each dimension of the input. Another representative visual explanation approach, class activation map (CAM) [2]}, generates the heatmap of discriminative regions corresponding to a specific class based on the linearity of global average pooling (GAP) and fully connected (FC) layer. However, the original method only works on this specific architecture configuration (i.e. GAP+FC) and needs re-training for visualizing other applications. Based on the gradient of the last convolutional layer instead of the input, Grad-CAM [7]} is proposed to generate activation maps for all convolutional neural network (CNN) architectures. Besides, other existing methods explore network ablation [3]}, the winner-take-all strategy [9]}, inversion [10]}, and perturbation [4]} for visual explanation.
