This paper presents two methods to disentangle and interpret contextual effects that are encoded in a pre-trained deep neural network. Unlike convolutional studies that visualize image appearances corresponding to the network output or a neural activation from a global perspective, our research aims to clarify how a certain input unit (dimension) collaborates with other units (dimensions) to constitute inference patterns of the neural network and thus contribute to the network output. The analysis of local contextual effects w.r.t. certain input units is of special values in real applications. In particular, we used our methods to explain the gaming strategy of the alphaGo Zero model in experiments, and our method successfully disentangled the rationale of each move during the game. Interpreting the decision-making logic hidden inside neural networks is an emerging research direction in recent years. The visualization of neural networks and the extraction of pixel-level inputoutput correlations are two typical methodologies. However, previous studies usually interpret the knowledge inside a pre-trained neural network from a global perspective. For example, BID17 BID14 BID10 mined input units (dimensions or pixels) that the network output is sensitive to; BID2 visualized receptive fields of filters in intermediate layers; BID33 BID15 BID24 BID5 BID6 BID20 illustrated image appearances that maximized the score of the network output, a filter's response, or a certain activation unit in a feature map.However, instead of visualizing the entire appearance that is responsible for a network output or an activation unit, we are more interested in the following questions.• How does a local input unit contribute to the network output? Here , we can vectorize the input of the network into a high-dimensional vector, and we treat each dimension as a specific "unit" without ambiguity. As we know, a single input unit is usually not informative enough to make independent contributions to the network output. Thus , we need to clarify which other input units the target input unit collaborates with to constitute inference patterns of the neural network, so as to pass information to high layers.• Can we quantitatively measure the significance of above contextual collaborations between the target input unit and its neighboring units?Method : Therefore, given a pre-trained convolutional neural network (CNN), we propose to disentangle contextual effects w.r.t. certain input units.As shown in Fig. 1 , we design two methods to interpret contextual collaborations at different scales, which are agnostic to the structure of CNNs. The first method estimates a rough region of contextual collaborations, i.e. clarifying whether the target input unit mainly collaborates with a few neighboring units or most units of the input. This method distills knowledge from the pre-trained network into a mixture of local models (see Fig. 2 ), where each model encodes contextual collaborations within a specific input region to make predictions. We hope that the knowledge-distillation strategy can help people determine quantitative contributions from different regions. Then, given a model for Extracting fine-grained contextual effects from a student net A lattice within the Go board Figure 1 : Explaining the alphaGo model. Given the state of the Go board and the next move, we use the alphaGo model to explain the rationale of the move. We first estimate a rough region of contextual collaborations w.r.t. the current move by distilling knowledge from the value net to student nets that receive different regions of the Go board as inputs. Then, given a student net, we analyze fine-grained contextual collaborations within its region of the Go board. In this figure, we use a board state from a real Go game between humans for clarity.local collaborations, the second method further analyzes the significance of detailed collaborations between each pair of input units, when we use the local model to make predictions on an image. In this paper, we have proposed two typical methods for quantitative analysis of contextual collaborations w.r.t. a certain input unit in the decision-making of a neural network. Extracting fine-grained contextual collaborations to clarify the reason why and how an input unit passes its information to the network output is of significant values in specific applications, but it has not been well explored before, to the best of our knowledge. In particular, we have applied our methods to the alphaGo Zero model, in order to explain the potential logic hidden inside the model that is automatically learned via self-play without human annotations. Experiments have demonstrated the effectiveness of the proposed methods.Note that there is no exact ground-truth for contextual collaborations of the Go game, and how to evaluate the quality of the extracted contextual collaborations is still an open problem. As a pioneering study, we do not require the explanation to be exactly fit human logics, because human logic is usually not the only correct explanations. Instead, we just aim to visualize contextual collaborations without manually pushing visualization results towards human-interpretable concepts. This is different from some previous studies of network visualization BID15 BID32 that added losses as the natural image prior, in order to obtain beautiful but biased visualization results. In the future, we will continue to cooperate with professional Go players to further refine the algorithm to visualize more accurate knowledge inside the alphaGo Zero model.
This paper presents methods to disentangle and interpret contextual effects that are encoded in a deep neural network.
{"src": "scitldr-AIC_train", "paper_id": "rJGgFjA9FQ"}