Interpreting neural networks is a crucial and challenging task in machine learning. In this paper, we develop a novel framework for detecting statistical interactions captured by a feedforward multilayer neural network by directly interpreting its learned weights. Depending on the desired interactions, our method can achieve significantly better or similar interaction detection performance compared to the state-of-the-art without searching an exponential solution space of possible interactions. We obtain this accuracy and efficiency by observing that interactions between input features are created by the non-additive effect of nonlinear activation functions, and that interacting paths are encoded in weight matrices. We demonstrate the performance of our method and the importance of discovered interactions via experimental results on both synthetic datasets and real-world application datasets. Despite their strong predictive power, neural networks have traditionally been treated as "black box" models, preventing their adoption in many application domains. It has been noted that complex machine learning models can learn unintended patterns from data, raising significant risks to stakeholders BID43 . Therefore, in applications where machine learning models are intended for making critical decisions, such as healthcare or finance, it is paramount to understand how they make predictions BID6 BID17 .Existing approaches to interpreting neural networks can be summarized into two types. One type is direct interpretation, which focuses on 1) explaining individual feature importance, for example by computing input gradients BID37 BID34 BID40 or by decomposing predictions BID2 BID36 , 2) developing attention-based models, which illustrate where neural networks focus during inference BID20 BID27 BID47 , and 3) providing model-specific visualizations, such as feature map and gate activation visualizations BID48 BID21 . The other type is indirect interpretation, for example post-hoc interpretations of feature importance BID32 and knowledge distillation to simpler interpretable models BID7 .It has been commonly believed that one major advantage of neural networks is their capability of modeling complex statistical interactions between features for automatic feature learning. Statistical interactions capture important information on where features often have joint effects with other features on predicting an outcome. The discovery of interactions is especially useful for scientific discoveries and hypothesis validation. For example, physicists may be interested in understanding what joint factors provide evidence for new elementary particles; doctors may want to know what interactions are accounted for in risk prediction models, to compare against known interactions from existing medical literature.In this paper, we propose an accurate and efficient framework, called Neural Interaction Detection (NID), which detects statistical interactions of any order or form captured by a feedforward neural network, by examining its weight matrices. Our approach is efficient because it avoids searching over an exponential solution space of interaction candidates by making an approximation of hidden unit importance at the first hidden layer via all weights above and doing a 2D traversal of the input weight matrix. We provide theoretical justifications on why interactions between features are created at hidden units and why our hidden unit importance approximation satisfies bounds on hidden unit gradients. Top-K true interactions are determined from interaction rankings by using a special form of generalized additive model, which accounts for interactions of variable order BID46 BID25 . Experimental results on simulated datasets and real-world datasets demonstrate the effectiveness of NID compared to the state-of-the-art methods in detecting statistical interactions.The rest of the paper is organized as follows: we first review related work and define notations in Section 2. In Section 3, we examine and quantify the interactions encoded in a neural network, which leads to our framework for interaction detection detailed in Section 4. Finally, we study our framework empirically and demonstrate its practical utility on real-world datasets in Section 5. We presented our NID framework, which detects statistical interactions by interpreting the learned weights of a feedforward neural network. The framework has the practical utility of accurately detecting general types of interactions without searching an exponential solution space of interaction candidates. Our core insight was that interactions between features must be modeled at common hidden units, and our framework decoded the weights according to this insight.In future work, we plan to detect feature interactions by accounting for common units in intermediate hidden layers of feedforward networks. We would also like to use the perspective of interaction detection to interpret weights in other deep neural architectures.A PROOF AND DISCUSSION FOR PROPOSITION 2Given a trained feedforward neural network as defined in Section 2.3, we can construct a directed acyclic graph G = (V, E) based on non-zero weights as follows. We create a vertex for each input feature and hidden unit in the neural network: V = {v ,i |∀i, }, where v ,i is the vertex corresponding to the i-th hidden unit in the -th layer. Note that the final output y is not included. We create edges based on the non-zero entries in the weight matrices, i.e., DISPLAYFORM0 Note that under the graph representation, the value of any hidden unit is a function of parent hidden units. In the following proposition, we will use vertices and hidden units interchangeably. Proposition 2 (Interactions at Common Hidden Units). Consider a feedforward neural network with input feature DISPLAYFORM1 , there exists a vertex v I in the associated directed graph such that I is a subset of the ancestors of v I at the input layer (i.e., = 0).Proof . We prove Proposition 2 by contradiction.Let I be an interaction where there is no vertex in the associated graph which satisfies the condition. Then, for any vertex v L,i at the L-th layer, the value f i of the corresponding hidden unit is a function of its ancestors at the input layer I i where I ⊂ I i .Next, we group the hidden units at the L-th layer into non-overlapping subsets by the first missing feature with respect to the interaction I. That is, for element i in I, we create an index set S i ∈ [p L ]: DISPLAYFORM2 Note that the final output of the network is a weighed summation over the hidden units at the L-th layer: DISPLAYFORM3 Since that j∈Si w y j f j x Ij is not a function of x i , we have that ϕ (·) is a function without the interaction I, which contradicts our assumption.The reverse of this statement, that a common descendant will create an interaction among input features, holds true in most cases. The existence of counterexamples is manifested when early hidden layers capture an interaction that is negated in later layers. For example, the effects of two interactions may be directly removed in the next layer, as in the case of the following expression: max{w 1 x 1 + w 2 x 2 , 0} − max{−w 1 x 1 − w 2 x 2 , 0} = w 1 x 1 + w 2 x 2 . Such an counterexample is legitimate ; however, due to random fluctuations, it is highly unlikely in practice that the w 1 s and the w 2 s from the left hand side are exactly equal.
We detect statistical interactions captured by a feedforward multilayer neural network by directly interpreting its learned weights.
{"src": "scitldr-AIC_train", "paper_id": "ByOfBggRZ"}