input. We denote the input of \(i\)-th layer in Transformer (Vaswani et al., 2017) as \(h_{i}\), with the corresponding output represented as \(h_{i+1}=T_{i}(h_{i})\), where \(T_{i}\) represents the parameters of the \(i\)-th layer. For a specific neuron, denoted as \(N_{k}^{(i)}\), within the \(i\)-th layer--whether located in the attention or feed-forward layer--the importance is quantified as the difference between output when \(N_{k}^{(i)}\) is either activated or deactivated.