The use of deep learning for a wide range of data problems has increased the need for understanding and diagnosing these models, and deep learning interpretation techniques have become an essential tool for data analysts. Although numerous model interpretation methods have been proposed in recent years, most of these procedures are based on heuristics with little or no theoretical guarantees. In this work, we propose a statistical framework for saliency estimation for black box computer vision models. We build a model-agnostic estimation procedure that is statistically consistent and passes the saliency checks of Adebayo et al. (2018). Our method requires solving a linear program, whose solution can be efficiently computed in polynomial time. Through our theoretical analysis, we establish an upper bound on the number of model evaluations needed to recover the region of importance with high probability, and build a new perturbation scheme for estimation of local gradients that is shown to be more efficient than the commonly used random perturbation schemes. Validity of the new method is demonstrated through sensitivity analysis.
 Deep learning models have achieved great predictive performance in many tasks. However, these complex, often un-tractable models are difficult to interpret and understand. This lack of interpretability is a major barrier for their wide adoption, especially in domains (e.g., medicine) where models need to be qualitatively understood and/or verified for robustness. In order to address these issues, several interpretation approaches have been proposed in the last few years. A group of methods are based on visualizations, either by quantifying the effect of particular neurons or features, or by creating new images that maximize the target score for specific classes (Erhan et al., 2009; Simonyan et al., 2013; Zeiler & Fergus, 2014) . A large collection of the techniques build saliency maps by attributing the gradients of the neural network to the input image through various procedures or by finding perturbations that significantly change the output (Springenberg et al., 2014; Bach et al., 2015; Montavon et al., 2017; Shrikumar et al., 2017; Zhou et al., 2016; Selvaraju et al., 2017; Smilkov et al., 2017; Fong & Vedaldi, 2017; Adebayo et al., 2018a; Dumitru et al., 2018; Singla et al., 2019) . Another class of approaches treat the deep learner as a black-box. In this domain, Baehrens et al. (2010) use a Parzen window classifier to approximate the target classifier locally. Ribeiro et al. (2016) propose the LIME procedure, where small perturbations on the instance are used to obtain additional samples with which a sparse linear model is fit. Lundberg & Lee (2017) propose SHapley Additive exPlanation(SHAP), which combines the Shapley value from the game theory with the additive feature attribution methods. They also make connections of the SHAP procedure with various existing methods including LRP, LIME and DeepLIFT. Chen et al. (2019) propose L-and C-Shapley procedures which can reliably approximate the Shapley values in linear time with respect to the number of features. Majority of the listed methods are heuristics which are constructed according to certain desirable qualities. For these methods, it is not clear what the main estimand is, if it can be consistently estimated or if (and how) the estimand can be computed more efficiently. In fact, according to the recent research by Adebayo et al. (2018b) , most methods with great visual inspection lack sensitivity to the model and the data generating process. Theoretical explanation for why guided back-propagation and deconvolutional methods perform image recovery is provided by Nie et al. (2018) . In this work, we propose a statistically valid technique for model-agnostic saliency estimation, and prove its consistency under reasonable assumptions. Furthermore, our method passes the sanity checks given by Adebayo et al. (2018b) . Through our analysis, we obtain insights into how to improve the accuracy and reliability of our approach. We note that there is recent work by Burns et al. (2019) where they provide a saliency estimation technique with theoretical guarantees -more specifically, FDR control. Although their procedure is very promising from a statistical perspective, and theoretically valid under a very general set of assumptions, their technique requires human input and has a significant computational load as it uses a generative model for filling in certain regions of the target image. Our main contributions are as follows: • We introduce a new saliency estimation framework for CNNs and propose a new method based on input perturbation. Our procedure requires solving a linear program, and hence the estimates can be computed very efficiently. Furthermore, the optimization problem can be recast as a "parametric simplex" (Vanderbei, 2014) , which allows the computation of the full solution path in an expedient manner. • We establish conditions under which the significant pixels in the input can be identified with high probability. We present finite-sample convergence rates that can be used to determine the number of necessary model evaluations. • We find that the noise distribution for the perturbation has a substantial effect on the convergence rate. We propose a new perturbation scheme which uses a highly correlated Gaussian, instead of the widely used independent Gaussian distribution. In the following section, we define the linearly estimated gradient (LEG), which is the saliency parameter of interest (i.e. the estimand), and introduce our statistical framework. In section 3, we propose a regularized estimation procedure for LEG that penalizes the anisotropic total-variation. We provide our theoretical results in Section 4 and the result of our numerical comparisons in Section 5. We have proposed a statistical framework for saliency estimation that relies on local linear approximations. Utilizing the new framework, we have built a computationally efficient saliency estimator that has theoretical guarantees. Using our theoretical analysis, we have identified how the sample complexity of the estimator can be improved by altering the model evaluation scheme. Finally, we have shown through empirical studies that (i) unlike most of its competitors, our method passes the recently proposed sanity checks for saliency estimation; and (ii) pixels identified through our approach are highly relevant for the predictions, and our method often chooses regions with higher saliency compared to regions suggested by its alternatives. Our linear program can also be recast by a change of variables and setting α = Dg. In this case, the elements of α correspond to differences between adjoint pixels. This program can be written as: + is the pseudo-inverse of D and U 2 is related to the left singular vectors of D. More precisely, letting D = U ΘV T denote the singular value decomposition of D, U 2 is the submatrix that corresponds to the columns of U for which Θ j is zero. The linearity constraint ensures that the differences between the adjoint pixels is proper. Derivation of the alternative formulation follows from Theorem 1 in Gaines et al. (2018) and is omitted. This formulation can be expressed in the standard augmented form, i.e. min Ax=b,x≥0 c T x, by writ- where y = 1 n n i=1f (x i )x i and m = 2p 1 p 2 −p 1 −p 2 . The γ coefficient in the original formulation can be obtained by setting A.2 PROOF OF THEOREM 1 Our proof depends on the following lemma. Lemma 2. For L ≥ 2 D + 1 log (p 1 p 2 / ) /n, γ * is in the feasibility set with probability 1 − , that is Proof. For ease of notation, let We also assume that the images have been rescaled so that the maximum value ofx i is 1 (without rescaling, the maximum would be given as the largest intensity, i.e. 255). Since, the function values are also in the range given by [-2,2], we can bound |z i,j |, that is Under review as a conference paper at ICLR 2020 The proof follows by applying the McDiarmid's inequality (Vershynin, 2018) for each row of the difference and then taking the supremum over the terms. By application of McDiarmid's inequality, we have that Let L = 2 D + 1 log (p 1 p 2 /2 ) /n. Then, taking a union bound over all variables, we have Now note that that the feasibility set for any L ≥ L contains that of L and thus γ * is automatically included. We now present the proof of the theorem. Note that the technique is based on the Confidence Set approach by Fan (2013) . In the proof, we use γ to refer to vec(γ) for ease of presentation. Proof. First, let the high probability set for which Lemma 2 holds by A. All of the following statements hold true for A. We let ∆ = D (γ − γ * ) . We know that Dγ 1 ≤ Dγ * 1 since both are in the feasibility set, as stated in Lemma 2. Let α * = Dγ * ,α = Dγ and define S = {j : α * j = 0}, and the complement of S as S C . By assumption of the Theorem, we have that the cardinality of S is s, i.e. |S| = s. Now let ∆ S as the elements of ∆ in S. Then, using the above statement, one can show that ∆ S 1 ≥ ∆ S C 1 . Note, and ∆ S 1 ≥ ∆ S C 1 follows immediately. Furthermore where the last line uses the previous result. Additionally, note that where the first inequality follows by Holder's inequality and the second follows from Lemma 2 and the fact that bothγ and γ * are in the feasibility set for L = 2 D + 1 log (p 1 p 2 / ) /n. We further bound the right hand side of the inequality by using the previous result, which gives Next, we bound ∆ 2 by combining the previous results. Now, by assumption of the Theorem, we have that Dividing both sides by ∆ 2 , we obtain that
We propose a statistical framework and a theoretically consistent procedure for saliency estimation.
{"src": "scitldr-AIC_train", "paper_id": "BJlrZyrKDB"}