A method that does not depend on the gradient of the model is LIME [1]}. For this reason, LIME is applicable to any model, even if the model internals are unknown. LIME generates a set of perturbed instances by removing random subsets of features from \(\mathbf {x}\)  and weighting the resulting instances with a proximity measure \(\pi _x\) . The model output on these perturbed instances is then computed, and a sparse linear model is trained to approximate the original model around \(\mathbf {x}\) . In the case of images, the instances are first converted into a “simplified representation” using a superpixel algorithm. LIME then returns an attribution value for each superpixel rather than each input dimension. This can easily be converted back to the original input space by assigning each input dimension inside a superpixel the same attribution value. KernelShap [2]} is a modification of LIME, where the perturbed instances are weighted by a specific kernel function. This kernel function causes the linear model to approximate the Shapley values of the input features, which have a number of desirable theoretical properties known as “axioms” [3]}.
