We studied the stochasticity of feature importance in the LIME  [1]} and SHAP  [2]} frameworks. Using those tools, we returned the \(n\)  most important features where \(n = 1...30\) . We iterated over all test data and recorded the \(n\)  most important features. We then evaluated the probability that each feature was one of the most influential variables in the test dataset when \(n\)  was set. We show feature importance attribution plots for LIME and SHAP where three features that were consistently picked as important for both are displayed in the bottom rows of Fig. REF  and the graphs for the rest of the features can be found in the Appendix figures REF  and REF . The horizontal axis in each of these plots shows the index of the feature in the importance ranking and the vertical axis shows the estimated probability of the feature attaining such rank. Generally, truly important features would have this probability raise quickly as a function of the rank index, and stay high. E.g., in the example shown in Fig.  REF , feature mean concave points appears slightly more important than worst concave points and worst concavity.
