Models of models, or surrogate models, can be helpful explanatory tools, but they are usually approximate, low-fidelity explainers. Aside from (1) a global or local summary of a complex model provided by a surrogate model can be helpful sometimes and (2) much work in explainable ML has been directed toward improving the fidelity and usefulness of surrogate models [1]}, [2]}, [3]}, [4]}, [5]}, many explainable ML techniques have nothing to do with surrogate models. One of the most exciting breakthroughs for supervised learning problems in explainable ML is the application of a coalitional game theory concept, Shapley values, to compute feature attributions which are consistent globally and accurate locally using the trained model itself [6]}, [7]}. An extension of this idea, called Tree SHAP, has already been implemented for popular tree ensemble methods [8]}.
<FIGURE>