TITLE: Two4Two: Evaluating Interpretable Machine Learning - A Synthetic Dataset For Controlled Experiments 

ABSTRACT: A growing number of approaches exist to generate explanations for image
classification. However, few of these approaches are subjected to human-subject
evaluations, partly because it is challenging to design controlled experiments
with natural image datasets, as they leave essential factors out of the
researcher's control. With our approach, researchers can describe their desired
dataset with only a few parameters. Based on these, our library generates
synthetic image data of two 3D abstract animals. The resulting data is suitable
for algorithmic as well as human-subject evaluations. Our user study results
demonstrate that our method can create biases predictive enough for a
classifier and subtle enough to be noticeable only to every second participant
inspecting the data visually. Our approach significantly lowers the barrier for
conducting human subject evaluations, thereby facilitating more rigorous
investigations into interpretable machine learning. For our library and
datasets see, https://github.com/mschuessler/two4two/
A growing number of approaches exist to generate explanations for image
classification. However, few of these approaches are subjected to human-subject
evaluations, partly because it is challenging to design controlled experiments
with natural image datasets, as they leave essential factors out of the
researcher's control. With our approach, researchers can describe their desired
dataset with only a few parameters. Based on these, our library generates
synthetic image data of two 3D abstract animals. The resulting data is suitable
for algorithmic as well as human-subject evaluations. Our user study results
demonstrate that our method can create biases predictive enough for a
classifier and subtle enough to be noticeable only to every second participant
inspecting the data visually. Our approach significantly lowers the barrier for
conducting human subject evaluations, thereby facilitating more rigorous
investigations into interpretable machine learning. For our library and
datasets see, https://github.com/mschuessler/two4two/
Two4Two: Evaluating Interpretable Machine Learning - A Synthetic Dataset For Controlled Experiments
http://arxiv.org/abs/2105.02825v1