Perspectives on Incorporating Expert Feedback into Model Updates
Machine learning (ML) practitioners are increasingly tasked with developing
models that are aligned with non-technical experts' values and goals. However,
there has been insufficient consideration on how practitioners should translate
domain expertise into ML updates. In this paper, we consider how to capture
interactions between practitioners and experts systematically. We devise a
taxonomy to match expert feedback types with practitioner updates. A
practitioner may receive feedback from an expert at the observation- or
domain-level, and convert this feedback into updates to the dataset, loss
function, or parameter space. We review existing work from ML and
human-computer interaction to describe this feedback-update taxonomy, and
highlight the insufficient consideration given to incorporating feedback from
non-technical experts. We end with a set of open questions that naturally arise
from our proposed taxonomy and subsequent survey.