The evaluation of interactive machine learning systems remains a difficult task. These systems learn from and adapt to the human, but at the same time, the human receives feedback and adapts to the system. Getting a clear understanding of these subtle mechanisms of co-operation and co-adaptation is challenging. In this chapter, we report on our experience in designing and evaluating various interactive machine learning applications from different domains. We argue for coupling two types of validation: algorithm-centered analysis, to study the computational behaviour of the system; and human-centered evaluation, to observe the utility and effectiveness of the application for end-users. We use a visual analytics application for guided search, built using an interactive evolutionary approach, as an exemplar of our work. Our observation is that human-centered design and evaluation complement algorithmic analysis, and can play an important role in addressing the "black-box" effect of machine learning. Finally, we discuss research opportunities that require human-computer interaction methodologies, in order to support both the visible and hidden roles that humans play in interactive machine learning.
This paper describes a new method to explore and discover within a large data set. We apply techniques from preference elicitation to automatically identify data elements that are of potential interest to the viewer. These "elements of interest (EOI)&#x201D; are bundled into spatially local clusters, and connected together to form a graph. The graph is used to build camera paths that allow viewers to "tour&#x201D; areas of interest (AOI) within their data. It is also visualized to provide wayfinding cues. Our preference model uses Bayesian classification to tag elements in a data set as interesting or not interesting to the viewer. The model responds in real time, updating the elements of interest based on a viewer's actions. This allows us to track a viewer's interests as they change during exploration and analysis. Viewers can also interact directly with interest rules the preference model defines. We demonstrate our theoretical results by visualizing historical climatology data collected at locations throughout the world.