Many different approaches for estimating the Interaction Quality (IQ) of Spoken Dialogue Systems have been investigated. While dialogues clearly have a sequential nature, statistical classification approaches designed for sequential problems do not seem to work better on automatic IQ estimation than static approaches, i.e., regarding each turn as being independent of the corresponding dialogue. Hence, we analyse this effect by investigating the subset of temporal features used as input for statistical classification of IQ. We extend the set of temporal features to contain the system and the user view. We determine the contribution of each feature sub-group showing that temporal features contribute most to the classification performance. Furthermore, for the feature sub-group modeling the temporal effects with a window, we modify the window size increasing the overall performance significantly by +15.69 .
This paper addresses a new approach for statistical modeling of user satisfaction in Spoken Dialogue Systems (SDS) and thereby allows an online monitoring of spoken human-machine interaction. The presented technique relies on a large set of input variables originating from system log files that quantify the ongoing spoken human-machine interaction. The target variable, user satisfaction (US), is captured in a lab study on a 5 point scale with 46 users interacting with an SDS. The model, which is based on Support Vector Machines (SVM) yields a performance of 49.2 unweighted average recall (Cohen's κ = .442, Spearman's ρ = .668) and significantly outperforms related work in that field.