Despite significant improvements in automatic speech recognition and spoken language understanding - human interaction with Virtual Personal Assistants (VPAs) through speech remains irregular and sporadic. According to recent studies, currently the usage of VPAs is constrained to basic tasks such as checking facts, playing music, and obtaining weather updates.In this paper, we present results of a survey (N = 118) that analyses usage of VPAs by frequent and infrequent users. We investigate how usage experience, performance expectations, and privacy concerns differ between these two groups. The results indicate that, compared with infrequent users, frequent users of VPAs are more satisfied with their assistants, more eager to use them in a variety of settings, yet equally concerned about their privacy.
: @cite_1 found that users tend to be more satisfied with using VPAs for simple tasks (such as device control) rather than more complex, multi-turn tasks (e.g. making travel arrangements etc.) where preserving the context is crucial. Despite being comprehensive, the study was limited to one type of virtual assistant (Cortana). Our survey adopts a broader perspective by probing users’ satisfaction with a variety of VPAs. @cite_0 focused on the experiences of infrequent VPA users and their reasons for not using VPAs on a regular basis. The feedback obtained from the focus groups indicates that privacy concerns over data usage, and lack of trust in the assistant's capability to perform the task are some of the main reasons for people not to use the technology on a regular basis. Although 's study was limited only to infrequent users of Siri, it can be argued that data permanency, ownership of data and limited human-like interaction abilities are the factors that are relevant across different devices. Thus, it would be worth extending the scope of investigation to other assistants. Our study applies a broad approach towards evaluation of VPAs by considering the feedback of different user groups (defined in Section ).
@cite_0 @cite_1
2331728731 2051626970 2189217339 2751379218
Voice-controlled intelligent personal assistants, such as Cortana, Google Now, Siri and Alexa, are increasingly becoming a part of users' daily lives, especially on mobile devices. They introduce a significant change in information access, not only by introducing voice control and touch gestures but also by enabling dialogues where the context is preserved. This raises the need for evaluation of their effectiveness in assisting users with their tasks. However, in order to understand which type of user interactions reflect different degrees of user satisfaction we need explicit judgements. In this paper, we describe a user study that was designed to measure user satisfaction over a range of typical scenarios of use: controlling a device, web search, and structured search dialogue. Using this data, we study how user satisfaction varied with different usage scenarios and what signals can be used for modeling satisfaction in the different scenarios. We find that the notion of satisfaction varies across different scenarios, and show that, in some scenarios (e.g. making a phone call), task completion is very important while for others (e.g. planning a night out), the amount of effort spent is key. We also study how the nature and complexity of the task at hand affects user satisfaction, and find that preserving the conversation context is essential and that overall task-level satisfaction cannot be reduced to query-level satisfaction alone. Finally, we shed light on the relative effectiveness and usefulness of voice-controlled intelligent agents, explaining their increasing popularity and uptake relative to the traditional query-response interaction. A review of popular technology adoption models identified several factors that are likely to influence Voice Activated Personal Assistant (VAPA) use in public spaces. To inform design decisions of how to make the private use of the VAPA in public spaces more acceptable from the users’ point of view, an online survey was conducted to investigate the likelihood of usage of the smartphone VAPA such as Apple’s Siri (compared to the usage of smartphone keyboard) as a function of location (private vs. public) and type of information (private vs. nonprivate). Responses from participants showed that users were more cautious of transmitting private than nonprivate information. This effect of type of information was amplified in the social context of public locations and when using conspicuous methods of information input such as the VAPA. Participants also preferred using the VAPA in private locations and showed no preference of location for keyboard entries. Correlations between likelihood of usage of VAPA and th... Voice-activated intelligent assistants, such as Siri, Google Now, and Cortana, are prevalent on mobile devices. However, it is challenging to evaluate them due to the varied and evolving number of tasks supported, e.g., voice command, web search, and chat. Since each task may have its own procedure and a unique form of correct answers, it is expensive to evaluate each task individually. This paper is the first attempt to solve this challenge. We develop consistent and automatic approaches that can evaluate different tasks in voice-activated intelligent assistants. We use implicit feedback from users to predict whether users are satisfied with the intelligent assistant as well as its components, i.e., speech recognition and intent classification. Using this approach, we can potentially evaluate and compare different tasks within and across intelligent assistants ac-cording to the predicted user satisfaction rates. Our approach is characterized by an automatic scheme of categorizing user-system interaction into task-independent dialog actions, e.g., the user is commanding, selecting, or confirming an action. We use the action sequence in a session to predict user satisfaction and the quality of speech recognition and intent classification. We also incorporate other features to further improve our approach, including features derived from previous work on web search satisfaction prediction, and those utilizing acoustic characteristics of voice requests. We evaluate our approach using data collected from a user study. Results show our approach can accurately identify satisfactory and unsatisfactory sessions. Intelligent Personal Assistants (IPAs) are widely available on devices such as smartphones. However, most people do not use them regularly. Previous research has studied the experiences of frequent IPA users. Using qualitative methods we explore the experience of infrequent users: people who have tried IPAs, but choose not to use them regularly. Unsurprisingly infrequent users share some of the experiences of frequent users, e.g. frustration at limitations on fully hands-free interaction. Significant points of contrast and previously unidentified concerns also emerge. Cultural norms and social embarrassment take on added significance for infrequent users. Humanness of IPAs sparked comparisons with human assistants, juxtaposing their limitations. Most importantly, significant concerns emerged around privacy, monetization, data permanency and transparency. Drawing on these findings we discuss key challenges, including: designing for interruptability; reconsideration of the human metaphor; issues of trust and data ownership. Addressing these challenges may lead to more widespread IPA use.