Intelligent conversational assistants, such as Apple's Siri, Microsoft's Cortana, and Amazon's Echo, have quickly become a part of our digital life. However, these assistants have major limitations, which prevents users from conversing with them as they would with human dialog partners. This limits our ability to observe how users really want to interact with the underlying system. To address this problem, we developed a crowd-powered conversational assistant, Chorus, and deployed it to see how users and workers would interact together when mediated by the system. Chorus sophisticatedly converses with end users over time by recruiting workers on demand, which in turn decide what might be the best response for each user sentence. Up to the first month of our deployment, 59 users have held conversations with Chorus during 320 conversational sessions. In this paper, we present an account of Chorus' deployment, with a focus on four challenges: (i) identifying when conversations are over, (ii) malicious users and workers, (iii) on-demand recruiting, and (iv) settings in which consensus is not enough. Our observations could assist the deployment of crowd-powered conversation systems and crowd-powered systems in general.
View @cite_7 , which was built upon the ideas introduced in VizWiz, used a continuous interaction between multiple crowd workers and an end user based on video. View, which aggregates workers answers, has showed that multiple workers answer more quickly, accurately, and completely than individuals. Unfortunately, to date, View has not been deployed in the wild. This is in part because of the cost of scaling this type of continuous interaction, as well as ensuring on-going reliability with minimal ability to automatically monitor interactions. Be My Eyes Be My Eyes: http: www.bemyeyes.org is a deployed application with a similar goal: answer visual questions asked by blind users by streaming video. However, while they draw from a crowd of remote people to answer questions, the interaction is one-on-one, which assumes reliable helpers are available. Be My Eyes relies on volunteers rather than paid crowd workers. However, in more general settings, relying on volunteers is not practical.
@cite_7
2116680608 2058556535 2788643321 2012339105
Blind people face a range of accessibility challenges in their everyday lives, from reading the text on a package of food to traveling independently in a new place. Answering general questions about one's visual surroundings remains well beyond the capabilities of fully automated systems, but recent systems are showing the potential of engaging on-demand human workers (the crowd) to answer visual questions. The input to such systems has generally been a single image, which can limit the interaction with a worker to one question; or video streams where systems have paired the end user with a single worker, limiting the benefits of the crowd. In this paper, we introduce Chorus:View, a system that assists users over the course of longer interactions by engaging workers in a continuous conversation with the user about a video stream from the user's mobile device. We demonstrate the benefit of using multiple crowd workers instead of just one in terms of both latency and accuracy, then conduct a study with 10 blind users that shows Chorus:View answers common visual questions more quickly and accurately than existing approaches. We conclude with a discussion of users' feedback and potential future work on interactive crowd support of blind users. Visual information pervades our environment. Vision is used to decide everything from what we want to eat at a restaurant and which bus route to take to whether our clothes match and how long until the milk expires. Individually, the inability to interpret such visual information is a nuisance for blind people who often have effective, if inefficient, work-arounds to overcome them. Collectively, however, they can make blind people less independent. Specialized technology addresses some problems in this space, but automatic approaches cannot yet answer the vast majority of visual questions that blind people may have. VizWiz addresses this shortcoming by using the Internet connections and cameras on existing smartphones to connect blind people and their questions to remote paid workers' answers. VizWiz is designed to have low latency and low cost, making it both competitive with expensive automatic solutions and much more versatile. The study of algorithms to automatically answer visual questions currently is motivated by visual question answering (VQA) datasets constructed in artificial VQA settings. We propose VizWiz, the first goal-oriented VQA dataset arising from a natural VQA setting. VizWiz consists of over 31,000 visual questions originating from blind people who each took a picture using a mobile phone and recorded a spoken question about it, together with 10 crowdsourced answers per visual question. VizWiz differs from the many existing VQA datasets because (1) images are captured by blind photographers and so are often poor quality, (2) questions are spoken and so are more conversational, and (3) often visual questions cannot be answered. Evaluation of modern algorithms for answering visual questions and deciding if a visual question is answerable reveals that VizWiz is a challenging dataset. We introduce this dataset to encourage a larger community to develop more generalized algorithms that can assist blind people. Fashion is a language. How we dress signals to others who we are and how we want to be perceived. However, this language is primarily visual, making it inaccessible to people with vision impairments. Someone who is low-vision or completely blind cannot see what others are wearing or readily know what constitutes the norms and extremes of fashion, but most everyone they encounter can see (and judge) their fashion choices. We describe our findings of a diary study with people with vision impairments that revealed the many accessibility barriers fashion presents, and how an online survey revealed that clothing decisions are often made collaboratively, regardless of visual ability. Based on these findings, we identified a need for a collaborative and real-time environment for fashion advice. We have tested the feasibility of providing this advice through crowdsourcing using VizWiz, a mobile phone application where participants receive nearly real-time answers to visual questions. Our pilot study results show that this application has the potential to address a great need within the blind community, but remaining challenges include improving photo capture and assembling a set of crowd workers with the requisite expertise. More broadly our research highlights the feasibility of using crowdsourcing for subjective, opinion-based advice.