Humans describe images in terms of nouns and adjectives while algorithms operate on images represented as sets of pixels. Bridging this gap between how humans would like to access images versus their typical representation is the goal of image parsing, which involves assigning object and attribute labels to pixels. In this article we propose treating nouns as object labels and adjectives as visual attribute labels. This allows us to formulate the image parsing problem as one of jointly estimating per-pixel object and attribute labels from a set of training images. We propose an efficient (interactive time) solution. Using the extracted labels as handles, our system empowers a user to verbally refine the results. This enables hands-free parsing of an image into pixel-wise object attribute labels that correspond to human semantics. Verbally selecting objects of interest enables a novel and natural interaction modality that can possibly be used to interact with new generation devices (e.g., smartphones, Google Glass, livingroom devices). We demonstrate our system on a large number of real-world images with varying complexity. To help understand the trade-offs compared to traditional mouse-based interactions, results are reported for both a large-scale quantitative evaluation and a user study.
Speech interface Speech interfaces are deployed when mouse based interactions are infeasible or cumbersome. Although research on integrating speech interfaces into software started in the 1980s @cite_50 , it is only recently that such interfaces have been widely deployed, (e.g. Apple's Siri, PixelTone ). However, most speech interface research is focused on natural language processing and to our knowledge there has been no prior work addressing image region selection through speech. The speech interface that most resembles our work is PixelTone , which allows users to attach object labels to scribble based segments. These labels allow subsequent voice reference. Independently, we have developed a hands-free parsing of an image into pixel-wise object attribute labels that correspond to human semantics. This provides a verbal option for selecting objects of interest and is potentially, a powerful additional tool for speech interfaces.
@cite_50
2731466267 2614846107 2042159198 2189217339
Abstract-People communicate through words and gestures,but current voice-based computer interfaces such as Siri exploitonly words. This is a shame: human-computer interfaces wouldbe natural if they incorporated gestures as well as words. Tosupport this goal, we present a new dataset of naturally occurringgestures made by people working collaboratively on blocks worldtasks. The dataset, called EGGNOG, contains over 8 hours ofRGB video, depth video, and Kinect v2 body position data of 40subjects. The data has been semi-automatically segmented into24,503 movements, each of which has been labeled accordingto (1) its physical motion and (2) the intent of the participant.We believe this dataset will stimulate research into natural andgestural human-computer interfaces Abstract Speech interfaces to conversational systems have been a focus in academia and industry for over a decade due to its applicability as a natural interface. Speech recognition and speech synthesis constitute the important input and output modules respectively for such spoken interface systems. In this paper, the speech recognition interface for question answering applications is reviewed, and existing limitations are discussed. The existing spoken question answering (QA) systems use an automatic speech recogniser by adapting acoustic and language models for the speech interface and off-the-shelf language processing systems for question interpretation. In the process, the impact of recognition errors and language processing inaccuracies is neglected. It is illustrated in the paper how a semantically rich knowledge graph can be used to solve automatic speech recognition and language processing specific problems. A simple concatenation of a speech recogniser and a natural language processing system is a shallow method for a speech interface. An effort beyond merely concatenating these two units is required to develop a successful spoken question answering system. It is illustrated in this paper how a knowledge graph based structured data can be used to build a unified system combining speech recognition and language understanding. This facilitates the use of a semantically rich data model for speech interface. The design of robust interfaces that process conversational speech is a challenging research direction largely because users' spoken language is so variable. This research explored a new dimension of speaker stylistic variation by examining whether users' speech converges systematically with the text-to-speech (TTS) heard from a software partner. To pursue this question, a study was conducted in which twenty-four 7 to 10-year-old children conversed with animated partners that embodied different TTS voices. An analysis of children's amplitude, durational features, and dialogue response latencies confirmed that they spontaneously adapt several basic acoustic-prosodic features of their speech 10--50p, with the largest adaptations involving utterance pause structure and amplitude. Children's speech adaptations were relatively rapid, bidirectional, and dynamically readaptable when introduced to new partners, and generalized across different types of users and TTS voices. Adaptations also occurred consistently, with 70--95p of children converging with their partner's TTS, although individual differences in magnitude of adaptation were evident. In the design of future conversational systems, users' spontaneous convergence could be exploited to guide their speech within system processing bounds, thereby enhancing robustness. Adaptive system processing could yield further significant performance gains. The long-term goal of this research is the development of predictive models of human-computer communication to guide the design of new conversational interfaces. Voice-activated intelligent assistants, such as Siri, Google Now, and Cortana, are prevalent on mobile devices. However, it is challenging to evaluate them due to the varied and evolving number of tasks supported, e.g., voice command, web search, and chat. Since each task may have its own procedure and a unique form of correct answers, it is expensive to evaluate each task individually. This paper is the first attempt to solve this challenge. We develop consistent and automatic approaches that can evaluate different tasks in voice-activated intelligent assistants. We use implicit feedback from users to predict whether users are satisfied with the intelligent assistant as well as its components, i.e., speech recognition and intent classification. Using this approach, we can potentially evaluate and compare different tasks within and across intelligent assistants ac-cording to the predicted user satisfaction rates. Our approach is characterized by an automatic scheme of categorizing user-system interaction into task-independent dialog actions, e.g., the user is commanding, selecting, or confirming an action. We use the action sequence in a session to predict user satisfaction and the quality of speech recognition and intent classification. We also incorporate other features to further improve our approach, including features derived from previous work on web search satisfaction prediction, and those utilizing acoustic characteristics of voice requests. We evaluate our approach using data collected from a user study. Results show our approach can accurately identify satisfactory and unsatisfactory sessions.