Turn-taking is essential to the structure of human teamwork. Humans are typically aware of team members' intention to keep or relinquish their turn before a turn switch, where the responsibility of working on a shared task is shifted. Future co-robots are also expected to provide such competence. To that end, this paper proposes the Cognitive Turn-taking Model (CTTM), which leverages cognitive models (i.e., Spiking Neural Network) to achieve early turn-taking prediction. The CTTM framework can process multimodal human communication cues (both implicit and explicit) and predict human turn-taking intentions in an early stage. The proposed framework is tested on a simulated surgical procedure, where a robotic scrub nurse predicts the surgeon's turn-taking intention. It was found that the proposed CTTM framework outperforms the state-of-the-art turn-taking prediction algorithms by a large margin. It also outperforms humans when presented with partial observations of communication cues (i.e., less than 40 of full actions). This early prediction capability enables robots to initiate turn-taking actions at an early stage, which facilitates collaboration and increases overall efficiency.
Physical turn-taking refers to the process where a hybrid human-robot team take turns on a physical task. Turn-taking has been studied in robotics and HRI to regulate shared resources among team members. Those resources include time (i.e., only one person can work on the shared task at a time) and space (i.e., only one person can access the working space at a time). In the context of human robot interaction, the types and usage frequencies of various implicit communication cues have been studied in a robot-assisted assembly task @cite_13 . The timing in multimodal turn-taking (i.e., speech, gaze, and gesture) was investigated through a collaborative Towers of Hanoi challenge accomplished by a hybrid human robot team @cite_1 . Nonverbal cues for timing coordination in physical turn-taking were studied in manufacturing @cite_9 . Such research focuses on turn-taking process modelling and the robot control, while neglecting predictions of the turn-taking timing and intention.
@cite_1 @cite_9 @cite_13
2553961644 2011685492
Nonverbal communication cues play an important role in human-human interaction and are expected to take a similar role in human-robot collaboration. In current industrial practice, human-robot turn-taking is explicitly human controlled, via a command channel such as switch or button. However, such a master-slave approach does not permit collaborative interaction, and requires the human to focus on both controlling the robot's behavior and on the task, thereby affecting overall performance. In this paper, implicit, nonverbal communication cues are examined as a non-explicit communication channel during a turn-taking task context. The aim of this study is to characterize the types and frequencies of nonverbal cues important to regulating turn taking during an assembly-task-type collaboration. This analysis will guide the selection of cues that can be expressed by the robot as implicit user inputs while human and robot complete a shared task. The goal of this work is to develop computational models of social intelligence that enable robots to work side by side with humans, solving problems and achieving task goals through dialogue and collaborative manipulation. A defining problem of collaborative behavior in an embodied setting is the manner in which multiple agents make use of shared resources. In a situated dialogue, these resources include physical bottlenecks such as objects or spatial regions, and cognitive bottlenecks such as the speaking floor. For a robot to function as an effective collaborative partner with a human, it must be able to seize and yield such resources appropriately according to social expectations. We describe a general framework that uses timed Petri nets for the modeling and execution of robot speech, gaze, gesture, and manipulation for collaboration. The system dynamically monitors resource requirements and availability to control real-time turn-taking decisions over resources that are shared with humans, reasoning about different resource types independently. We evaluate our approach with an experiment in which our robot Simon performs a collaborative assembly task with 26 different human partners, showing that the multimodal reciprocal approach results in superior task performance, fluency, and balance of control. Humans naturally use the multiple modalities of speech, gesture, and gaze when they communicate. They also engage in turn-taking to manage the seizing and yielding of the speaking oor, a process that controls the execution of turns comprising discrete speech, gesture, and gaze events. We are interested in modeling such interaction processes for a social robot to use that can be transferred between dierent domains. Timed Petri nets (TPNs) are currently uncommon in human-robot interaction (HRI) but oer an attractive representation for modeling concurrency and synchronization when controlling behavior for multiple modalities. Their representation also permits the intuitive and modular combination of rule-based behavior expression with statistical timing models. We describe their utility in application to multimodal interaction and compare them to nite state In the real world, noisy sensors and limited communication make it difficult for robot teams to coordinate in tightly coupled tasks. Team members cannot simply apply single-robot solution techniques for partially observable problems in parallel because they do not take into account the recursive effect that reasoning about the beliefs of others has on policy generation. Instead, we must turn to a game theoretic approach to model the problem correctly. Partially observable stochastic games (POSGs) provide a solution model for decentralized robot teams, however, this model quickly becomes intractable. In previous work we presented an algorithm for lookahead search in POSGs. Here we present an extension which reduces computation during lookahead by clustering similar observation histories together. We show that by clustering histories which have similar profiles of predicted reward, we can greatly reduce the computation time required to solve a POSG while maintaining a good approximation to the optimal policy. We demonstrate the power of the clustering algorithm in a real-time robot controller as well as for a simple benchmark problem. Rescue robotics has been suggested by a recent DARPA NSF study as an application domain for the research in human-robot integration (HRI). This paper provides a short tutorial on how robots are currently used in urban search and rescue (USAR) and discusses the HRI issues encountered over the past eight years. A domain theory of the search activity is formulated. The domain theory consists of two parts: 1) a workflow model identifying the major tasks, actions, and roles in robot-assisted search (e.g., a workflow model) and 2) a general information flow model of how data from the robot is fused by various team members into information and knowledge. The information flow model also captures the types of situation awareness needed by each agent in the rescue robot system. The article presents a synopsis of the major HRI issues in reducing the number of humans it takes to control a robot, maintaining performance with geographically distributed teams with intermittent communications, and encouraging acceptance within the existing social structure. We describe the design and evaluation of Chaski, a robot plan execution system that uses insights from human-human teaming to make human-robot teaming more natural and fluid. Chaski is a task-level executive that enables a robot to collaboratively execute a shared plan with a person. The system chooses and schedules the robot's actions, adapts to the human partner, and acts to minimize the human's idle time. We evaluate Chaski in human subject experiments in which a person works with a mobile and dexterous robot to collaboratively assemble structures using building blocks. We measure team performance outcomes for robots controlled by Chaski compared to robots that are verbally commanded, step-by-step by the human teammate. We show that Chaski reduces the human's idle time by 85 , a statistically significant difference. This result supports the hypothesis that human-robot team performance is improved when a robot emulates the effective coordination behaviors observed in human teams. Objective: We conducted an empirical analysis of human teamwork to investigate the ways teammates incorporate coordination behaviors, including verbal and nonverbal cues, into their action planning. Background: In space, military, aviation, and medical industries, teams of people effectively coordinate to perform complex tasks under stress induced by uncertainty, ambiguity, and time pressure. As robots increasingly are introduced into these domains, we seek to understand effective human-team coordination to inform natural and effective human-robot coordination. Method: We conducted teamwork experiments in which teams of two people performed a complex task, involving ordering, timing, and resource constraints. Half the teams performed under time pressure, and half performed without time pressure. We cataloged the coordination behaviors used by each team and analyzed the speed of response and specificity of each coordination behavior. Results: Analysis shows that teammates respond to explicit cues, includin... When humans and mobile robots share the same space, one of their challenges is to navigate around each other and manage their mutual navigational intents. While humans have developed excellent skills in inferring their counterpart's intentions via a number of implicit and non-verbal cues, making navigation also in crowds an ease, this kind of effective and efficient communication often falls short in human-robot encounters. In this paper, two alternative approaches to convey navigational intent of a mobile robot to humans in a shared environment are proposed and analysed. The first is utilising anthropomorphic features of the mobile robot to realise an implicit joint attention using gaze to represent the direction of navigational intent. In the second approach, a more technical design adopting the semantics of car's turn indicators, has been implemented. The paper compares both approaches with each other and against a control behaviour without any communication of intent. Both approaches show statistically significant differences in comparison to the control behaviour. However, the second approach using indicators has shown as being more effective in conveying the intent and also has a higher positive impact on the comfort of the humans encountering the robot. Grounded language learning bridges words like 'red' and 'square' with robot perception. The vast majority of existing work in this space limits robot perception to vision. In this paper, we build perceptual models that use haptic, auditory, and proprioceptive data acquired through robot exploratory behaviors to go beyond vision. Our system learns to ground natural language words describing objects using supervision from an interactive humanrobot "I Spy" game. In this game, the human and robot take turns describing one object among several, then trying to guess which object the other has described. All supervision labels were gathered from human participants physically present to play this game with a robot. We demonstrate that our multi-modal system for grounding natural language outperforms a traditional, vision-only grounding framework by comparing the two on the "I Spy" task. We also provide a qualitative analysis of the groundings learned in the game, visualizing what words are understood better with multimodal sensory information as well as identifying learned word meanings that correlate with physical object properties (e.g. 'small' negatively correlates with object weight). We report on an empirical study of a multiparty turn-taking model for physically situated spoken dialog systems. We present subjective and objective performance measures that show how the model, supported with a basic set of sensory competencies and turn-taking policies, can enable interactions with multiple participants in a collaborative task setting. The analysis brings to the fore several phenomena and frames challenges for managing multiparty turn taking in physically situated interaction. In manufacturing, advanced robotic technology has opened up the possibility of integrating highly autonomous mobile robots into human teams. However, with this capability comes the issue of how to maximize both team efficiency and the desire of human team members to work with these robotic counterparts. To address this concern, we conducted a set of experiments studying the effects of shared decision-making authority in human---robot and human-only teams. We found that an autonomous robot can outperform a human worker in the execution of part or all of the process of task allocation ( @math p<0.001 for both), and that people preferred to cede their control authority to the robot @math (p<0.001). We also established that people value human teammates more than robotic teammates; however, providing robots authority over team coordination more strongly improved the perceived value of these agents than giving similar authority to another human teammate @math (p<0.001). In post hoc analysis, we found that people were more likely to assign a disproportionate amount of the work to themselves when working with a robot @math (p<0.01) rather than human teammates only. Based upon our findings, we provide design guidance for roboticists and industry practitioners to design robotic assistants for better integration into the human workplace. This paper presents an overview of our work towards building humanoid robots that can work alongside people as cooperative teammates. We present our theoretical framework based on a novel combination of joint intention theory and collaborative discourse theory, and demonstrate how it can be applied to allow a human to work cooperatively with a humanoid robot on a joint task using speech, gesture, and expressive cues. Such issues must be addressed to enable many new and exciting applications for humanoid robots that require them to assist ordinary people in daily activities or to work as capable members of human-robot teams. ABSTRACT: With the advent of collaborative robots, there is a great potential to improve work performance by human-robot collaboration in engineering tasks. Construction is no exception. Many construction tasks are based on the movement of objects (e.g., material), which are viable candidates for human-robot collaboration. However, due to the physically imposing nature of robot operations and the unstructured environments typical in construction, it is crucial to provide a safe and reliable environment for human workers when performing collaborative work with robots. In this paper, we use Immersive Virtual Environments (IVEs) to evaluate a human response to robots (e.g. perceived safety, trust, and team identification) while performing collaborative construction tasks with robots. By adopting IVEs, various types of robots, interactions, and tasks can be easily tested and evaluated to determine the best HRC practice, without the need to build and evaluate a physical prototype. Several experimental scenarios simulating collaborative masonry tasks were implemented using the Unity3D Game Engine and an Oculus Rift 3D Head-Mounted Display (HMD). The results demonstrate that it is important to take into account work environment of human-robot collaboration in order to understand how humans perceive robots when working with them. In this letter, we propose a framework for task allocation in human-robot collaborative assembly planning. Our framework distinguishes between two main layers of abstraction and allocation. In the higher layer, we use an abstract world model, incorporating a multiagent human-robot team approach in order to describe the collaborative assembly planning problem. From this, nominal co-ordinated skill sequences for every agent are produced. In order to be able to treat humans and robots as agents of the same form, we move relevant differences peculiarities into distinct cost functions. The layer beneath handles the concrete skill execution. On atomic level, skills are composed of complex hierarchical and concurrent hybrid state machines, which in turn co-ordinate the real-time behavior of the robot. Their careful design allows to cope with unpredictable events also on decisional level without having to explicitly plan for them, instead one may rely also on manually designed skills. Such events are likely to happen in dynamic and potentially partially known environments, which is especially true in case of human presence. In this work, we design and evaluate a computational learning model that enables a human-robot team to co-develop joint strategies for performing novel tasks that require coordination. The joint strategies are learned through "perturbation training," a human team-training strategy that requires team members to practice variations of a given task to help their team generalize to new variants of that task. We formally define the problem of human-robot perturbation training and develop and evaluate the first end-to-end framework for such training, which incorporates a multi-agent transfer learning algorithm, human-robot co-learning framework and communication protocol. Our transfer learning algorithm, Adaptive Perturbation Training (AdaPT), is a hybrid of transfer and reinforcement learning techniques that learns quickly and robustly for new task variants. We empirically validate the benefits of AdaPT through comparison to other hybrid reinforcement and transfer learning techniques aimed at transferring knowledge from multiple source tasks to a single target task. @PARASPLIT We also demonstrate that AdaPT's rapid learning supports live interaction between a person and a robot, during which the human-robot team trains to achieve a high level of performance for new task variants. We augment AdaPT with a co-learning framework and a computational bi-directional communication protocol so that the robot can co-train with a person during live interaction. Results from large-scale human subject experiments (n=48) indicate that AdaPT enables an agent to learn in a manner compatible with a human's own learning process, and that a robot undergoing perturbation training with a human results in a high level of team performance. Finally, we demonstrate that human-robot training using AdaPT in a simulation environment produces effective performance for a team incorporating an embodied robot partner. The upcoming generation of humanoid robots will have to be equipped with state-of-the-art technical features along with high industrial quality, but they should also offer the prospect of effective physical human interaction. In this paper we introduce a new humanoid robot capable of interacting with a human environment and targeting industrial applications. Limitations are outlined and used together with the feedback from the DARPA Robotics Challenge, and other teams leading the field in creating new humanoid robots. The resulting robot is able to handle weights of 6 kg with an out-stretched arm, and has powerful motors to carry out fast movements. Its kinematics have been specially designed for screwing and drilling motions. In order to make interaction with human operators possible, this robot is equipped with torque sensors to measure joint effort and high resolution encoders to measure both motor and joint positions. The humanoid robotics field has reached a stage where robustness and repeatability is the next watershed. We believe that this robot has the potential to become a powerful tool for the research community to successfully navigate this turning point, as the humanoid robot HRP-2 was in its own time. In shared control teleoperation, the robot assists the user in accomplishing the desired task, making teleoperation easier and more seamless. Rather than simply executing the user's input, which is hindered by the inadequacies of the interface, the robot attempts to predict the user's intent, and assists in accomplishing it. In this work, we are interested in the scientific underpinnings of assistance: we propose an intuitive formalism that captures assistance as policy blending, illustrate how some of the existing techniques for shared control instantiate it, and provide a principled analysis of its main components: prediction of user intent and its arbitration with the user input. We define the prediction problem, with foundations in inverse reinforcement learning, discuss simplifying assumptions that make it tractable, and test these on data from users teleoperating a robotic manipulator. We define the arbitration problem from a control-theoretic perspective, and turn our attention to what users consider good arbitration. We conduct a user study that analyzes the effect of different factors on the performance of assistance, indicating that arbitration should be contextual: it should depend on the robot's confidence in itself and in the user, and even the particulars of the user. Based on the study, we discuss challenges and opportunities that a robot sharing the control with the user might face: adaptation to the context and the user, legibility of behavior, and the closed loop between prediction and user behavior. Human communication involves a number of nonverbal cues that are seemingly unintentional, unconscious, and automatic-both in their production and perception-and convey rich information on the emotional state and intentions of an individual. One family of such cues is called "nonverbal leakage." In this paper, we explore whether people can read nonverbal leakage cues-particularly gaze cues-in humanlike robots and make inferences on robots' intentions, and whether the physical design of the robot affects these inferences. We designed a gaze cue for Geminoid-a highly humanlike android-and Robovie-a robot with stylized, abstract humanlike features-that allowed the robots to "leak" information on what they might have in mind. In a controlled laboratory experiment, we asked participants to play a game of guessing with either of the robots and evaluated how the gaze cue affected participants' task performance. We found that the gaze cue did, in fact, lead to better performance, from which we infer that the cue led to attributions of mental states and intentionality. Our results have implications for robot design, particularly for designing expression of intentionality, and for our understanding of how people respond to human social cues when they are enacted by robots. Abstract The interaction between humans and robot teams is highly relevant in many application domains, for example in collaborative manufacturing, search and rescue, and logistics. It is well-known that humans and robots have complementary capabilities: Humans are excellent in reasoning and planning in unstructured environments, while robots are very good in performing tasks repetitively and precisely. In consequence, one of the key research questions is how to combine human and robot team decision making and task execution capabilities in order to exploit their complementary skills. From a controls perspective this question boils down to how control should be shared among them. This article surveys advances in human-robot team interaction with special attention devoted to control sharing methodologies. Additionally, aspects affecting the control sharing design, such as human behavior modeling, level of autonomy and human-machine interfaces are identified. Open problems and future research directions towards joint decision making and task execution in human-robot teams are discussed. Trained human co-workers can often easily predict each other's intentions based on prior experience. When collaborating with a robot coworker, however, intentions are hard or impossible to infer. This difficulty of mental introspection makes human-robot collaboration challenging and can lead to dangerous misunderstandings. In this paper, we present a novel, object-aware projection technique that allows robots to visualize task information and intentions on physical objects in the environment. The approach uses modern object tracking methods in order to display information at specific spatial locations taking into account the pose and shape of surrounding objects. As a result, a human co-worker can be informed in a timely manner about the safety of the workspace, the site of next robot manipulation tasks, and next subtasks to perform. A preliminary usability study compares the approach to collaboration approaches based on monitors and printed text. The study indicates that, on average, the user effectiveness and satisfaction is higher with the projection based approach.