The detection of anomalous executions is valuable for reducing potential hazards in assistive manipulation. Multimodal sensory signals can be helpful for detecting a wide range of anomalies. However, the fusion of high-dimensional and heterogeneous modalities is a challenging problem. We introduce a long short-term memory based variational autoencoder (LSTM-VAE) that fuses signals and reconstructs their expected distribution. We also introduce an LSTM-VAE-based detector using a reconstruction-based anomaly score and a state-based threshold. For evaluations with 1,555 robot-assisted feeding executions including 12 representative types of anomalies, our detector had a higher area under the receiver operating characteristic curve (AUC) of 0.8710 than 5 other baseline detectors from the literature. We also show the multimodal fusion through the LSTM-VAE is effective by comparing our detector with 17 raw sensory signals versus 4 hand-engineered features.
In this paper, we propose an estimation method of human joint movements from measured EMG signals for assistive robot control. We focus on how to estimate joint movements using multiple EMG electrodes even under sensor failure situations. In real world applications, EMG sensor electrodes might become disconnected or detached from skin surfaces. If we consider EMG-based robot control for assistive robots, such sensor failures lead to significant errors in the estimation of user joint movements. To cope with these sensor failures, we propose a state estimation model that takes uncertain observations into account. Sensor channel anomalies are found by checking the covariance of the EMG signals measured by multiple EMG electrodes. To validate the proposed control framework, we artificially disconnect an EMG electrode or detach one side of an EMG probe from the skin surface during elbow joint movement estimation. We show proper control of a one-DOF exoskeleton robot based on the estimated joint torque using our proposed method even when one EMG electrode has a sensor problem; a standard method with no tolerability against uncertain observations was unable to deal with these fault situations. Furthermore, the errors of the estimated joint torque with our proposed method were smaller than the standard method or a method with a conventional sensor fault detection algorithm. Autonomous mobile robots are designed to behave appropriately in changing real-world environments without human intervention. In order to satisfy the requirements of autonomy, robots have to cope with unknown settings and several issues of uncertainties in dynamic, unstructured and complex environments. A first step is to provide a robot with cognitive capabilities and the ability of self-examination to detect behavioral abnormalities. Unfortunately, most existing anomaly detection systems are neither suitable for the domain of robotic behavior nor flexible enough or even well generalizable. In the following article, we introduce a novel anomaly detection framework based on spatial-temporal models for robotic behaviors which is generally applicable for e.g., plan execution monitoring. The introduced framework combines the methodology of Kohonen’s Self-organizing Maps (SOMs) and Probabilistic Graphical Models (PGM) exploiting all advantages of both concepts. The underlying methods of the framework are discussed briefly, whereas the data-driven training of the spatial-temporal model and the reasoning process are described in detail. Finally, the framework is evaluated with different scenarios to emphasize its potential and its high level of generalization and flexibility in robotic application. Iteration is often sufficient for a simple hand to accomplish complex tasks, at the cost of an increase in the expected time to completion. In this paper, we minimize that overhead time by allowing a simple hand to abort early and retry as soon as it realizes that the task is likely to fail. We present two key contributions. First, we learn a probabilistic model of the relationship between the likelihood of success of a grasp and its grasp signature—the trace of the state of the hand along the entire grasp motion. Second, we model the iterative process of early abort and retry as a Markov chain and optimize the expected time to completion of the grasping task by effectively thresholding the likelihood of success. Experiments with our simple hand prototype tasked with grasping and singulating parts from a bin show that early abort and retry significantly increases efficiency. This paper addresses failure detection in automated parts assembly, using the force signature captured during the contact phase of the assembly process. We use a supervised learning approach, specifically a Support Vector Machine (SVM), to distinguish between successful and failed assemblies. This paper describes our implementation and experimental results obtained with an electronic assembly application. We also analyze the tradeoff between system accuracy and number of training examples. We show that a less expensive sensor (a single-axis load cell instead of a six-axis force torque sensor) provides enough information to detect failure. Finally, we use Principal Component Analysis (PCA) to compress the force signature and as a result reduce the number of examples required to train the system. This paper addresses an application of anomaly detection from subsequences of time series (STS) to autonomous robots’ behaviors. An important aspect of mining sequential data is selecting the temporal parameters, such as the subsequence length and the degree of smoothing. For example in the task at hand, the patterns of the robot’s velocity, which is one of its fundamental features, vary significantly subject to the interval for measuring the displacement. Selecting the time scale and resolution is difficult in unsupervised settings, and is often more critical than the choice of the method. In this paper, we propose an ensemble framework for aggregating anomaly detection from different perspectives, i.e., settings of user-defined, temporal parameters. In the proposed framework, each behavior is labeled whether it is an anomaly in multiple settings. The set of labels are used as meta-features of the respective behaviors. Cluster analysis in a meta-feature space partitions anomalous behaviors pertained to a specific range of parameters. The framework also includes a scalable implementation of the instance-based anomaly detection. We evaluate the proposed framework by ROC analysis, in comparison to conventional ensemble methods for anomaly detection. Safety is one of the key issues in the use of robots, especially when human–robot interaction is targeted. Although unforeseen environment situations, such as collisions or unexpected user interaction, can be handled with specially tailored control algorithms, hard- or software failures typically lead to situations where too large torques are controlled, which cause an emergency state: hitting an end stop, exceeding a torque, and so on—which often halts the robot when it is too late. No sufficiently fast and reliable methods exist which can early detect faults in the abundance of sensor and controller data. This is especially difficult since, in most cases, no anomaly data are available. In this paper we introduce a new robot anomaly detection system (RADS) which can cope with abundant data in which no or very little anomaly information is present. One of the main challenges in autonomous manipulation is to generate appropriate multi-modal reference trajectories that enable feedback controllers to compute control commands that compensate for unmodeled perturbations and therefore to achieve the task at hand. We propose a data-driven approach to incrementally acquire reference signals from experience and decide online when and to which successive behavior to switch, ensuring successful task execution. We reformulate this online decision making problem as a pair of related classification problems. Both process the current sensor readings, composed from multiple sensor modalities, in real-time (at 30 Hz). Our approach exploits that movement generation can dictate sensor feedback. Thus, enforcing stereotypical behavior will yield stereotypical sensory events which can be accumulated and stored along with the movement plan. Such movement primitives, augmented with sensor experience, are called Associative Skill Memories (ASMs). Sensor experience consists of (real) sensors, including haptic, auditory information and visual information, as well as additional (virtual) features. We show that our approach can be used to teach dexterous tasks, e.g. a bimanual manipulation task on a real platform that requires precise manipulation of relatively small objects. Task execution is robust against perturbation and sensor noise, because our method decides online whether or not to switch to alternative ASMs due to unexpected sensory signals.  Online detection of anomalous execution can be valuable for robot manipulation, enabling robots to operate more safely, determine when a behavior is inappropriate, and otherwise exhibit more common sense. By using multiple complementary sensory modalities, robots could potentially detect a wider variety of anomalies, such as anomalous contact or a loud utterance by a human. However, task variability and the potential for false positives make online anomaly detection challenging, especially for long-duration manipulation behaviors. In this paper, we provide evidence for the value of multimodal execution monitoring and the use of a detection threshold that varies based on the progress of execution. Using a data-driven approach, we train an execution monitor that runs in parallel to a manipulation behavior. Like previous methods for anomaly detection, our method trains a hidden Markov model (HMM) using multimodal observations from non-anomalous executions. In contrast to prior work, our system also uses a detection threshold that changes based on the execution progress. We evaluated our approach with haptic, visual, auditory, and kinematic sensing during a variety of manipulation tasks performed by a PR2 robot. The tasks included pushing doors closed, operating switches, and assisting able-bodied participants with eating yogurt. In our evaluations, our anomaly detection method performed substantially better with multimodal monitoring than single modality monitoring. It also resulted in more desirable ROC curves when compared with other detection threshold methods from the literature, obtaining higher true positive rates for comparable false positive rates.