Understanding user instructions in natural language is an active research topic in AI and robotics. Typically, natural user instructions are high-level and can be reduced into low-level tasks expressed in common verbs (e.g., take', get', put'). For robots understanding such instructions, one of the key challenges is to process high-level user instructions and achieve the specified tasks with robots' primitive actions. To address this, we propose novel algorithms by utilizing semantic roles of common verbs defined in semantic dictionaries and integrating multiple open knowledge to generate task plans. Specifically, we present a new method for matching and recovering semantics of user instructions and a novel task planner that exploits functional knowledge of robot's action model. To verify and evaluate our approach, we implemented a prototype system using knowledge from several open resources. Experiments on our system confirmed the correctness and efficiency of our algorithms. Notably, our system has been deployed in the KeJia robot, which participated the annual RoboCup@Home competitions in the past three years and achieved encouragingly high scores in the benchmark tests.
The approaches that are most related to ours are the ones using OMICS for robots to complete household tasks. The first attempt to utilize OMICS to accomplish a household task is @cite_26 , which proposed a generative model based on the Markov chain techniques. Later on, @cite_17 @cite_9 @cite_27 presented a system called KNOWROB for processing knowledge in order to achieve more flexible and general behavior. Most recently, we proposed a formal description of knowledge gaps between user instructions and local knowledge in robotic system for instruction understanding @cite_25 @cite_24 @cite_16 @cite_10 . However, in these efforts using OMICS for robot task planning with user instructions, common verbs are normally not defined in the knowledge base, which limits their performance on utilizing existing open knowledge. Thus, our work is proposed to address the weakness of state-of-the-art methods.
@cite_26 @cite_9 @cite_24 @cite_27 @cite_16 @cite_10 @cite_25 @cite_17
2285097965 2114653216 2396580185 2017197542 2082543769 2186444419 2156415207
A system and a method are disclosed that provide plans for autonomous machines such as humanoid robots to perform indoor task. Human subjects contribute plans to a knowledge database. Information in the knowledge database is pre-processed to identify task steps and characterize them as action-object pairs, from which a plan database is created. A discriminative technique uses hierarchical agglomerative clustering to select an existing plan from the plan database. A generative technique formulates new plans from the plan database using first-order Markov chains, and may take into account information about the operational environment. Experimentation and evaluation by human subjects confirm the efficacy of both techniques. Unlike people, household robots cannot rely on commonsense knowledge when accomplishing everyday tasks. We believe that this is one of the reasons why they perform poorly in comparison to humans. By integrating extensive collections of commonsense knowledge into mobile robot's knowledge bases, the work proposed in this paper enables robots to flexibly infer control decisions under changing environmental conditions. We present a system that converts commonsense knowledge from the large Open Mind Indoor Common Sense database from natural language into a Description Logic representation that allows for automated reasoning and for relating it to other sources of knowledge. Users may ask a service robot to accomplish various tasks so that the designer of the robot cannot program each of the tasks beforehand. As more and more open-source knowledge resources become available, it is worthwhile trying to make use of open-source knowledge resources for service robots. The challenge lies in the autonomous identification, acquisition and utilization of missing knowledge about a user task at hand. In this paper, the core problem is formalized and the complexity results of the main reasoning issues are provided. A mechanism for task planning with open-knowledge rules which are provided by non-experts in semi-structured natural language and thus generally underspecified are introduced. Techniques for translating the semi-structured knowledge from a large open-source knowledge base are also presented. Experiments showed a remarkable improvement of the system performance on a test set consisting of hundreds of user desires from the open-source knowledge base. Autonomous service robots will have to understand vaguely described tasks, such as âset the tableâ or âclean upâ. Performing such tasks as intended requires robots to fully, precisely, and appropriately parameterize their low-level control programs. We propose knowledge processing as a computational resource for enabling robots to bridge the gap between vague task descriptions and the detailed information needed to actually perform those tasks in the intended way. In this article, we introduce the KnowRob knowledge processing system that is specifically designed to provide autonomous robots with the knowledge needed for performing everyday manipulation tasks. The system allows the realization of âvirtual knowledge basesâ: collections of knowledge pieces that are not explicitly represented but computed on demand from the robot's internal data structures, its perception system, or external sources of information. This article gives an overview of the different kinds of knowledge, the different inference mechanisms, and interfaces for acquiring knowledge from external sources, such as the robot's perception system, observations of human activities, Web sites on the Internet, as well as Web-based knowledge bases for information exchange between robots. We evaluate the system's scalability and present different integrated experiments that show its versatility and comprehensiveness. Correctly interpreting human instructions is the first step to human-robot interaction. Previous approaches to semantically parsing the instructions relied on large numbers of training examples with annotation to widely cover all words in a domain. Annotating large enough instructions with semantic forms needs exhaustive engineering efforts. Hence, we propose propagating the semantic lexicon to learn a semantic parser from limited annotations, whereas the parser still has the ability of interpreting instructions on a large scale. We assume that the semantically-close words have the same semantic form based on the fact that human usually uses different words to refer to a same object or task. Our approach softly maps the unobserved words phrases to the semantic forms learned from the annotated copurs through a metric for knowledge-based lexical similarity. Experiments on the collected instructions showed that the semantic parser learned with lexicon propagation outperformed the baseline. Our approach provides an opportunity for the robots to understand the human instructions on a large scale. As more and more open knowledge resources become available, it is interesting to explore opportunities of enhancing autonomous agents' capacities by utilizing the knowledge in these resources, instead of hand-coding knowledge for agents. A major challenge towards this goal lies in the translation of the open knowledge organized in multiple modes, unstructured or semi-structured, into the internal representations of agents. In this paper we present a set of multi-mode NLP techniques to formalize the open knowledge for autonomous agents. Two case studies are reported in which our robot, equipped with the multi-mode NLP techniques, succeeded in acquiring knowledge from the microwave oven manual and from the open knowledge database, OMICS, and solving problems that could not be solved before the robot acquired the knowledge. Experiments for evaluating the performance of our approach show that our approach is promising. This paper presents an effort to enable robots to utilize open-source knowledge resources autonomously for human-robot interaction. The main challenges include how to extract knowledge in semi-structured and unstructured natural languages, how to make use of multiple types of knowledge in decision making, and how to identify the knowledge that is missing. A set of techniques for multi-mode natural language processing, integrated decision making, and open knowledge searching is proposed. The OK-KeJia robot prototype is implemented and evaluated, with special attention to two tests on 11,615 user tasks and 467 user desires. The experiments show that the overall performance improves remarkably due to the use of appropriate open knowledge.