Robots are required to execute increasingly complex instructions in dynamic environments, which can lead to a disconnect between the user's intent and the robot's representation of the instructions. In this paper we present a natural language instruction grounding framework which uses formal synthesis to enable the robot to identify necessary environment assumptions for the task to be successful. These assumptions are then expressed via natural language questions referencing objects in the environment. The user is prompted to confirm or reject the assumption. We demonstrate our approach on two tabletop pick-and-place tasks.
Existing work that leveraged formal synthesis to generate verbal feedback @cite_21 relied on manually-defined groundings for actions, whereas our model learns these symbols. Other work in formally representing robot instructions uses Combinatorial Categorical Grammars to infer logical representations of navigation instructions @cite_11 .
@cite_21 @cite_11
2295357975 46490633
Abstract : This paper addresses the challenge of enabling non-expert users to command robots to perform complex highleveltasks using natural language. It describes an integrated system that combines the power of formalmethods with the accessibility of natural language, providing correct-by-construction controllers for high-levelspecifications that can be implemented, and easy-to-understand feedback to the user on those that cannot be achieved.This is among the first works to close this feedback loop, enabling users to interact with the robot in order to identifya succinct cause of failure and obtain the desired controller. The supported language and logical capabilities areillustrated using examples involving a robot assistant in a hospital. As robots become more ubiquitous and capable of performing complex tasks, the importance of enabling untrained users to interact with them has increased. In response, unconstrained natural-language interaction with robots has emerged as a significant research area. We discuss the problem of parsing natural language commands to actions and control structures that can be readily implemented in a robot execution system. Our approach learns a parser based on example pairs of English commands and corresponding control language expressions. We evaluate this approach in the context of following route instructions through an indoor environment, and demonstrate that our system can learn to translate English commands into sequences of desired actions, while correctly capturing the semantic intent of statements involving complex control structures. The procedural nature of our formal representation allows a robot to interpret route instructions online while moving through a previously unknown environment.