@inproceedings{
xu2023creative,
title={Creative Robot Tool Use with Large Language Models},
author={Mengdi Xu and Wenhao Yu and Peide Huang and Shiqi Liu and Xilun Zhang and Yaru Niu and Tingnan Zhang and Fei Xia and Jie Tan and Ding Zhao},
booktitle={NeurIPS 2023 Foundation Models for Decision Making Workshop},
year={2023},
url={https://openreview.net/forum?id=ghGXVeXqEt}
}
Tool use is a hallmark of advanced intelligence, exemplified in both animal behavior and robotic capabilities. This paper investigates the feasibility of imbuing robots with the ability to creatively use tools in tasks that involve implicit physical constraints and long-term planning. Leveraging Large Language Models (LLMs), we develop RoboTool, a system that accepts natural language instructions and outputs executable code for controlling robots in both simulated and real-world environments. RoboTool incorporates four pivotal components: (i) an “Analyzer” that interprets natural language to discern key task-related concepts, (ii) a “Planner” that generates comprehensive strategies based on the language input and key concepts, (iii) a “Calculator” that computes parameters for each skill, and (iv) a “Coder” that translates these plans into executable Python code. Our results show that RoboTool can not only comprehend implicit physical constraints and environmental factors but also demonstrate creative tool use. Unlike traditional Task and Motion Planning (TAMP) methods that rely on explicit optimization and are confined to formal logic, our LLM-based system offers a more flexible, efficient, and user-friendly solution for complex robotics tasks. Through extensive experiments, we validate that RoboTool is proficient in handling tasks that would otherwise be infeasible without the creative use of tools, thereby expanding the capabilities of robotic systems.
Creative Robot Tool Use with Large Language Models
Mengdi Xu Wenhao Yu Peide Huang Shiqi Liu Xilun Zhang Yaru Niu Tingnan Zhang Fei Xia Jie Tan Ding Zhao