@inproceedings{
bu2024closedloop,
title={Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation},
author={Qingwen Bu and Jia Zeng and Li Chen and Yanchao Yang and Guyue Zhou and Junchi Yan and Ping Luo and Heming Cui and Yi Ma and Hongyang Li},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=1ptdkwZbMG}
}
Despite significant progress in robotics and embodied AI in recent years, deploying robots for long-horizon tasks remains a great challenge. Majority of prior arts adhere to an open-loop philosophy and lack real-time feedback, leading to error accumulation and undesirable robustness. A handful of approaches have endeavored to establish feedback mechanisms leveraging pixel-level differences or pre-trained visual representations, yet their efficacy and adaptability have been found to be constrained. Inspired by classic closed-loop control systems, we propose CLOVER, a closed-loop visuomotor control framework that incorporates feedback mechanisms to improve adaptive robotic control. CLOVER consists of a text-conditioned video diffusion model for generating visual plans as reference inputs, a measurable embedding space for accurate error quantification, and a feedback-driven controller that refines actions from feedback and initiates replans as needed. Our framework exhibits notable advancement in real-world robotic tasks and achieves state-of-the-art on CALVIN benchmark, improving by 8% over previous open-loop counterparts. Code and checkpoints are maintained at https://github.com/OpenDriveLab/CLOVER.
Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation
Qingwen Bu Jia Zeng Li Chen Yanchao Yang Guyue Zhou Junchi Yan Ping Luo Heming Cui Yi Ma Hongyang Li