@inproceedings{
ni2024peria,
title={{PERIA}: Perceive, Reason, Imagine, Act via Holistic Language and Vision Planning for Manipulation},
author={Fei Ni and Jianye HAO and Shiguang Wu and Longxin Kou and Yifu Yuan and Zibin Dong and Jinyi Liu and MingZhi Li and Yuzheng Zhuang and YAN ZHENG},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=zw2K6LfFI9}
}
Long-horizon manipulation tasks with general instructions often implicitly encapsulate multiple sub-tasks, posing significant challenges in instruction following.
While language planning is a common approach to decompose general instructions into stepwise sub-instructions, text-only guidance may lack expressiveness and lead to potential ambiguity. Considering that humans often imagine and visualize sub-instructions reasoning out before acting, the imagined subgoal images can provide more intuitive guidance and enhance the reliability of decomposition. Inspired by this, we propose **PERIA**(**PE**rceive, **R**eason, **I**magine, **A**ct), a novel framework that integrates holistic language planning and vision planning for long-horizon manipulation tasks with complex instructions, leveraging both logical and intuitive aspects of task decomposition.
Specifically, we first perform a lightweight multimodal alignment on the encoding side to empower the MLLM to perceive visual details and language instructions. 
The MLLM is then jointly instruction-tuned with a pretrained image-editing model to unlock capabilities of simultaneous reasoning of language instructions and generation of imagined subgoals. Furthermore, we introduce a consistency alignment loss to encourage coherent subgoal images and align with their corresponding instructions, mitigating potential hallucinations and semantic conflicts between the two planning manners.
Comprehensive evaluations across three task domains demonstrate that PERIA, benefiting from holistic language and vision planning, significantly outperforms competitive baselines in both instruction following accuracy and task success rate on complex manipulation tasks.
PERIA: Perceive, Reason, Imagine, Act via Holistic Language and Vision Planning for Manipulation
Fei Ni Jianye HAO Shiguang Wu Longxin Kou Yifu Yuan Zibin Dong Jinyi Liu MingZhi Li Yuzheng Zhuang YAN ZHENG