Procedure planning in instructional videos [1]} (as shown in Figure REF ) aims to make goal-conditioned decisions by planning a sequence of high-level actions that can bring the agent from current observation to the goal.
Planning in instructional videos is a meaningful task since the ability to perform effective planning is crucial for building an instruction following agent.
Although learning from instructional videos is natural to humans, it is challenging for the AI system because it requires understanding human behaviors in the videos, focusing on actions and intentions. How to learn structured and plannable state and action spaces directly from unstructured videos is the key technical challenge of our task. Other challenges include: 1) Learning to make accurate predictions with high-dimensional observations is still challenging [2]}, especially for visually complex long-horizon tasks. 2) Appearance gap between the training and validation dataset could be large for unstructured videos. Thus, the agent needs to have the generalization capability. 3) These gaps lead to decision errors that compound over the steps.
