First, we choose a real-world instructional video dataset CrossTask [1]} to conduct our experiments. CrossTask comprises \(2,750\)  videos (212 hours). Each video depicts one of the 18 primary long-horizon tasks—for example, Make Pancakes, Add Oil to Your Car or Make Lemonade. To test the trained agent’s generalization capability, for the videos in each task, we randomly divide the videos in each task into \(70\%/30\%\)  splits for training and testing.
There are various procedure steps in different tasks. For example, the most simple task is "Jack Up a Car", which only contains 3 steps. Complex tasks such as "Grill Steak", "Make Bread and Butter Pickles", "Pickle Cucumbers", and "Change a Tire" require 11 steps to finish. For each video, there are dense temporal boundaries and action labels that describe the person's actions in the video.
Each video can be regarded as a sequence of images \(V_i\)  (where \(i\)  is the index of frames) that have annotated with a sequence of action labels \(a_j\)  and each action starts at frame index \(s_j\)  and ends at frame index \(e_j\) .
Same as the setup of [2]}: we choose frames around the beginning of the captions \(V_{s_t-\delta /2:s_t +\delta /2}\)  as \(o_t\)  , caption description \(a_t\)  as the semantic meaning of action, and images nearby the end \(V_{e_t-\delta /2:e_t +\delta /2}\)  as the next observation \(o_{t+1}\) . Here, \(\delta \)  controls the duration of each observation, and we set \(\delta = 2\)  for all data we have used in our paper.
Our state-space \(\mathcal {S}\)  is the pre-computed features provided in CrossTask: each second of the video is encoded into a \(3,200\) -dimensional feature vector
, which is a concatenation of the I3D [3]}, Resnet-152 [4]}, and audio VGG features [5]}.
The action space \(\mathcal {A}\)  is constructed by enumerating all combinations of predicates and objects, which provides 105 action labels and is shared across all 18 tasks.
Our method is suitable for modeling longer trajectories, but we restrict the experiments to horizontal lengths \(T=3\sim 4\)  to maintain a consistent comparison with state-of-the-art methods.
