Prior approaches propose to address this issue by learning about useful sub-routines directly from data [1]}, [2]}, [3]}, [4]}.
Of particular interest is the fully unsupervised setting, where the learner is only given access to state-action trajectories from an (expert) policy. Two relevant works can be distinguished: CompILE [3]} and Ordered Memory Policy Network (OMPN) [4]}.
In CompILE, expert trajectories are modeled using a latent variable model based on a recurrent neural network (RNN), which infers a pre-determined number of boundary points by iteratively processing the entire trajectory multiple times to recover segments associated with reusable sub-routines.
Alternatively, OMPN equips an RNN with a multi-layer hierarchical memory, where information processing at different levels in the memory can be interpreted as belonging to separate segments.
While OMPN additionally includes a strong preference for capturing hierarchical relationships between different sub-routines, both methods are limited insofar that they process the trajectory in an entirely sequential manner.
This prevents them from revising earlier decisions about boundary points in light of new information that becomes only accessible at a future stage.
Moreover, iteratively processing the sequence multiple times (as in CompILE) or interfacing with a deep hierarchical memory (as in OMPN) incurs significant computational costs.
