To address the model design issue, we discuss several recent solutions (He et al., 2016b; Li et al., 2016; <cite>Xiong et al., 2017)</cite> . We then focus on a new case study of hierarchical deep reinforcement learning for video captioning (Wang et al., 2018b) , discussing the techniques of leveraging hierarchies in DRL for NLP generation problems.