We introduce a novel extension to the Hierarchical Recurrent Encoder-Decoder (HRED) model (Serban et al., 2016) and show that our implementation significantly outperforms the model of <cite>Saha et al. (2017)</cite> by modelling the full multimodal context.