Our work is based on the recently released <cite>Multimodal Dialogue</cite> (<cite>MMD</cite>) dataset <cite>(Saha et al., 2017)</cite> in the fashion domain. We introduce a multimodal extension to the Hierarchical Recurrent Encoder-Decoder (HRED) model and show that this extension outperforms strong baselines in terms of text-based similarity metrics.