Given the image and one corresponding ground-truth caption \((x, y)\) , we use a pre-trained Faster R-CNN network [1]} to extract a sequence of features \(R \in \mathbb {R}^{L_r \times d}\)  from \(x\)  where \(L_r\)  and \(d\)  denote the number of the detected objects and the dimension of features, respectively. The pre-trained Faster R-CNN network is fixed during further training. Recent work mainly formulated image captioning as a single-task learning problem, directly maximizing \(p(y|R)\)  while we formulate image captioning as a DMT problem, which maximizes the log-probability:
\(\log p(y, m|R) = \log p(m|R) + \log p(y|R, m)\) 
