Image encoder \(f\)  and text encoder \(g\) . We adopt an image encoder \( f\)  to extract the visual vector representations and sub-region features from the image samples. Furthermore, we utilize a text encoder \(g\)  to extract the textual vector representations and word features from the text samples. The text encoder \(g\)  is shared in the framework. Our architecture has the flexibility of allowing various choices of deep neural network models.
To have a fair comparison with the baselines, we adopt the same Inception-v3 [1]} and Bi-directional Long Short-Term
Memory (Bi-LSTM) [2]} to instantiate the image encoder \(f\)  and text encoder \(g\) .
