subsection2-8plus-4minus-46plus4minus4Qualitative Analysis
In qualitative study, we present a number of showcases to demonstrate the performance of MTM intuitively. Comparing to the results from UpDown attention [1]} model and ShowTell CNN-LSTM [2]} model shown in Fig. REF , the proposed MTM is able to generate sentences in a accurate, detailed and grammatically correct way.
Specifically, after modality transition, the language model can precisely identify object-interactions from the scene. For example, in the first row, the first example illustrates that the MTM can detect the fact that a man is “standing next to” rather then “riding” the elephant. The ability of differentiate human-object interactions also is also shown in the 4th image of the top row. Although only one man is holding the frisbee in that picture, from the action and pose we can conclude that a group of people are playing rather than only one people.
Furthermore, the details can be better preserved by exploiting global caption representations in MTM. The pictures on the 2nd column give accurate colours of the objects, while the 3rd image on the top and the 1st on the bottom reveal details such as “bottles of wine”, “two computer monitors” rather than the general expressions from other models.
<FIGURE><FIGURE>