On the other hand, attention has been proved very effective in action recognition [1]}. In HM-AN, to capture the implicit relationships between the inputs and outputs in sequence to sequence problems, we apply both hard and soft attention mechanisms to explicitly learn the important and relevant image features regarding the specific outputs. More detailed explanation follows.
