In recent years, recurrent neural networks (RNNs) implemented by long short-term memory (LSTM) especially show good performances in sequence data processing and they are widely used as decoders to generate a natural language description from an image in many methods [3, <cite>4,</cite> 5, 6, 7] . High-performance approaches on convolutional neural networks (CNNs) have been proposed [8, 9] , which are employed to represent the input image with a feature vector for the caption generation [3, <cite>4,</cite> 5] .