We encoded the sentence embeddings from scratch using both LSTM [1]} and GRU [2]}. To be specific, the captions were firstly tokenized, and word embeddings were obtained using a bag of words. The word embeddings were initialized with GloVe [3]} unless the tokenized word does not exist in GloVe vocabulary. The word embeddings were forwarded by 2-layered LSTM and GRU both, then the concatenation of the final hidden state output from LSTM and GRU was used for sentence embedding. We observe no improvement by using the average of word embeddings by GloVe for sentence embedding. For image encoder, we used ResNet-101 and ResNet-152. Both text and image encoders were trained from scratch without pretraining on any external data. For training the LSTM and the GRU from scratch, we initialize the word embedding with the concatenation of three GloVe vectorshttps://github.com/stanfordnlp/GloVe learned from Wikipedia, Twitter, and Common Crawl that results in 900-dimensional input for the text encoder.
