LSTM-Attn: The second LSTM architecture we implemented, LSTM-Attn, to perform sentiment analysis in the text is based on the aforementioned LSTM architecture, but with the addition of an Attention layer [1]} between the LSTM layer and the Linear layer. The idea behind the Attention mechanism is to increase the importance of specific parts of the input sentence [2]}. There are two types of attention mechanisms, the global and the local ones. The difference is that in the global mechanisms, all the hidden states of the previous layer are considered for deriving the context vector, whilst on the local attention mechanisms, only some hidden states are considered [3]}. In this implementation, the Attention layer is a global attention mechanism that computes the soft alignment score between the output of the LSTM and its final hidden state.
