NeuralNetwork-based features
As illustrated in Figure REF , we have different deep learning models to extract features separately for both word embedding and similarity vectors matrices. In our work, we experiment with Convectional Neural Network (CNN) models and Long short-term memory (LSTM) models. Variations of CNN models where applied to NLP downstream tasks as feature extraction methods for text classification. In our work, we used two variations of CNN. In one model, we used a CNN as a one-head 1D-CNN with kernel size of 5 allowing the model to extract features with 5-grams vectors using 32 filters. Followed by a max pooling layer with pool size of 2 then flattened layer. In another model, we used a CNN as a multi-headed 2D-CNN with 1, 2, 3, 5 grams filter sizes, initialising the kernel weights with a Rectified Linear Unit (ReLU) activation function and normal distribution weights. Followed by a max pooling layer with different pooling sizes taken as one columns pooling filter with the maximum text length excluding few grams sizes. For the LSTM, we used two variants. One is a simple bidirectional LSTM of 64 units followed by concatenations of max pooling and average pooling layers, and attention bidirectional LSTM proposed by [1]} using 64 units followed by 128 units then attention layershttps://www.kaggle.com/mlwhiz/attention-pytorch-and-keras.
