Deep Learning in Natural Language Processing
Deep learning has emerged as a highly efficient technique in machine learning to perform text analytics, which includes text classification, text sequence learning, sentiment analysis, question answer engine, etc. This paper presents the concept and steps of using deep learning in NLP, ways to pull out the features from the text, ways to prepare data for deep learning and popular methods that are being used for NLP, including RNN, CNN, LSTM. We also present the results of a working example based on a movie review shared by IMDB. Feature Representation When feeding textual data to Artificial Neural Networks (ANN) we need to consider few points about the way we create the data. We input the data vector x in dimensions and generate the output in out dimensions. Instead of creating an individual dimension for each and every feature, we need to embed each feature into a D Dimensional space and represent as a dense vector in the space. The main advantage of dense vector representation is that if there exists a similarity sequence which we can learn, that can be captured best in dense vector representation. In the example presented, we used continuous bag ofwords (CBOW) approach to build our dense vectors. Feature Extraction Let us now look at some of the methods used to extract features from the given text: CBOW: Continuous bag of words is a dense vector based feature extraction approach. Neural networks assume a fixed input. Thus for training the NN we need to extract features from textual data. CBOW approach enables feature extraction by expressing each word in the training data set as unique feature. This creates a vector representation for each sentence to be used in training NN. For specific importance association we provide a weighted vector as an input to NN. This weight association is done majorly using two approaches summing or averaging of embedded features and weighted averaging of embedded features. An example is tf idf of collection of bag of words and giving weight to different dimensions based upon importance. Note: We can adopt other methods also as discussed in section titled 'Word Embedding' Distance and Position Features: The distance can be calculated by subtracting the feature vectors of the corresponding words, and this information can be used to train the Neural Networks. Dimensionality: Decision needs to be made on the dimensionality of the embedded vectors. It is a trade off between processing speed and task accuracy. In our example, we used the context window of ten words. Introduction In recent years multi layer neural networks are gaining back their importance in the field of Natural Language Processing (NLP) as they are becoming capable of generating state of the art results with a relatively decent speed due to utilization of Graphical Processing Units. The introduction of different methods in multi layer neural network, such as Recurrent Neural Networks (RNN) Convolutional Neural Networks (CNN) Longest State Time Memory Neural Networks (LSTM) has proved to be very efficient in generating better results. In this paper, we will discuss different methods that can be adopted to perform NLP using deep learning methodologies. We will start our discussion with representation methods available to present the data, then we move to feed forward NN, RNN, CNN and see how to use these methods for NLP. 3 Training Functions Given below are the types of functions that are used in modeling deep learning neural networks. The selection of these functions depends majorly on the problem statement. Non linear Functions Neural networks are universal approximators and thus can approximate any non zero linear or non linear function. The two basic things that should be kept in mind before selecting a non linear function are: 1) It should cover the entire input range, from negative infinity to positive infinity, as its domain. 2) Its output should be bounded in a set range. The popular non linear functions used are: Sigmoid, Hard Tanh, Hyperbolic tangent, Rectifier, Relu. Output Transformation Functions In many cases we use a transformation function in the outer most layer of neural network. The most popular function used here is softmax, to represent multi class classification. Loss Function When training the neural network, we require a loss function which acts as an indicator to identify how far is the network trained output from the actual output. Selection of lost function depends upon the type, requirement and extent of tolerance of loss function. Most popular loss functions are: Hinge binary, Hinge multiclass, log loss, categorical cross entropy loss, ranking loss. Word Embedding Let us now look at the methodology to represent each of the features into a low dimensionality space. This technique is popularly used for sequential pattern training of the sentences in the documents. Continuous Bag of Words This approach is used to capture the sequential information in the sentences. The process is as follows: Total number of unique words are collected and all the words are given a vector representation, this is called one hot vector representation. Depending upon the window size, words that are sequential part of a sentence are collected and a training model is developed. E.g. A quick brown fox jumps right over a lazy dog. In the above sentence, suppose we represent a word with one word to its left and one to its right, then the training data would look like: (a, brown),quick](quick,fox),brown] etc. Then a neural network is taken with input layer's size C(Context)*V(vocabulary size) and a hidden layer's size N(heuristic) and output layer's size V(vocabulary size) An optimization technique like gradient descent, is used to train the model. Figure 1 represents single word context CBOW.