In the input layer of neural models each word is mapped into embeddings. Mesnil2013InvestigationOR compared several embeddings, namely pre-trained SENNA [1]}, RNN Language Model (RNNLM) [2]}, and random embeddings. SENNA gives the best result compared to other embeddings, and, typically, further fine-tuning word embeddings improves performance. [3]} report that embeddings learned from scratch directly on ATIS data (task-specific embeddings) are better than SENNA. However, task-specific embeddings are composed not only by words but also by named entities (NE) and syntactic featuresGold named entity and syntactic information. NE improves performance significantly while part-of-speech only adds small benefits. Ravuri2015RecurrentNN emphasizes the importance of character representation to handle OOV issues.
