Natural language processing (NLP) models often require a massive number of parameters for word embeddings, resulting in a large storage or memory footprint. Deploying neural NLP models to mobile devices requires compressing the word embeddings without any significant sacrifices in performance. For this purpose, we propose to construct the embeddings with few basis vectors. For each word, the composition of basis vectors is determined by a hash code. To maximize the compression rate, we adopt the multi-codebook quantization approach instead of binary coding scheme. Each code is composed of multiple discrete numbers, such as (3, 2, 1, 8), where the value of each component is limited to a fixed range. We propose to directly learn the discrete codes in an end-to-end neural network by applying the Gumbel-softmax trick. Experiments show the compression rate achieves 98% in a sentiment analysis task and 94% ~ 99% in machine translation tasks without performance loss. In both tasks, the proposed method can improve the model performance by slightly lowering the compression rate. Compared to other approaches such as character-level segmentation, the proposed method is language-independent and does not require modifications to the network architecture. Word embeddings play an important role in neural-based natural language processing (NLP) models. Neural word embeddings encapsulate the linguistic information of words in continuous vectors. However, as each word is assigned an independent embedding vector, the number of parameters in the embedding matrix can be huge. For example, when each embedding has 500 dimensions, the network has to hold 100M embedding parameters to represent 200K words. In practice, for a simple sentiment analysis model, the word embedding parameters account for 98.8% of the total parameters.As only a small portion of the word embeddings is selected in the forward pass, the giant embedding matrix usually does not cause a speed issue. However, the massive number of parameters in the neural network results in a large storage or memory footprint. When other components of the neural network are also large, the model may fail to fit into GPU memory during training. Moreover, as the demand for low-latency neural computation for mobile platforms increases, some neural-based models are expected to run on mobile devices. Thus, it is becoming more important to compress the size of NLP models for deployment to devices with limited memory or storage capacity.In this study, we attempt to reduce the number of parameters used in word embeddings without hurting the model performance. Neural networks are known for the significant redundancy in the connections BID8 . In this work, we further hypothesize that learning independent embeddings causes more redundancy in the embedding vectors, as the inter-similarity among words is ignored. Some words are very similar regarding the semantics. For example, "dog" and "dogs" have almost the same meaning, except one is plural. To efficiently represent these two words, it is desirable to share information between the two embeddings. However, a small portion in both vectors still has to be trained independently to capture the syntactic difference.Following the intuition of creating partially shared embeddings, instead of assigning each word a unique ID, we represent each word w with a code C w = (C an integer number in [1, K] . Ideally, similar words should have similar codes. For example, we may desire C dog = (3, 2, 4, 1) and C dogs = (3, 2, 4, 2). Once we have obtained such compact codes for all words in the vocabulary, we use embedding vectors to represent the codes rather than the unique words. More specifically, we create M codebooks E 1 , E 2 , ..., E M , each containing K codeword vectors. The embedding of a word is computed by summing up the codewords corresponding to all the components in the code as DISPLAYFORM0 where DISPLAYFORM1 w -th codeword in the codebook E i . In this way, the number of vectors in the embedding matrix will be M × K, which is usually much smaller than the vocabulary size. FIG0 gives an intuitive comparison between the compositional approach and the conventional approach (assigning unique IDs). The codes of all the words can be stored in an integer matrix, denoted by C. Thus, the storage footprint of the embedding layer now depends on the total size of the combined codebook E and the code matrix C.Although the number of embedding vectors can be greatly reduced by using such coding approach, we want to prevent any serious degradation in performance compared to the models using normal embeddings. In other words, given a set of baseline word embeddingsẼ(w), we wish to find a set of codesĈ and combined codebookÊ that can produce the embeddings with the same effectiveness asẼ(w). A safe and straight-forward way is to minimize the squared distance between the baseline embeddings and the composed embeddings as DISPLAYFORM2 where |V | is the vocabulary size. The baseline embeddings can be a set of pre-trained vectors such as word2vec BID29 or GloVe BID34 embeddings.In Eq. 3, the baseline embedding matrixẼ is approximated by M codewords selected from M codebooks. The selection of codewords is controlled by the code C w . Such problem of learning compact codes with multiple codebooks is formalized and discussed in the research field of compressionbased source coding, known as product quantization BID16 and additive quantization BID1 BID28 . Previous works learn compositional codes so as to enable an efficient similarity search of vectors. In this work, we utilize such codes for a different purpose, that is, constructing word embeddings with drastically fewer parameters.Due to the discreteness in the hash codes, it is usually difficult to directly optimize the objective function in Eq. 3. In this paper, we propose a simple and straight-forward method to learn the codes in an end-to-end neural network. We utilize the Gumbel-softmax trick BID27 BID15 to find the best discrete codes that minimize the loss. Besides the simplicity, this approach also allows one to use any arbitrary differentiable loss function, such as cosine similarity.The contribution of this work can be summarized as follows:• We propose to utilize the compositional coding approach for constructing the word embeddings with significantly fewer parameters. In the experiments, we show that over 98% of the embedding parameters can be eliminated in sentiment analysis task without affecting performance. In machine translation tasks, the loss-free compression rate reaches 94% ∼ 99%.• We propose a direct learning approach for the codes in an end-to-end neural network, with a Gumbel-softmax layer to encourage the discreteness.• The neural network for learning codes will be packaged into a tool 1 . With the learned codes and basis vectors, the computation graph for composing embeddings is fairly easy to implement, and does not require modifications to other parts in the neural network. In this work, we propose a novel method for reducing the number of parameters required in word embeddings. Instead of assigning each unique word an embedding vector, we compose the embedding vectors using a small set of basis vectors. The selection of basis vectors is governed by the hash code of each word. We apply the compositional coding approach to maximize the storage efficiency. The proposed method works by eliminating the redundancy inherent in representing similar words with independent embeddings. In our work, we propose a simple way to directly learn the discrete codes in a neural network with Gumbel-softmax trick. The results show that the size of the embedding layer was reduced by 98% in IMDB sentiment analysis task and 94% ∼ 99% in machine translation tasks without affecting the performance.Our approach achieves a high loss-free compression rate by considering the semantic inter-similarity among different words. In qualitative analysis, we found the learned codes of similar words are very close in Hamming space. As our approach maintains a dense basis matrix, it has the potential to be further compressed by applying pruning techniques to the dense matrix. The advantage of compositional coding approach will be more significant if the size of embedding layer is dominated by the hash codes.A APPENDIX: SHARED CODESIn both tasks, when we use a small code decomposition, we found some hash codes are assigned to multiple words. Table 6 lists some samples of shared codes with their corresponding words from the sentiment analysis task. This phenomenon does not cause a problem in either task, as the words only have shared codes when they have almost the same sentiments or target translations.shared code words 4 7 7 0 4 7 1 1 homes cruises motel hotel resorts mall vacations hotels 6 6 7 1 4 0 2 0 basketball softball nfl nascar baseball defensive ncaa tackle nba 3 7 3 2 4 3 3 0 unfortunately hardly obviously enough supposed seem totally ... 4 6 7 0 4 7 5 0 toronto oakland phoenix miami sacramento denver minneapolis ... 7 7 6 6 7 3 0 0 yo ya dig lol dat lil bye Table 6 : Examples of words sharing same codes when using a 8 × 8 code decomposition B APPENDIX: SEMANTICS OF CODES In order to see whether each component captures semantic meaning. we learned a set of codes using a 3 x 256 coding scheme, this will force the model to decompose each embedding into 3 vectors. In order to maximize the compression rate, the model must make these 3 vectors as independent as possible. Table 7 : Some code examples using a 3 × 256 coding scheme.As we can see from Table 7 , we can transform "man/king" to "woman/queen" by change the subcode "210" in the first component to "232". So we can think "210" must be a "male" code, and "232" must be a "female" code. Such phenomenon can also be observed in other words such as city names.
Compressing the word embeddings over 94% without hurting the performance.
{"src": "scitldr-AIC_train", "paper_id": "BJRZzFlRb"}