When the memory for training word embedding is also limited, we need to modify the training algorithms by introducing new data structures to reduce the bits used to encode the values. In practice, we found that in the stochastic gradient descent (SGD) iteration in word2vec algorithms (Mikolov et al., 2013a; <cite>Mikolov et al., 2013b)</cite> , the updating vector's values are often very small numbers (e.g., < 10 −5 ). In this case, if we directly apply the rounding method to certain precisions (e.g., 8 bits), the update of word vectors will always be zero. For example, the 8-bit precision is 2 −7 = 0.0078, so 10 −5 is not significant enough to update the vector with 8-bit values.