In order to speed up the training process, we implement momentum optimizer and mini-batch with a batch size of 1,000 for oSG training and stochastic gradient descent for SGNS, as in [1]}. Learning rate \(\alpha \)  for oSG is set to 1.0 and momentum is 0.9. For SGNS, \(\alpha \)  is choosen to be 0.1, and the number of negative samples is 15. The number of epochs is chosen so that the negative log likelihood value is not significantly reduced. For instance, in the case of oSG, 200 and 90 epochs respectively were selected for text8 and Wikipedia, while for SGNS, 15 epochs were selected for text8. Practically, these optimization settings achieve the best performance in our experiment. For example, the best word analogy (WA) scores for text8 are 32.6% (SGNS) and 38.6% (oGS), the corresponding value for Wikipedia is 50.5% (oNS).
