Though Baroni et al. (2014) suggested that predictive models which use neural networks to generate the distributed word representations (also known as embeddings in this context) outperform counting models which work on co-occurrence matrices, recent work shows evidence to the contrary (Levy et al., 2014;<cite> Salle et al., 2016)</cite> . In this paper, we focus on improving a state-ofthe-art counting model, LexVec <cite>(Salle et al., 2016)</cite> , which performs factorization of the positive pointwise mutual information (PPMI) matrix using window sampling and negative sampling (WSNS).