<commit_before>Distributed Word Embedding
==========

The Distributed Word Embedding tool is a parallelization of the Word2Vec algorithm on top of our DMTK parameter server. It provides an efficient "scaling to industry size" solution for word embedding.

For more details, please view our website [http://www.dmtk.io](http://www.dmtk.io).

#Linux Installation

cmake CMakeLists.txt

make

# Windows Installation

1. Get and build the DMTK Framework [multiverso](https://github.com/Microsoft/multiverso.git).

2. Open distributed_word_embedding/distributed_word_embedding.sln, change configuration and platform to Release and x64, set the ```include``` and ```lib``` path of multiverso in project property.

3. Enable openmp 2.0 support.

   To set this compiler option in the Visual Studio development environment
  
   1)Open the project's Property Pages dialog box. For details, see How to: Open Project Property Pages.
  
   2)Expand the Configuration Properties node.
  
   3)Expand the C/C++ node.
  
   4)Select the Language property page.
  
   5)Modify the OpenMP Support property.

4. Build the solution.<commit_msg>update the README.md<commit_after>Distributed Word Embedding
==========

The Distributed Word Embedding tool is a parallelization of the Word2Vec algorithm on top of our DMTK parameter server. It provides an efficient "scaling to industry size" solution for word embedding.

For more details, please view our website [http://www.dmtk.io](http://www.dmtk.io).

#Linux Installation

1. cmake ./CMakeLists.txt

2. make

# Windows Installation

1. Get and build the DMTK Framework [multiverso](https://github.com/Microsoft/multiverso.git).

2. Open distributed_word_embedding/distributed_word_embedding.sln, change configuration and platform to Release and x64, set the ```include``` and ```lib``` path of multiverso in project property.

3. Enable openmp 2.0 support.

   To set this compiler option in the Visual Studio development environment
  
   1)Open the project's Property Pages dialog box. For details, see How to: Open Project Property Pages.
  
   2)Expand the Configuration Properties node.
  
   3)Expand the C/C++ node.
  
   4)Select the Language property page.
  
   5)Modify the OpenMP Support property.

4. Build the solution.
{'fraction_non_alphanumeric': 0.08072342302602559, 'fraction_numerical': 0.013233348037053375, 'mean_word_length': 4.154545454545454, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '230972', 'n_tokens_mistral': 660, 'n_tokens_neox': 629, 'n_words': 271}