<commit_before>Preparing Code Samples
================


## Collect Rosetta Code

1. Clone the [RosettaCodeData](https://github.com/acmeism/RosettaCodeData)
  ```
  git clone <EMAIL>:acmeism/RosettaCodeData.git
  ```
2. Build this `misc` executable
  ```
  go build
  ```
3. Run the `collectRosetta` according to the cloned RosettaCodeData, and collect
   files to `../samples`
  ```
  ./misc collectRosetta path/to/clones/RosettaCodeData ../samples
  ```

## Build Bag-of-Words and Convert Samples to Libsvm

```
./misc convertLibsvm ../samples
```<commit_msg>Update convertLibsvm README.<commit_after>Preparing Code Samples
================


## Collect Rosetta Code

1. Clone the [RosettaCodeData](https://github.com/acmeism/RosettaCodeData)
  ```
  git clone <EMAIL>:acmeism/RosettaCodeData.git
  ```
2. Build this `misc` executable
  ```
  go build
  ```
3. Run the `collectRosetta` according to the cloned RosettaCodeData, and collect
   files to `../samples`
  ```
  ./misc collectRosetta path/to/clones/RosettaCodeData ../samples
  ```

## Build Bag-of-Words and Convert Samples to Libsvm

```
./misc convertLibsvm ../samples ../samples.libsvm
```
{'fraction_non_alphanumeric': 0.16017316017316016, 'fraction_numerical': 0.005194805194805195, 'mean_word_length': 3.96137339055794, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4688891', 'n_tokens_mistral': 408, 'n_tokens_neox': 366, 'n_words': 99}