<commit_before>README.txt
==========

Building
--------

In order to build kuromoji from source, please do as follows:

1. Provide Java with 512MB heap size

    % export MAVEN_OPTS=-Xmx512m
  
2. Run Maven using

    % mvn -Ddownload=true package

   to download the dictionary data and build a distribution

3. A distribution should now be available as

    target/kuromoji-<version>.tar.gz (and .zip)

  and a jar file should also be available in the target directory<commit_msg>Stylistic changes<commit_after>README.txt
==========

Building
--------

In order to build kuromoji from source, please do as follows:

1. Provide Java with 512MB heap size

    `% export MAVEN_OPTS=-Xmx512m`
  
2. Run Maven using

    `% mvn -Ddownload=true package`

   to download the dictionary data and build a distribution

3. A distribution should now be available as

    `target/kuromoji-<version>.tar.gz # (and .zip)`

  and a jar file should also be available in the target directory
{'fraction_non_alphanumeric': 0.0932377049180328, 'fraction_numerical': 0.018442622950819672, 'mean_word_length': 3.303964757709251, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5464251', 'n_tokens_mistral': 316, 'n_tokens_neox': 290, 'n_words': 122}