<commit_before>+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
| Short  | Long                                   | Meaning                                                                                                                                                  |
+========+========================================+==========================================================================================================================================================+
|        | ``--num-reducers``                     | Number of reduce tasks. Specify 0 to only perform map phase                                                                                              |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|        | ``--no-override-home``                 | Don't set the script's HOME directory to the $HOME in your environment.  Hadoop will set it to the value of the 'mapreduce.admin.user.home.dir' property |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|        | ``--no-override-env``                  | Use the default python executable and environment instead of overriding HOME, LD_LIBRARY_PATH and PYTHONPATH                                             |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``-D`` |                                        | Set a Hadoop property, e.g., -D mapred.compress.map.output=true                                                                                          |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|        | ``--python-zip``                       | Additional python zip file                                                                                                                               |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|        | ``--upload-file-to-cache``             | Upload and add this file to the distributed cache.                                                                                                       |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|        | ``--upload-archive-to-cache``          | Upload and add this archive file to the distributed cache.                                                                                               |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|        | ``--log-level``                        | Logging level                                                                                                                                            |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|        | ``--job-name``                         | name of the job                                                                                                                                          |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+<commit_msg>Document new options<commit_after>+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
| Short  | Long                                   | Meaning                                                                                                                                                  |
+========+========================================+==========================================================================================================================================================+
|        | ``--num-reducers``                     | Number of reduce tasks. Specify 0 to only perform map phase                                                                                              |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|        | ``--no-override-home``                 | Don't set the script's HOME directory to the $HOME in your environment.  Hadoop will set it to the value of the 'mapreduce.admin.user.home.dir' property |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|        | ``--no-override-env``                  | Use the default python executable and environment instead of overriding HOME, PATH, LD_LIBRARY_PATH and PYTHONPATH                                       |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|        | ``--no-override-ld-path``              | Use the default LD_LIBRARY_PATH, instead of copying it from the client                                                                                   |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|        | ``--no-override-pypath``               | Use the default PYTHONPATH, instead of copying it from the client                                                                                        |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|        | ``--no-override-path``                 | Use the default PATH, instead of copying it from the client                                                                                              |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|        | ``--set-env``                          | Set an environment variable for task. Specify it as VAR=value.                                                                                           |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``-D`` |                                        | Set a Hadoop property, e.g., -D mapred.compress.map.output=true                                                                                          |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|        | ``--python-zip``                       | Additional python zip file                                                                                                                               |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|        | ``--upload-file-to-cache``             | Upload and add this file to the distributed cache.                                                                                                       |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|        | ``--upload-archive-to-cache``          | Upload and add this archive file to the distributed cache.                                                                                               |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|        | ``--log-level``                        | Logging level                                                                                                                                            |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|        | ``--job-name``                         | name of the job                                                                                                                                          |
+--------+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
{'fraction_non_alphanumeric': 0.5453934095494284, 'fraction_numerical': 0.00019214141608223654, 'mean_word_length': 0.26227719170607494, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 105, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7295250', 'n_tokens_mistral': 916, 'n_tokens_neox': 825, 'n_words': 244}