<commit_before>"""Patched version of standard library tokenize, to deal with various bugs.

Based on Python 3.2 code.

Patches:

- <NAME>' patch for Python issue #12691 (untokenizing)
  - Except we don't encode the output of untokenize
  - Python 2 compatible syntax, so that it can be byte-compiled at installation
- Newlines in comments and blank lines should be either NL or NEWLINE, depending
  on whether they are in a multi-line statement. Filed as Python issue #17061.
- Export generate_tokens & TokenError
- u and rb literals are allowed under Python 3.3 and above.

------------------------------------------------------------------------------
Tokenization help for Python programs.

tokenize(readline) is a generator that breaks a stream of bytes into
Python tokens.  It decodes the bytes according to PEP-0263 for
determining source file encoding.

It accepts a readline-like method which is called repeatedly to get the
next line of input (or b"" for EOF).  It generates 5-tuples with these
members:

    the token type (see token.py)
    the token (a string)<commit_msg>Add linebreak to docstring to fix markup and silence Sphinx waraning.<commit_after>"""Patched version of standard library tokenize, to deal with various bugs.

Based on Python 3.2 code.

Patches:

- <NAME>' patch for Python issue #12691 (untokenizing)
  - Except we don't encode the output of untokenize
  - Python 2 compatible syntax, so that it can be byte-compiled at installation
- Newlines in comments and blank lines should be either NL or NEWLINE, depending
  on whether they are in a multi-line statement. Filed as Python issue #17061.
- Export generate_tokens & TokenError
- u and rb literals are allowed under Python 3.3 and above.

------------------------------------------------------------------------------

Tokenization help for Python programs.

tokenize(readline) is a generator that breaks a stream of bytes into
Python tokens.  It decodes the bytes according to PEP-0263 for
determining source file encoding.

It accepts a readline-like method which is called repeatedly to get the
next line of input (or b"" for EOF).  It generates 5-tuples with these
members:

    the token type (see token.py)
    the token (a string)
{'fraction_non_alphanumeric': 0.11820224719101123, 'fraction_numerical': 0.017977528089887642, 'mean_word_length': 2.884816753926702, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4119770', 'n_tokens_mistral': 594, 'n_tokens_neox': 553, 'n_words': 307}