<commit_before>==========
cnprep
==========

Chinese text preprocess
---------------------------

You can extract numbers, email, website, emoji, tex, and delete spaces, punctuations.

Install
-------------

.. code:: python
pip install cnprep

Usage
-----------

.. code:: python
from cnprep import Extractor
ext = Extractor(delete=True, args=['email', 'number'], blur=True, limit=5)
ext.extract(message)

.. code:: python
delete: delete the found info (except blur)
args: option
    e.g. ['email', 'number'] or 'email, number'
blur: convert Chinese to pinyin and extract useful numbers
limit: parameter for get_number

Also, you can use `ext.reset_param()` to reset the parameters.<commit_msg>readme.rst<commit_after>==========
cnprep
==========

Chinese text preprocess
---------------------------

You can extract numbers, email, website, emoji, tex, and delete spaces, punctuations.

Install
-------------

..
\>> pip install cnprep
:: python

Usage
-----------

..
from cnprep import Extractor
ext = Extractor(delete=True, args=['email', 'number'], blur=True, limit=5)
ext.extract(message)
:: python

..
delete: delete the found info (except blur)
args: option
    e.g. ['email', 'number'] or 'email, number'
blur: convert Chinese to pinyin and extract useful numbers
limit: parameter for get_number
:: python

Also, you can use `ext.reset_param()` to reset the parameters.
{'fraction_non_alphanumeric': 0.20304568527918782, 'fraction_numerical': 0.0014503263234227702, 'mean_word_length': 3.3125, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '40087', 'n_tokens_mistral': 415, 'n_tokens_neox': 392, 'n_words': 148}