<commit_before>Follows `datapackages`_ format.

- `datapackage.json format`_
- `json table schema`_ 

Structure
---------

.. code-block::

    # dataset metadata, schema information.
    datapackage.json

    # (future) when this package is stable, unihan.csv will be provided
    data/unihan.csv

    # script to download dataset and convert to clean CSV.
    scripts/process.py

    # cihai's python related modules
    __init__.py

Cihai is *not* required for:

    - ``data/unihan.csv`` - `simple data format`_ compatible csv file.
    - ``scripts/process.py`` - create a ``data/unihan.csv``.

When this module is stable, ``data/unihan.csv`` will have prepared
releases, without requires using ``scripts/process.py``. ``process.py``
will not require external libraries.

Intended usage
--------------

(Not ready)

.. code-block:: bash

    $ ./scripts/process.py

Creates ``data/unihan.csv``.
<commit_msg>unihan update README.rst<commit_after>Follows `datapackages`_ format.

- `datapackage.json format`_
- `json table schema`_ 

Structure
---------

.. code-block::

    # dataset metadata, schema information.
    datapackage.json

    # (future) when this package is stable, unihan.csv will be provided
    data/unihan.csv

    # script to download dataset and convert to clean CSV.
    scripts/process.py

    # cihai's python related modules, public-facing python API.
    __init__.py

    # unihan module code
    unihan.py


Cihai is *not* required for:

    - ``data/unihan.csv`` - `simple data format`_ compatible csv file.
    - ``scripts/process.py`` - create a ``data/unihan.csv``.

When this module is stable, ``data/unihan.csv`` will have prepared
releases, without requires using ``scripts/process.py``. ``process.py``
will not require external libraries.

Intended usage
--------------

(Not ready)

.. code-block:: bash

    $ ./scripts/process.py

Creates ``data/unihan.csv``.

{'fraction_non_alphanumeric': 0.13782219884271435, 'fraction_numerical': 0.0, 'mean_word_length': 2.937888198757764, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6517621', 'n_tokens_mistral': 626, 'n_tokens_neox': 576, 'n_words': 181}