<commit_before>Template of a dataset README.rst file
=====================================

When creating a dataset, its good idea to provide a small documentation of the structure and information of the data.

When creating such documentation, it is good practice to follow a common format to keep things similar.

The following scheme details a template format on how to write the documentation of a new dataset and what information to provide to the end user.

.. code-block:: bash

    Dataset Name
    ============

    <small intro covering the dataset.>

    <bonus: some sample images of the dataset are always good to have.
    Visual aid is often good enough to help dismiss a dataset before delving into the details of the data.>

    Properties
    ^^^^^^^^^^

    - name: <name alias of the dataset>
    - tasks:
        - classification:
    - data size: <total size of the dataset on disk>
    - metadata size: <total size of the processed hdf5 metdata file on disk for all tasks>
    - keywords: <keywords assign to the dataset. Example: object detection, pedestrians, keypoints, etc.>
    - has annotations: (yes - full, yes - partial, no)
        - which: <list of relevant types of annotations it has. Example: labels, bbox, masks, etc.>
    - is downloadable: (yes - full, yes - partial, no)
        - note: <any particular reason needed to be clarified here>

    Data structure (hdf5)
    ^^^^^^^^^^^^^^^^^^^^^

    <hdf5 data structure for each task>


    Source
    ^^^^^^

    <info of the websiter where the dataset was retrieved from.
    It should containg a link to the original website/source.><commit_msg>doc: Update template<commit_after>Template of a dataset README.rst file
=====================================

When creating a dataset, its good idea to provide a small documentation of the structure and information of the data.

When creating such documentation, it is good practice to follow a common format to keep things similar.

The following scheme details a template format on how to write the documentation of a new dataset and what information to provide to the end user.

.. code-block:: bash

    Dataset Name
    ============

    <small intro covering the dataset.>

    <bonus: some sample images of the dataset are always good to have.
    Visual aid is often good enough to help dismiss a dataset before delving into the details of the data.>

    Properties
    ----------

    - name: <name alias of the dataset>
    - tasks:
        - classification:
    - keywords: <keywords assign to the dataset. Example: object detection, pedestrians, keypoints, etc.>
    - dataset size: <total size of the dataset on disk>
    - metadata size: <total size of the processed hdf5 metdata file on disk for all tasks>
    - has annotations: (yes, yes - partial, no)
        - which: <list of relevant types of annotations it has. Example: labels, bbox, masks, etc.>
    - is downloadable: (yes, yes - partial, no)
        - note: <any particular reason needed to be clarified here>

    Data structure (hdf5)
    ---------------------

    <hdf5 data structure for each task>


    Disclaimer
    ----------

    <info of the website where the dataset was retrieved from.
    It should containg a link to the original website/source.>
{'fraction_non_alphanumeric': 0.10649984742142203, 'fraction_numerical': 0.00183094293561184, 'mean_word_length': 2.9878345498783454, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7051985', 'n_tokens_mistral': 845, 'n_tokens_neox': 791, 'n_words': 439}