<commit_before>==============================
Schema Configuration Reference
==============================

Every project in PySemantic can be configured via a data dictionary or a
schema, which is a yaml file. This file houses the details of how PySemantic
should treat a project's constituent datasets. A typical data dictionary
follows the following pattern:

.. code-block:: yaml

  dataset_name:
    dataset_param_1: value1
    dataset_param_2: value2
    # etc

PySemantic reads this as a dictionary where the parameter names are keys and
their values are the values in the dictionary. Thus, the schema for a whole
project is a dictionary of dictionaries.

--------------------------
Basic Schema Configuration
--------------------------

Here is a list of different dataset parameters that PySemantic is sensitive
to:

* ``path`` (Required) The absolute path to the file containing the data. Note that the path must be absolute. This can also be a list of files if the dataset spans multiple files. If that is the case, the path parameter can be specified as:

.. code-block:: yaml

  path:
    - absolulte/path/to/file/1
    - absolulte/path/to/file/2
    # etc

* ``demlimiter`` (Optional, default: ``,``) The delimiter used in the file. This has to be a character delimiter, not words like "comma" or "tab".

* ``md5`` (Optional) The MD5 checksum of the file to read. This necessary
  because sometimes we read files and after processing it, rewrite to the same
  path. This parameter helps keep track of whether the file is correct.

* ``header``: (Optional) The header row of the file.

* ``column_names``: (Optional) Specify the names of columns to use in the
  loaded dataframe. This option can have multiple types of values. It can be:

    1. A list of strings to use as column names:

    .. code-block:: yaml

      column_names:
        - column_1
        - column_2
        - column_3

    2. A dictionary that maps original column names to new ones:

    .. code-block:: yaml

      column_names:
        org_colname_1: new_colname_a
        org_colname_2: new_colname_b
        org_colname_3: new_colname_c

    3. A Python function that translates the name of every column in the loaded
       dataframe:

    .. code-block:: yaml

      column_names: !!python/name:module_name.translate_column_name

* ``nrows``: (Optional) Number of rows to read from the file. If not specified, all rows from the file are read.

* ``use_columns``: (Optional) The list of the columns to read from the dataset. The format for specifying this parameter is as follows:

.. code-block:: yaml

    use_columns:
      - column_1
      - column_2
      - column_3

If this parameter is not specified, all columns present in the dataset are read.

* ``exclude_columns``: This option can be used to specify columns that are
  explicityly to be ignored. This is useful when there are large number of
  columns in the dataset and we only wish to exclude a few. Note that this
  option overrides the ``use_columns`` option, i.e. if a column name is present
  in both lists, it will be dropped.

* ``converters``: A dictionary of functions to be applied to columns when loading data. Any Python callable can be added to this list. This parameter makes up the ``converters`` argument of Pandas parsers. The usage is as follows:

.. code-block:: yaml

    converters:
      col_a: !!python/name:numpy.int

This results in the ``numpy.int`` function being called on the column ``col_a``

* ``dtypes`` (Optional) Data types of the columns to be read. Since types in Python are native objects, PySemantic expects them to be so in the schema. This can be formatted as follows:

.. code-block:: yaml

  dtypes:
    column_name: !!python/name:python_object

For example, if you have three columns named ``foo``, ``bar``, and ``baz``,
which have the types ``string``, ``integer`` and ``float`` respectively, then your schema
should look like:

.. code-block:: yaml

  dtypes:
    foo: !!python/name:__builtin__.str
    bar: !!python/name:__builtin__.int
    baz: !!python/name:__builtin__.float

Non-builtin types can be specified too:

.. code-block:: yaml

   dtypes:
     datetime_column: !!python/name:datetime.date

*Note*: You can figure out the yaml representation of a Python type by doing
the following:

.. code-block:: python

  import yaml
  x = type(foo) # where foo is the object who's type is to be yamlized
  print yaml.dump(x)

* ``combine_dt_columns`` (Optional) Columns containing Date/Time values can be combined into one column by using the following schema:

.. code-block:: yaml

  combine_dt_columns:
    output_col_name:
      - col_a
      - col_b

This will parse columns ``col_a`` and ``col_b`` as datetime columns, and put the result in a column named ``output_col_name``. Specifying the output name is optional. You may declare the schema as:

.. code-block:: yaml

  combine_dt_columns:
    - col_a
    - col_b

In this case the parser will simply name the output column as ``col_a_col_b``, as is the default with Pandas.

*NOTE*: Specifying this column will make PySemantic ignore any columns that have been declared as having the datetime type in the ``dtypes`` parameter.

* ``pickle`` (Optional) Absolute path to file which contains pickled arguments for the
  parser. This option can be used if readability or declaratives are not a concern. The file should contain a picked dictionary that is directly passed
  to the parser, i.e. if the loaded pickled data is in a dict named ``data``,
  then parser invocation becomes ``parser(**data)``.

*NOTE*: If any of the above options are present, they will override the
  corresponding arguments contained in the pickle file. In PySemantic,
  declarative statements have the right of way.

----------------------------
Column Schema Configuration
----------------------------

PySemantic also allows specifying rules and validators independently for each<commit_msg>More doc formatting<commit_after>==============================
Schema Configuration Reference
==============================

Every project in PySemantic can be configured via a data dictionary or a
schema, which is a yaml file. This file houses the details of how PySemantic
should treat a project's constituent datasets. A typical data dictionary
follows the following pattern:

.. code-block:: yaml

  dataset_name:
    dataset_param_1: value1
    dataset_param_2: value2
    # etc

PySemantic reads this as a dictionary where the parameter names are keys and
their values are the values in the dictionary. Thus, the schema for a whole
project is a dictionary of dictionaries.

--------------------------
Basic Schema Configuration
--------------------------

Here is a list of different dataset parameters that PySemantic is sensitive
to:

* ``path`` (Required) The absolute path to the file containing the data. Note that the path must be absolute. This can also be a list of files if the dataset spans multiple files. If that is the case, the path parameter can be specified as:

.. code-block:: yaml

  path:
    - absolulte/path/to/file/1
    - absolulte/path/to/file/2
    # etc

* ``demlimiter`` (Optional, default: ``,``) The delimiter used in the file. This has to be a character delimiter, not words like "comma" or "tab".

* ``md5`` (Optional) The MD5 checksum of the file to read. This necessary
  because sometimes we read files and after processing it, rewrite to the same
  path. This parameter helps keep track of whether the file is correct.

* ``header``: (Optional) The header row of the file.

* ``column_names``: (Optional) Specify the names of columns to use in the
  loaded dataframe. This option can have multiple types of values. It can be:

    1. A list of strings to use as column names:

    .. code-block:: yaml

      column_names:
        - column_1
        - column_2
        - column_3

    2. A dictionary that maps original column names to new ones:

    .. code-block:: yaml

      column_names:
        org_colname_1: new_colname_a
        org_colname_2: new_colname_b
        org_colname_3: new_colname_c

    3. A Python function that translates the name of every column in the loaded
       dataframe:

    .. code-block:: yaml

      column_names: !!python/name:module_name.translate_column_name

* ``nrows``: (Optional) Number of rows to read from the file. If not specified, all rows from the file are read.

* ``use_columns``: (Optional) The list of the columns to read from the dataset. The format for specifying this parameter is as follows:

.. code-block:: yaml

    use_columns:
      - column_1
      - column_2
      - column_3

If this parameter is not specified, all columns present in the dataset are read.

* ``exclude_columns``: This option can be used to specify columns that are
  explicityly to be ignored. This is useful when there are large number of
  columns in the dataset and we only wish to exclude a few. Note that this
  option overrides the ``use_columns`` option, i.e. if a column name is present
  in both lists, it will be dropped.

* ``converters``: A dictionary of functions to be applied to columns when loading data. Any Python callable can be added to this list. This parameter makes up the ``converters`` argument of Pandas parsers. The usage is as follows:

.. code-block:: yaml

    converters:
      col_a: !!python/name:numpy.int

This results in the ``numpy.int`` function being called on the column ``col_a``

* ``dtypes`` (Optional) Data types of the columns to be read. Since types in Python are native objects, PySemantic expects them to be so in the schema. This can be formatted as follows:

.. code-block:: yaml

  dtypes:
    column_name: !!python/name:python_object

For example, if you have three columns named ``foo``, ``bar``, and ``baz``,
which have the types ``string``, ``integer`` and ``float`` respectively, then your schema
should look like:

.. code-block:: yaml

  dtypes:
    foo: !!python/name:__builtin__.str
    bar: !!python/name:__builtin__.int
    baz: !!python/name:__builtin__.float

Non-builtin types can be specified too:

.. code-block:: yaml

   dtypes:
     datetime_column: !!python/name:datetime.date

*Note*: You can figure out the yaml representation of a Python type by doing
the following:

.. code-block:: python

  import yaml
  x = type(foo) # where foo is the object who's type is to be yamlized
  print yaml.dump(x)

* ``combine_dt_columns`` (Optional) Columns containing Date/Time values can be combined into one column by using the following schema:

.. code-block:: yaml

  combine_dt_columns:
    output_col_name:
      - col_a
      - col_b

This will parse columns ``col_a`` and ``col_b`` as datetime columns, and put the result in a column named ``output_col_name``. Specifying the output name is optional. You may declare the schema as:

.. code-block:: yaml

  combine_dt_columns:
    - col_a
    - col_b

In this case the parser will simply name the output column as ``col_a_col_b``, as is the default with Pandas.

*NOTE*: Specifying this column will make PySemantic ignore any columns that have been declared as having the datetime type in the ``dtypes`` parameter.

* ``pickle`` (Optional) Absolute path to file which contains pickled arguments for the
  parser. This option can be used if readability or declaratives are not a concern. The file should contain a picked dictionary that is directly passed
  to the parser, i.e. if the loaded pickled data is in a dict named ``data``,
  then parser invocation becomes ``parser(**data)``.

*NOTE*: If any of the above options are present, they will override the corresponding arguments contained in the pickle file. In PySemantic, declarative statements have the right of way.

----------------------------
Column Schema Configuration
----------------------------

PySemantic also allows specifying rules and validators independently for each
{'fraction_non_alphanumeric': 0.09875294910684193, 'fraction_numerical': 0.003370407819346141, 'mean_word_length': 3.358795446199045, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4076530', 'n_tokens_mistral': 3266, 'n_tokens_neox': 3155, 'n_words': 1581}