<commit_before>schemas.yml
###########
The `schemas.yml` file is used to describe the input/output files used in CEA.

e.g.
::

    get_database_construction_standards:
      created_by: [data-initializer]
      file_path: inputs/technology/archetypes/CONSTRUCTION_STANDARDS.xlsx
      file_type: xlsx
      schema:
        STANDARD_DEFINITION:
          columns:
            STANDARD:
              type: string
              primary: true
            Description:
              type: string
            YEAR_START:
              type: int
              min: 0
            YEAR_END:
              type: int
              min: 0
          constraints:
            YEAR:
              YEAR_START < YEAR_END
        used_by: []

Each file is represented by a key that is the name of the InputLocator method that points to it.

It provides other information of the file based on these set of properties:
    - created_by
        - lists which script creates this file. Useful for script suggestions for missing files.
    - file_path
        - path where the file can be found relative to the path of the scenario
    - file_type
        - type of file which helps decide what kind of parser to use to read the data found in the file
    - schema
        - describes the data structure of the file and how it interacts with data from other files
    - used_by
        - lists which script uses this file.


The *schema* of the file would be separated into two sections: **columns** and **constraints**
Below we will describe the various properties used by *schema*.

Columns Property
======================
The schema stores information regarding the values found in the columns of the given data and is described by giving the expected type of value.

This information would be used by the `InputValidator` class to validate each value found in the column.

Currently the schema supports types:
    - String
    - Numeric (Float and Integer)
    - Choice
    - Boolean

Each of these types will inherit from a set of base properties.

Base Properties
---------------
<commit_msg>Add file_type info<commit_after>schemas.yml
###########
The `schemas.yml` file is used to describe the input/output files used in CEA.

e.g.
::

    get_database_construction_standards:
      created_by: [data-initializer]
      file_path: inputs/technology/archetypes/CONSTRUCTION_STANDARDS.xlsx
      file_type: xlsx
      schema:
        STANDARD_DEFINITION:
          columns:
            STANDARD:
              type: string
              primary: true
            Description:
              type: string
            YEAR_START:
              type: int
              min: 0
            YEAR_END:
              type: int
              min: 0
          constraints:
            YEAR:
              YEAR_START < YEAR_END
        used_by: []

Each file is represented by a key that is the name of the InputLocator method that points to it.

It provides other information of the file based on these set of properties:
    - created_by
        - lists which script creates this file. Useful for script suggestions for missing files.
    - file_path
        - path where the file can be found relative to the path of the scenario
    - file_type
        - type of file which helps decide what kind of parser to use to read the data found in the file
    - schema
        - describes the data structure of the file and how it interacts with data from other files
    - used_by
        - lists which script uses this file.

File type
+++++++++
*file_type* affects how CEA parses the information found in the *schema* property.

This is because some file types would contain more than one table in them
e.g. excel files usually contain multiple tables depending on the number of sheets that it has
while .csv/.dbf files usually only contain one table.


To workaround this, the level of keys found in schema would be different based on its type,
where the table name for multiple table type files would encapsulate each *schema* properties
as found in single table type files.

e.g. single table type files
::


    $locator_method_name:
      schema:
        columns:
            $column_name:
                $column_properties
        constraints:
            $constraints


e.g. multiple table type files
::


    $locator_method_name:
      schema:
        $table_name:
            columns:
                $column_name:
                    $column_properties
            constraints:
                $constraints


Schema
++++++
The *schema* of the file would be separated into two sections: **columns** and **constraints**
Below we will describe the various properties used by *schema*.

Columns Property
======================
The schema stores information regarding the values found in the columns of the given data and is described by giving the expected type of value.

This information would be used by the `InputValidator` class to validate each value found in the column.

Currently the schema supports types:
    - String
    - Numeric (Float and Integer)
    - Choice
    - Boolean

Each of these types will inherit from a set of base properties.

Base Properties
---------------

{'fraction_non_alphanumeric': 0.06288699690402477, 'fraction_numerical': 0.0007739938080495357, 'mean_word_length': 2.2632575757575757, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7516726', 'n_tokens_mistral': 1304, 'n_tokens_neox': 1257, 'n_words': 614}