<commit_before>#One-Hot-Encoder

Encode a collection of categorical features using a 1-of-K encoding scheme.
Input columns to the one-hot-encoder must by of type int, string, dict, or
list.  The transformed output is a column of type dictionary (max_categories
per column dimension sparse vector) where the key corresponds to the index of 
the categorical variable and the value is 1.

The behaviour of the one-hot-encoder for each input data column type is as 
follows. (see transform() for examples of the same).

 - string : The key in the output dictionary is the string category and
   the value is 1.

 - int : Behave similar to string columns.

 - list : Each value in the list is treated like an individual string.
   Hence, a list of categorical variables can be used to represent a feature 
   where all categories in the list are simultaneously hot.

 - dict : They key of the dictionary is treated as a namespace and the
  value is treated as a sub-category in the namespace. The categorical variable 
  being encoded in this case is a combination of the namespace and the sub-category.


 You can specify the number of categories with the parameter max_categories. <commit_msg>edited for clarity<commit_after>#One-Hot-Encoder

Encode a collection of categorical features using a 1-of-K encoding scheme.
Input columns to the one-hot-encoder must by of type int, string, dict, or
list.  The transformed output is a column of type dictionary (max_categories
per column dimension sparse vector) where the key corresponds to the index of 
the categorical variable and the value is 1.

The behaviour of the OneHotEncoder for each input data column type is as 
follows (see transform() for examples of the same):

 - **string** : The key in the output dictionary is the string category and
   the value is 1.

 - **int** : Behave similar to string columns.

 - **list** : Each value in the list is treated like an individual string.
   Hence, a list of categorical variables can be used to represent a feature 
   where all categories in the list are simultaneously hot.

 - **dict** : They key of the dictionary is treated as a namespace and the
  value is treated as a sub-category in the namespace. The categorical variable 
  being encoded in this case is a combination of the namespace and the sub-category.


 You can specify the number of categories with the parameter max_categories. 
{'fraction_non_alphanumeric': 0.042118432026688905, 'fraction_numerical': 0.0025020850708924102, 'mean_word_length': 3.8269617706237424, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2750021', 'n_tokens_mistral': 590, 'n_tokens_neox': 556, 'n_words': 370}