<commit_before>Glossary - Terms for Data Transformation
========================================

This glossary is an attempt to construct a more-or-less consistent
terminology for transformations on tables of data, and other related
operations. It's also an attempt to identify a useful and convenient
set of *primitive* transformation operations, i.e., the fundamental
operations in terms of which all other transformations can be
expressed.

Where possible, terminology used in other projects/tools for
equivalent or similar operations is given in the "Implementations"
sub-sections.

Because so many of these terms have been used in a variety of
contexts, sometimes with very different senses, there is no hope of
creating a terminology that is both intuitive and acceptable to
everyone. However, I've tried in the "Discussion" sub-sections to give
my rationale for choosing each particular term. Feel free to argue
with me and present alternatives.

Cut - select and/or re-order fields
-----------------------------------

**Definition** 

The Cut transformation maps each row in the input to a single row in
the output, such that some fields may be omitted from rows in the
output, and fields may also be re-ordered.

**Example**

Input:

+-------+-------+-------+
| 'foo' | 'bar' | 'baz' |
+=======+=======+=======+
| 'A'   | 1     | True  |
+-------+-------+-------+
| 'B'   | 2     | False |
+-------+-------+-------+

Transformation: Cut, selecting fields ('baz', 'foo').

Output:

+-------+-------+
| 'baz' | 'foo' |
+=======+=======+
| True  | 1     |
+-------+-------+
| False | 2     |
+-------+-------+

**Discussion**

"Cut" is used in csvkit by analogy with the linux "cut" command. TODO explain rationale for supporting "Cut". 

**Implementations** 

 - csvcut in csvkit
 - TODO terminology used for this transformation in other tools.

Slice - select a subsequence of data rows
-----------------------------------------

**Definition**

TODO

**Example**

TODO

**Discussion**

TODO

**Implementations**

TODO

Cat - concatenate two or more tables
------------------------------------

**Definition**

Concatenate two or more input tables into a single output table. Input
tables may have different fields, in which case the fields in the
output table are the union of the fields in the inputs, and any
missing fields in input rows are padded with a configurable value.

**Example**

Input 1:

+-------+-------+
| 'foo' | 'bar' |
+=======+=======+
| 1     | 'A'   |
+-------+-------+
| 2     | 'B'   |
+-------+-------+

Input 2:

+-------+-------+
| 'bar' | 'baz' |
+=======+=======+
| 'C'   | True  |
+-------+-------+
| 'D'   | False |
+-------+-------+

Transformation: Cat inputs, padding missing fields with value None.

Output:

+-------+-------+-------+
| 'foo' | 'bar' | 'baz' |
+=======+=======+=======+
| 1     | 'A'   | None  |
+-------+-------+-------+
| 2     | 'B'   | None  |
+-------+-------+-------+
| None  | 'C'   | True  |
+-------+-------+-------+
| None  | 'D'   | False |
+-------+-------+-------+

**Discussion**

TODO Arguably not primitive...

**Implementations**

TODO

Map - arbitrarily transform rows
--------------------------------

**Definition**

TODO

**Example**

TODO

**Discussion**

TODO

**Implementations**

TODO

Melt
----

**Definition**

TODO

**Example**

TODO

**Discussion**

TODO

**Implementations**

TODO

Unmelt
------

TODO
<commit_msg>a bit more writing<commit_after>Glossary - Terms for Data Transformation
========================================

This glossary is an attempt to construct a more-or-less consistent
terminology for transformations on tables of data, and other related
operations. It's also an attempt to identify a useful and convenient
set of *primitive* transformation operations, i.e., the fundamental
operations in terms of which all other transformations can be
expressed.

Where possible, terminology used in other projects/tools for
equivalent or similar operations is given in the "Implementations"
sub-sections.

Because so many of these terms have been used in a variety of
contexts, sometimes with very different senses, there is no hope of
creating a terminology that is both intuitive and acceptable to
everyone. However, I've tried in the "Discussion" sub-sections to give
my rationale for choosing each particular term. Feel free to argue
with me and present alternatives.

Cut - select and/or re-order fields
-----------------------------------

**Definition** 

The Cut transformation maps each row in the input to a single row in
the output, such that some fields may be omitted from rows in the
output, and fields may also be re-ordered.

**Example**

Input:

+-------+-------+-------+
| 'foo' | 'bar' | 'baz' |
+=======+=======+=======+
| 'A'   | 1     | True  |
+-------+-------+-------+
| 'B'   | 2     | False |
+-------+-------+-------+

Transformation: Cut, selecting fields ('baz', 'foo').

Output:

+-------+-------+
| 'baz' | 'foo' |
+=======+=======+
| True  | 1     |
+-------+-------+
| False | 2     |
+-------+-------+

**Discussion**

"Cut" is used in csvkit by analogy with the linux "cut" command. TODO explain rationale for supporting "Cut". 

**Implementations** 

 - csvcut in csvkit
 - TODO terminology used for this transformation in other tools.

Slice - select a subsequence of data rows
-----------------------------------------

**Definition**

Select a continuous subsequence of data rows from an input table.

**Example**

Input:

+-------+-------+-------+
| 'foo' | 'bar' | 'baz' |
+=======+=======+=======+
| 'A'   | 1     | True  |
+-------+-------+-------+
| 'B'   | 2     | False |
+-------+-------+-------+
| 'C'   | 4     | False |
+-------+-------+-------+
| 'F'   | 7     | True  |
+-------+-------+-------+

Transformation: Slice input, selecting rows from 1 to 2.

Output:

+-------+-------+-------+
| 'foo' | 'bar' | 'baz' |
+=======+=======+=======+
| 'B'   | 2     | False |
+-------+-------+-------+
| 'C'   | 4     | False |
+-------+-------+-------+

**Discussion**

N.B., this operation is about slicing data rows, so 0 (zero) indexes
the first data row, and not the header row.

**Implementations**

TODO

Cat - concatenate two or more tables
------------------------------------

**Definition**

Concatenate two or more input tables into a single output table. Input
tables may have different fields, in which case the fields in the
output table are the union of the fields in the inputs, and any
missing fields in input rows are padded with a configurable value.

**Example**

Input 1:

+-------+-------+
| 'foo' | 'bar' |
+=======+=======+
| 1     | 'A'   |
+-------+-------+
| 2     | 'B'   |
+-------+-------+

Input 2:

+-------+-------+
| 'bar' | 'baz' |
+=======+=======+
| 'C'   | True  |
+-------+-------+
| 'D'   | False |
+-------+-------+

Transformation: Cat inputs, padding missing fields with value None.

Output:

+-------+-------+-------+
| 'foo' | 'bar' | 'baz' |
+=======+=======+=======+
| 1     | 'A'   | None  |
+-------+-------+-------+
| 2     | 'B'   | None  |
+-------+-------+-------+
| None  | 'C'   | True  |
+-------+-------+-------+
| None  | 'D'   | False |
+-------+-------+-------+

**Discussion**

TODO Arguably not primitive...

**Implementations**

TODO

Map - arbitrarily transform rows
--------------------------------

**Definition**

Map each row in an input table to a new row in the output table. The
mapping is arbitrary, i.e., any function could be defined to perform
the transformation.

**Example**

Input:

+-------+-------+
| 'foo' | 'bar' |
+=======+=======+
| 1     | 'A'   |
+-------+-------+
| 2     | 'B'   |
+-------+-------+

Transformation: Map the input table, such that the output field
'foo_times_2' equals the value of the input field 'foo' multiplied by
2, and the output field 'foo_bars' equals the value of the input field
'bar' repeated 'foo' times.

Output:

+---------------+------------+
| 'foo_times_2' | 'foo_bars' |
+===============+============+
| 2             | 'A'        |
+---------------+------------+
| 4             | 'BB'       |
+---------------+------------+

**Discussion**

N.B., here 'Map' is restricted to a proper mapping of rows, i.e., one
row in, one row out. Other types of transformation where one input row
could result in less than or more than one output row are not included
in the definition of 'Map'.

TODO

**Implementations**

TODO

Melt
----

**Definition**

TODO

**Example**

TODO

**Discussion**

TODO

**Implementations**

TODO

Unmelt
------

TODO

{'fraction_non_alphanumeric': 0.28151803548349197, 'fraction_numerical': 0.0042298202326401125, 'mean_word_length': 2.030259878960484, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2529508', 'n_tokens_mistral': 2644, 'n_tokens_neox': 2508, 'n_words': 933}