<commit_before>============================================================================
ABOUT
============================================================================

This is command-line software for performing (relatively) rapid pairwise 
analysis of *mixed-type* covariate data to identify *associations* between 
covariates.

The prototypical kind of association is (Pearson) correlation, but Pearson 
correlation is, in fact, only one of a wide variety of statistical tests
that can be used to establish that two variables ("features") are related.
This program implements a selection of tests and automatically chooses
the appropriate test for a given pair of covariates. It also carries out
a number of other processes on data to increase the efficacy of the
statistical tests.

This software has three high-level functions:

	1. (Optionally) select pairs of features (rows from the input matrix)
	   specified in one of several ways by the command line.
	   By default it iterates through *all* pairs.
	2. For every pair of features it chooses and executes a statistical test 
	   according to the features' statistical classes (boolean, categorical, 
	   ordinal, continuous).
	3. Report a configurable amount of information on each test in either
	   tabular or (soon) JSON format subject to configurable filters.
	   Output filters include the option of Benjamini-Hochberg FDR conrol.

Every run of pairwise involves these function, but the exact behavior
is under the control of *many* options, some described herein some 
described only in the online help.

Input consists of textual matrices of tab-separated values in which
rows represent features and columns represent samples. See the libmtm
project in a neighboring directory tree for thorough documentation of 
input options and constraints as well as rationales behind the "statistical
classes" mentioned above.

**In the interest of minimizing stale and out-of-sync documentation functionality
that is thoroughly documented in the command-line executable's help text is not
repeated here.**

^^^^

============================================================================
STATISTICS
============================================================================

Three separate statistics are computed for each pair of features.
One of the following is calculated *between* the features using only 
those samples present in both features.

======================= ================================
Features                Statistic
======================= ================================
categorical/categorical Chisquare or Fisher Exact
categorical/numerical   Kruskal-Wallis
numerical/numerical     Spearman correlation
======================= ================================

Kruskal-Wallis is a non-parametric analog of 1-way ANOVA, 
which is approximately Chi-square distributed. The Spearman
correlation is the same as Pearson correlation but is calculated
on the ranks, not actual values, of the data.

Additionally, *within* each feature (F) the Kruskal-Wallis statistic is calculated 
between the samples of F present in both and samples present in F but missing in "the other."
For example, in the figure below where 'X' indicates missing data, 
the Kruskal-Wallis statistic is calculated in Feature 1 
between the light green and dark green samples and in Feature 2 between the light
blue and dark blue samples. The primary statistic (whichever of the above is used)
is calculated between the dark blue and dark green.

.. image:: ./doc/subsets.png

The K-W statistic *within* each feature provides some indication of whether 
the statistical result *between* the features was skewed by missing data.

The Spearman correlation is calculated for all pairs involving ordered
data. This obviously includes continuous pairs, but it also includes
categorical data with only two categories (i.e. boolean) since two categories
are always orderable (implicitly or otherwise). In other words, the Spearman
correlation is computed as long as one of the features is *not* strictly 
categorical with more than 2 categories. The absolute value of the Spearman 
correlation in some cases may not be meaningful, but its sign is.

The case where both features are categorical is slightly more
complex. In the interest of having statistically "good" tables
a preprocessing step is performed in which rows and/or columns
with empty cells are removed in such a way as to minimize the
total loss of samples. The approach, however, is greedy and not
guaranteed to be optimal. If after this "culling" step either of
the table's dimensions exceeds 2, a Chi-Square test is performed. 
Otherwise, if the table is 2x2, the Fisher exact test is performed.

**Contingency table cleaning going to be revised/supplemented by univariate outlier
detection and/or class balancing in the near future.**

^^^^

============================================================================
OUTPUT (FILTERING AND FORMATTING)
============================================================================

The *content* and *format* of output are *independently* configured.
Ouput is provided in one of two overarching formats: tabular and JSON_.
The "local" formatting of certain fields is also specifiable as noted
below.

.. _JSON : http://json.org

----------------------------
Content filtering
----------------------------

Every pair analysis produces 17 items of data, any or all of which
can be reported. You can select items for reporting as well as
specify the order by passing a "format specifier string" as the
argument of the -f (or -J) option on the command line. See online
help.

A format specifier is essentially a whitespace separated series of descriptive
identifiers or identifier *prefixes* as described in the tables below. 
*Any type* and *any amount* of whitespace (space, newline, tab) can separate 
each token, and tokens must not contain any whitespace.
If prefixes are used, the prefix need only be enough to uniquely identify
the intended identifier. Regular expression matches enforce this.

Some data are associated with the "left" and "right" features (e.g. 
feature names); others are associated with the covariate pair. Data
with left/right association are also independently specified by 
prefixing the identifier with '<' and/or '>'.

=========================== ==================================
Covariate information       Identifier regular expression
=========================== ==================================
sample count                c(ount)? [1]_
statistic name              st(at)?
statistic value             v(alue)?("[2]_")?
test error                  e(rror)? [3]_
sign of correlation         si(gn)?("[2]_")? [4]_ 
p-value                     p(rob)?("[2]_")?
-log10(p-value)             P(rob)?("[2]_")?
extra info                  e?x(tra)? [5]_
=========================== ==================================


=========================== ================================
Univariate information      Identifier regular expression 
=========================== ================================
feature name                [<>]{1,2}f(eature)?
feature offset              [<>]{1,2}o(ffset)?
feature statistical class   [<>]{1,2}cl(ass)?
preprocessing description   [<>]{1,2}pre(proc)?
count of unused values      [<>]{1,2}u(nused)?
statistic name              [<>]{1,2}s(tat)?
statistic value             [<>]{1,2}v(alue)? [2]_
p-value                     [<>]{1,2}pro(b)? [2]_
extra info                  [<>]{1,2}e?x(tra)?
=========================== ================================

.. [1] The number of covariate *pairs* analyzed. This is the number
       of pairs in which *neither* value was missing.

.. [2] A printf-style format string valid for a floating-point value 
       delimited by double quotes. This must match the regular
       expression: /%[-#0+]?([1-9][0-9]*)?(.[0-9]*)?[eEfFgG]"/.
       This is a *restricted* version of the printf format
       for floating-point values.

.. [3] A 2-character hexadecimal value reporting the bit 
       flags of the test. See online help.

.. [4] Currently the Spearman rho. In the future it will be
       +1.0 or -1.0 according to the sign of the correlation.
       Correlation value is also reported as the statistic value
       for continuous-continuous feature pairs.

.. [5] A loosely structured string consisting of any additional
       information made available by the statistical test (e.g. number of
       ties in rank data, or number of empty cells in a contingency table).

For example, "<>f p%.3f" specifies printing of *both* feature names
as well as the covariate p-value with 6 decimal places. Given a 
single 'p' as the argument to the --format option causes pairwise
to emit nothing but the covariate p-value.

**Some fields are not yet implemented, and will report as much if used.**

----------------------------
False discovery rate control
----------------------------

Coming soon...<commit_msg>README corrections<commit_after>============================================================================
ABOUT
============================================================================

This is command-line software for performing (relatively) rapid pairwise 
analysis of *mixed-type* covariate data to identify *associations* between 
covariates.

The prototypical kind of association is (Pearson) correlation, but Pearson 
correlation is, in fact, only one of a wide variety of statistical tests
that can be used to establish that two variables ("features") are related.
This program implements a selection of tests and automatically chooses
the appropriate test for a given pair of covariates. It also carries out
a number of other processes on data to increase the efficacy of the
statistical tests.

This software has three high-level functions:

	1. (Optionally) select pairs of features (rows from the input matrix)
	   specified in one of several ways by the command line.
	   By default it iterates through *all* pairs.
	2. For every pair of features it chooses and executes a statistical test 
	   according to the features' statistical classes (boolean, categorical, 
	   ordinal, continuous).
	3. Report a configurable amount of information on each test in either
	   tabular or (soon) JSON format subject to configurable filters.
	   Output filters include the option of Benjamini-Hochberg FDR conrol.

Every run of pairwise involves these function, but the exact behavior
is under the control of *many* options, some described herein some 
described only in the online help.

Input consists of textual matrices of tab-separated values in which
rows represent features and columns represent samples. See the libmtm
project in a neighboring directory tree for thorough documentation of 
input options and constraints as well as rationales behind the "statistical
classes" mentioned above.

**In the interest of minimizing stale and out-of-sync documentation functionality
that is thoroughly documented in the command-line executable's help text is not
repeated here.**

^^^^

============================================================================
STATISTICS
============================================================================

Three separate statistics are computed for each pair of features.
One of the following is calculated *between* the features using only 
those samples present in both features.

======================= ================================
Features                Statistic
======================= ================================
categorical/categorical Chisquare or Fisher Exact
categorical/numerical   Kruskal-Wallis
numerical/numerical     Spearman correlation
======================= ================================

Kruskal-Wallis is a non-parametric analog of 1-way ANOVA, 
which is approximately Chi-square distributed. The Spearman
correlation is the same as Pearson correlation but is calculated
on the ranks, not actual values, of the data.

Additionally, *within* each feature (F) the Kruskal-Wallis statistic is calculated 
between the samples of F present in both and samples present in F but missing in "the other."
For example, in the figure below where 'X' indicates missing data, 
the Kruskal-Wallis statistic is calculated in Feature 1 
between the light green and dark green samples and in Feature 2 between the light
blue and dark blue samples. The primary statistic (whichever of the above is used)
is calculated between the dark blue and dark green.

.. image:: ./doc/subsets.png

The K-W statistic *within* each feature provides some indication of whether 
the statistical result *between* the features was skewed by missing data.

The Spearman correlation is calculated for all pairs involving ordered
data. This obviously includes continuous pairs, but it also includes
categorical data with only two categories (i.e. boolean) since two categories
are always orderable (implicitly or otherwise). In other words, the Spearman
correlation is computed as long as one of the features is *not* strictly 
categorical with more than 2 categories. The absolute value of the Spearman 
correlation in some cases may not be meaningful, but its sign is.

The case where both features are categorical is slightly more
complex. In the interest of having statistically "good" tables
a preprocessing step is performed in which rows and/or columns
with empty cells are removed in such a way as to minimize the
total loss of samples. The approach, however, is greedy and not
guaranteed to be optimal. If after this "culling" step either of
the table's dimensions exceeds 2, a Chi-Square test is performed. 
Otherwise, if the table is 2x2, the Fisher exact test is performed.

**Contingency table cleaning going to be revised/supplemented by univariate outlier
detection and/or class balancing in the near future.**

^^^^

============================================================================
OUTPUT (FILTERING AND FORMATTING)
============================================================================

The *content* and *format* of output are *independently* configured.
Ouput is provided in one of two overarching formats: tabular and JSON_.
The "local" formatting of certain fields is also specifiable as noted
below.

.. _JSON : http://json.org

----------------------------
Content filtering
----------------------------

Every pair analysis produces 17 items of data, any or all of which
can be reported. You can select items for reporting as well as
specify the order by passing a "format specifier string" as the
argument of the -f (or -J) option on the command line. See online
help.

A format specifier is essentially a whitespace separated series of descriptive
identifiers or identifier *prefixes* as described in the tables below. 
*Any type* and *any amount* of whitespace (space, newline, tab) can separate 
each token, and tokens must not contain any whitespace.
If prefixes are used, the prefix need only be enough to uniquely identify
the intended identifier. Regular expression matches enforce this.

Some data are associated with the "left" and "right" features (e.g. 
feature names); others are associated with the covariate pair. Data
with left/right association are also independently specified by 
prefixing the identifier with '<' and/or '>'.

=========================== ==================================
Covariate information       Identifier regular expression
=========================== ==================================
sample count                c(ount)? [1]_
statistic name              st(at)?
statistic value             v(alue)?([2]_)?
test error                  e(rror)? [3]_
sign of correlation         si(gn)?([2]_)? [4]_ 
p-value                     p(rob)?([2]_)?
-log10(p-value)             P(rob)?([2]_)?
extra info                  e?x(tra)? [5]_
=========================== ==================================


=========================== ================================
Univariate information      Identifier regular expression 
=========================== ================================
feature name                [<>]{1,2}f(eature)?
feature offset              [<>]{1,2}o(ffset)?
feature statistical class   [<>]{1,2}cl(ass)?
preprocessing description   [<>]{1,2}pre(proc)?
count of unused values      [<>]{1,2}u(nused)?
statistic name              [<>]{1,2}s(tat)?
statistic value             [<>]{1,2}v(alue)?([2]_)?
p-value                     [<>]{1,2}pro(b)?([2]_)?
extra info                  [<>]{1,2}e?x(tra)?
=========================== ================================

.. [1] The number of covariate *pairs* analyzed. This is the number
       of pairs in which *neither* value was missing.

.. [2] A printf-style format string valid for a floating-point value 
       delimited by double quotes. The \[2\] in the *interior* of the 
       parenthesis above must be replace by a string matching the
       regular expression: /%[-#0+]?([1-9][0-9]*)?(.[0-9]*)?[eEfFgG]/.
       This is a *restricted* version of the printf format
       for floating-point values.

.. [3] A 2-character hexadecimal value reporting the bit 
       flags of the test. See online help.

.. [4] Currently the Spearman rho. In the future it will be
       +1.0 or -1.0 according to the sign of the correlation.
       Correlation value is also reported as the statistic value
       for continuous-continuous feature pairs.

.. [5] A loosely structured string consisting of any additional
       information made available by the statistical test (e.g. number of
       ties in rank data, or number of empty cells in a contingency table).

For example, "<>f p%.5f" specifies printing of *both* feature names
as well as the covariate p-value with 5 decimal places. A 
single 'p' as the argument to the --format option causes pairwise
to emit nothing but the covariate p-value in a default floating-point
format (exponential).

**Some fields are not yet implemented, and will report as much if used.**

----------------------------
False discovery rate control
----------------------------

Coming soon...
{'fraction_non_alphanumeric': 0.1762270192679291, 'fraction_numerical': 0.006790702810136366, 'mean_word_length': 4.0080176942217305, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3470920', 'n_tokens_mistral': 4456, 'n_tokens_neox': 4073, 'n_words': 2198}