<commit_before>==================
Configuration file
==================

Understanding BEASTling is, mostly, a matter of understanding the configuration file format.  Config files have the following form:

::

	[section1]
	param1a = value1a
	param2a = value2a
	param3a = value3a
	param4a = value4a
	param5a = value5a
	[section2]
	param1b = value1b
	param2b = value2b
	param3b = value3b
	[section3]
	param1c = value1c
	param2c = value2c
	param3c = value3c
	param4c = value4c
	...

i.e. they are divided into *sections*, which are indicated by names enclosed in square brackets (in the above, the section names are ``section1``, ``section2``, etc.), and each section consists of some number of *parameters* and assigned *values*.  Each line of each section corresponds to assigning one value to one parameter, the the parameter name on the left of the equals sign and the value on the right.

BEASTling configuration files can range from very simple (the only section which is compulsory is one or more ``model`` sections) to relatively complicated - although in all cases they are vastly simpler than any BEAST XML file.  If you provide minimal configuration information, "sensible defaults" will be used for all settings. 

*It is your responsibility to know what the defaults are and to make sure that they truly are sensible for your application*.

The recognised config file sections are as follows:

admin section
-------------

The ``admin`` section may contain the following parameters:

* ``basename``: this is any user-friendly string which will be used in e.g. filenames.  If the basename is, say, "IE_cognates", then the BEAST XML file which BEASTling produces will be named IE_cognates.xml, and when the BEAST analysis is run, the trees will be logged in IE_trees.nex, etc.  If unspecified, it will default to "beastling".

* ``glottolog_release``: the number of a Glottolog release (>=2.7), from which to obtain the language classification.

* ``screenlog``: this must be set to "True" or "False" and controls whether or not BEAST should output basic MCMC data like ESS to the screen while running.  Default is True.

* ``log_probabilities``: "True" or "False".  Controls whether or not the prior, likelihood and posterior should be logged to a file called basename.log.  This is generally a good idea, so that you can check e.g. ESSes for these things in Tracer, so the default is True.

* ``log_params``: "True" or "False".  Controls whether or not all model parameters are also included in basename.log.  Default is False.

* ``log_trees``: "True" or "False".  Controls whether or not sampled trees should be logged to basename.nex.  Default is True.

* ``log_all``: "True" or "False".  Setting this true is simply a shorthand for setting ``log_probabilities`` and ``log_params`` and ``log_trees`` to all be true.  Default is False.

* ``log_every``: an integer specifying how many MCMC samples should elapse between consecutive entries in the log file.  If not specified, BEASTling will set this based on the chainlength such that the log file will be 10,000 entries long.  This is a good compromise between getting lots of information about the posterior and conserving disk space.

MCMC section
------------

The ``MCMC`` section may contain the following parameters:

* ``chainlength``: number of iterations to run the MCMC chain for.  Default is 10,000,000.

* ``sample_from_prior``: "True" or "False".  If True, BEAST will ignore all supplied data and the tree, all clock rates and any model parameters will all be sampled from their prior distributions.  Default is False.

languages section
-----------------

The ``languages`` section may contain the following parameters:

* ``exclusions``: One of:
   * A comma-separated list of language names or codes to exclude from the analysis, spelled exactly as they are in the data file(s).
   * The path to a file which contains one language per line.

This can be used by itself to remove a few problematic languages from a data file, or in conjunction with ``families`` or ``macroareas`` to better control which languages are included, e.g. you may set ``macroareas = Africa``, but use ``excludes`` to remove some outliers like Austronesian languages on Madagascar or Indo-European languages in South Africa.

* ``families``: One of:
   * A comma-separated list of language families to include in the analysis, spelled exactly as they are in Glottolog.  E.g. ``Indo-European, Uralic, Dravidian``.
   * The path to a file which contains one language family per line.

If no value is assigned to this parameter, all languages present in the data file will be included (unless ``languages`` (see below) is used.  ``families`` and ``languages`` cannot both be used in a single configuration.

* ``languages``: One of:
   * A comma-separated list of language names or codes to include in the analysis, spelled exactly as they are in the data file(s).
   * The path to a file which contains one language per line.  

If no value is assigned to this parameter, all languages present in the data file will be included (unless ``families`` (see above) is used.  ``languages`` and ``families`` cannot both be used in a single configuration.

* ``macroareas``: One of:
   * A comma-separated list of Glottolog macroareas to include in the analysis
   * The path to a file which contains one macroarea per line.  

Valid macroareas are: ``Africa``, ``Australia``, ``Eurasia``, ``North America``, ``Papunesia``, ``South America``.  This can be used in conjunction with ``languages`` or ``families``, in which case a language must meet both criteria to be included.  E.g. if you set ``families = Afro-Asiatic`` and ``macroareas = Africa``, you will get only the Afro-Asiatic languages located in Africa, and those located in Eurasia will be excluded.

* ``monophyly`` (or ``monophyletic``): "True" or "False".  Controls whether or not to impose the family structure in Glottolog as monophyly constraints in the BEAST analysis.  Default is False.  If True, very fine-grained control over exactly how much constraint is opposed can be gained by using additional options, documented below.

* ``monophyly_levels``: An integer specifying how many levels of the Glottolog classification to impost as a monophyly constraints.  By default, levels are added in a top-down fashion (but see ``monophyly_direction`` below).  E.g. if ``monophyly_levels = 3`` is specified, then Indo-European languages will be constrained to be monophyletic (one level), and so will Armenian, Celtic and Germanic, among others (two levels), and so will be Gothic and Northwest Germanic, among others (three levels), but North Germanic and West Germanic, or any descendant groups, will *not* be.  This allows one to enforce the high level structure of Glottolog, while leaving the "fine details" of relationships among leaves to be inferred from data.  If no value is specified, the entire Glottolog classification will be imposed.

* ``monophyly_direction``: One of ``top_down`` (the default) or ``bottom_up``.  Determines the effect of ``monophyly_levels``.  If ``monophyly_direction = top_down``, constraints will be added from the roots of Glottolog trees downward (e.g. Indo-European, Germanic, North Germanic,...).  If ``bottom_up``, constraints will be added from the leaves upward (e.g. Macro-Swedish, East Scandinavian, North Germanic,...).

* ``monophyly_start_depth``: An integer specifying an initial number of levels of the Glottolog classification to skip over when implying constraints (default 0).  E.g., with top down constraints, setting ``monophyly_start_depth=2`` will skip over Indo-European and Germanic, so that if ``monophyly_levles=3``, the imposed levels will be, e.g. Western Germanic, Franconian and High Franconian.  With bottom up constraints, this controls skipping initial levels above the leaves.

* ``monophyly_end_depth``: An integer specifying a level in the Glottolog classification below which constraints will not be imposed.  If ``monophyly_end_depth`` is specified, then ``monophyly_direction`` and ``monophyly_levels`` are ignored.  The imposed constraints will be those between ``monophyly_start_depth`` and ``monophyly_end_depth``, interpreted in a top down fashion.  This is a "low level" approach to controling monophyly, and in general the "configurational sugar" of using ``monophyly_direction``, ``monophyly_start`` and ``monophyly_levels`` should be preferred.

* ``overlap``: One of ``union`` or ``intersection``.  Controls how to deal with language sets mismatches between input data.
   * If set to ``union`` (the default), languages missing in one data set will be added with missing datapoints ("?") for all features.
   * If set to ``intersection``, only languages present in all data sets will be used.

* ``starting_tree``: Used to provide a starting tree.  Can be a Newick format tree or the name of a file which contains a Newick format tree.  If not specified, a random starting tree (compatible with monophyly constraints, if active) will be used.

* ``sample_branch_lengths``: If True, the branch lengths of the starting tree.  If False, the starting branch lengths will be kept fixed.  Use this in conjunction with ``starting_tree`` when you have a tree you trust and want to fit model parameters to it.  Default is True.

* ``sample_topology``: If true, the topology of the starting tree (i.e. the details of which leaves are connected to which and how) will be sampled during the analysis to fit the data.  If false, the topology will be kept fixed.  Use this in conjunction with ``starting_tree`` when you have a tree you trust and want to fit model parameters to it.  Default is True.


calibration section
-------------------

The ``calibration`` section should contain one parameter for each distinct calibration point that you wish to include in the analysis.

The name of each parameter should be a comma-separated list of family names or Glottocodes.  Optionally, the name can be enclosed in ``originate( )`` to place the calibration not on the MRCA of the languages/families specificed, but on the originate, i.e. the top of the branch leading to the MRCA.  The value for each calibration can be a string in one of several supported formats.  The two simplest formats are to specify a range of ages, or a single upper or lower bounding age.

Ranges can be specified as follows:

::

	Austronesian = 4750 - 5800

You may use arbitrary units without problems, i.e. you could provide dates in millenia BP:

::

	Austronesian = 4.75 - 5.8

The only time this matters is when it comes time to interpret tree heights or clock and/or mutation rates.  With this kind of calibration, BEASTling will set a normal distribution prior on the age of the family indicated.  The mean of the distribution will be equal to the midpoint of the provided range (5275 in the above case).  The standard deviation will be set such that 95\% of the probability mass will lie within the range provided.  In other words, the range you provide is treated as a 95\% credibility interval.

Bounds can be specified as follows:

::

	Austronesian = > 4750
       
or

::

        Austronesian = < 5800

With this kind of calibration, BEASTling will set a uniform distribution prior on the age of the family indicated.  The upper or lower bound will be set to the provided age, and the other bound will be set to zero or infinity as appropriate.

If you require more control over your priors, you can explicitly provide the type of distribution (either normal, lognormal or uniform) and the parameters as follows:

::

	Austronesian = normal(5275, 535.71)           # First param is mean, second is standard deviation
	Austronesian = lognormal(8.57, 0.05)          # First param is mean, second is standard deviation, both are in logspace
	Austronesian = uniform(4.75, 5.80)            # First param is lower bound, second is upper bound

Finally, it is possible to specify an age range and ask for a lognormal distribution to be fitted to it, as follows:

::

	Austronesian = lognormal(4750 - 5800)

With this kind of calibration, BEASTling will set a lognormal distribution prior on the age of the family indicated.  The mean of the distribution will be set so that the median of the lognormal distribution equals the midpoint of the range provided.  The standard deviation will be set to the mean of two values: one with the property that the provided lower bound is at the 5th percentile of the lognormal distribution, and one with the property that the provided upper bound is at the 95th percentile.  The provided interval does not quite end up being a 95% credible interval, but it is roughly so.  Explicitly set the lognormal parameters as shown above if you need more control over the matching than this.

model sections
--------------<commit_msg>Document new calibration specification options.<commit_after>==================
Configuration file
==================

Understanding BEASTling is, mostly, a matter of understanding the configuration file format.  Config files have the following form:

::

	[section1]
	param1a = value1a
	param2a = value2a
	param3a = value3a
	param4a = value4a
	param5a = value5a
	[section2]
	param1b = value1b
	param2b = value2b
	param3b = value3b
	[section3]
	param1c = value1c
	param2c = value2c
	param3c = value3c
	param4c = value4c
	...

i.e. they are divided into *sections*, which are indicated by names enclosed in square brackets (in the above, the section names are ``section1``, ``section2``, etc.), and each section consists of some number of *parameters* and assigned *values*.  Each line of each section corresponds to assigning one value to one parameter, the the parameter name on the left of the equals sign and the value on the right.

BEASTling configuration files can range from very simple (the only section which is compulsory is one or more ``model`` sections) to relatively complicated - although in all cases they are vastly simpler than any BEAST XML file.  If you provide minimal configuration information, "sensible defaults" will be used for all settings. 

*It is your responsibility to know what the defaults are and to make sure that they truly are sensible for your application*.

The recognised config file sections are as follows:

admin section
-------------

The ``admin`` section may contain the following parameters:

* ``basename``: this is any user-friendly string which will be used in e.g. filenames.  If the basename is, say, "IE_cognates", then the BEAST XML file which BEASTling produces will be named IE_cognates.xml, and when the BEAST analysis is run, the trees will be logged in IE_trees.nex, etc.  If unspecified, it will default to "beastling".

* ``glottolog_release``: the number of a Glottolog release (>=2.7), from which to obtain the language classification.

* ``screenlog``: this must be set to "True" or "False" and controls whether or not BEAST should output basic MCMC data like ESS to the screen while running.  Default is True.

* ``log_probabilities``: "True" or "False".  Controls whether or not the prior, likelihood and posterior should be logged to a file called basename.log.  This is generally a good idea, so that you can check e.g. ESSes for these things in Tracer, so the default is True.

* ``log_params``: "True" or "False".  Controls whether or not all model parameters are also included in basename.log.  Default is False.

* ``log_trees``: "True" or "False".  Controls whether or not sampled trees should be logged to basename.nex.  Default is True.

* ``log_all``: "True" or "False".  Setting this true is simply a shorthand for setting ``log_probabilities`` and ``log_params`` and ``log_trees`` to all be true.  Default is False.

* ``log_every``: an integer specifying how many MCMC samples should elapse between consecutive entries in the log file.  If not specified, BEASTling will set this based on the chainlength such that the log file will be 10,000 entries long.  This is a good compromise between getting lots of information about the posterior and conserving disk space.

MCMC section
------------

The ``MCMC`` section may contain the following parameters:

* ``chainlength``: number of iterations to run the MCMC chain for.  Default is 10,000,000.

* ``sample_from_prior``: "True" or "False".  If True, BEAST will ignore all supplied data and the tree, all clock rates and any model parameters will all be sampled from their prior distributions.  Default is False.

languages section
-----------------

The ``languages`` section may contain the following parameters:

* ``exclusions``: One of:
   * A comma-separated list of language names or codes to exclude from the analysis, spelled exactly as they are in the data file(s).
   * The path to a file which contains one language per line.

This can be used by itself to remove a few problematic languages from a data file, or in conjunction with ``families`` or ``macroareas`` to better control which languages are included, e.g. you may set ``macroareas = Africa``, but use ``excludes`` to remove some outliers like Austronesian languages on Madagascar or Indo-European languages in South Africa.

* ``families``: One of:
   * A comma-separated list of language families to include in the analysis, spelled exactly as they are in Glottolog.  E.g. ``Indo-European, Uralic, Dravidian``.
   * The path to a file which contains one language family per line.

If no value is assigned to this parameter, all languages present in the data file will be included (unless ``languages`` (see below) is used.  ``families`` and ``languages`` cannot both be used in a single configuration.

* ``languages``: One of:
   * A comma-separated list of language names or codes to include in the analysis, spelled exactly as they are in the data file(s).
   * The path to a file which contains one language per line.  

If no value is assigned to this parameter, all languages present in the data file will be included (unless ``families`` (see above) is used.  ``languages`` and ``families`` cannot both be used in a single configuration.

* ``macroareas``: One of:
   * A comma-separated list of Glottolog macroareas to include in the analysis
   * The path to a file which contains one macroarea per line.  

Valid macroareas are: ``Africa``, ``Australia``, ``Eurasia``, ``North America``, ``Papunesia``, ``South America``.  This can be used in conjunction with ``languages`` or ``families``, in which case a language must meet both criteria to be included.  E.g. if you set ``families = Afro-Asiatic`` and ``macroareas = Africa``, you will get only the Afro-Asiatic languages located in Africa, and those located in Eurasia will be excluded.

* ``monophyly`` (or ``monophyletic``): "True" or "False".  Controls whether or not to impose the family structure in Glottolog as monophyly constraints in the BEAST analysis.  Default is False.  If True, very fine-grained control over exactly how much constraint is opposed can be gained by using additional options, documented below.

* ``monophyly_levels``: An integer specifying how many levels of the Glottolog classification to impost as a monophyly constraints.  By default, levels are added in a top-down fashion (but see ``monophyly_direction`` below).  E.g. if ``monophyly_levels = 3`` is specified, then Indo-European languages will be constrained to be monophyletic (one level), and so will Armenian, Celtic and Germanic, among others (two levels), and so will be Gothic and Northwest Germanic, among others (three levels), but North Germanic and West Germanic, or any descendant groups, will *not* be.  This allows one to enforce the high level structure of Glottolog, while leaving the "fine details" of relationships among leaves to be inferred from data.  If no value is specified, the entire Glottolog classification will be imposed.

* ``monophyly_direction``: One of ``top_down`` (the default) or ``bottom_up``.  Determines the effect of ``monophyly_levels``.  If ``monophyly_direction = top_down``, constraints will be added from the roots of Glottolog trees downward (e.g. Indo-European, Germanic, North Germanic,...).  If ``bottom_up``, constraints will be added from the leaves upward (e.g. Macro-Swedish, East Scandinavian, North Germanic,...).

* ``monophyly_start_depth``: An integer specifying an initial number of levels of the Glottolog classification to skip over when implying constraints (default 0).  E.g., with top down constraints, setting ``monophyly_start_depth=2`` will skip over Indo-European and Germanic, so that if ``monophyly_levles=3``, the imposed levels will be, e.g. Western Germanic, Franconian and High Franconian.  With bottom up constraints, this controls skipping initial levels above the leaves.

* ``monophyly_end_depth``: An integer specifying a level in the Glottolog classification below which constraints will not be imposed.  If ``monophyly_end_depth`` is specified, then ``monophyly_direction`` and ``monophyly_levels`` are ignored.  The imposed constraints will be those between ``monophyly_start_depth`` and ``monophyly_end_depth``, interpreted in a top down fashion.  This is a "low level" approach to controling monophyly, and in general the "configurational sugar" of using ``monophyly_direction``, ``monophyly_start`` and ``monophyly_levels`` should be preferred.

* ``overlap``: One of ``union`` or ``intersection``.  Controls how to deal with language sets mismatches between input data.
   * If set to ``union`` (the default), languages missing in one data set will be added with missing datapoints ("?") for all features.
   * If set to ``intersection``, only languages present in all data sets will be used.

* ``starting_tree``: Used to provide a starting tree.  Can be a Newick format tree or the name of a file which contains a Newick format tree.  If not specified, a random starting tree (compatible with monophyly constraints, if active) will be used.

* ``sample_branch_lengths``: If True, the branch lengths of the starting tree.  If False, the starting branch lengths will be kept fixed.  Use this in conjunction with ``starting_tree`` when you have a tree you trust and want to fit model parameters to it.  Default is True.

* ``sample_topology``: If true, the topology of the starting tree (i.e. the details of which leaves are connected to which and how) will be sampled during the analysis to fit the data.  If false, the topology will be kept fixed.  Use this in conjunction with ``starting_tree`` when you have a tree you trust and want to fit model parameters to it.  Default is True.


calibration section
-------------------

The ``calibration`` section should contain one parameter for each distinct calibration point that you wish to include in the analysis.

The name of each parameter should be a comma-separated list of family names or Glottocodes.  Optionally, the name can be enclosed in ``originate( )`` to place the calibration not on the MRCA of the languages/families specificed, but on the originate, i.e. the top of the branch leading to the MRCA.  The value for each calibration can be a string in one of several supported formats.  The two simplest formats are to specify a range of ages, or a single upper or lower bounding age.

Ranges can be specified as follows:

::

	Austronesian = 4750 - 5800

You may use arbitrary units without problems, i.e. you could provide dates in millenia BP:

::

	Austronesian = 4.75 - 5.8

The only time this matters is when it comes time to interpret tree heights or clock and/or mutation rates.  With this kind of calibration, BEASTling will set a normal distribution prior on the age of the family indicated.  The mean of the distribution will be equal to the midpoint of the provided range (5275 in the above case).  The standard deviation will be set such that 95\% of the probability mass will lie within the range provided.  In other words, the range you provide is treated as a 95\% credibility interval.

Bounds can be specified as follows:

::

	Austronesian = > 4750
       
or

::

        Austronesian = < 5800

With this kind of calibration, BEASTling will set a uniform distribution prior on the age of the family indicated.  The upper or lower bound will be set to the provided age, and the other bound will be set to zero or infinity as appropriate.

If you require more control over your priors, you can explicitly provide the type of distribution (either normal, lognormal or uniform) and the parameters, as well as specify an offset, as follows:

::

	Austronesian = normal(5275, 535.71)           # First param is mean, second is standard deviation
	Austronesian = lognormal(8.57, 0.05)          # First param is mean (in log space), second is standard deviation
	Austronesian = rognormal(5275, 0.05)          # First param is mean (in real space), second is standard deviation
	Iranian = 2600 + rlognormal(400, 0.8)         # As per above but with an offset

Finally, it is possible to specify an age range and ask for a lognormal distribution to be fitted to it, as follows:

::

	Austronesian = lognormal(4750 - 5800)

With this kind of calibration, BEASTling will set a lognormal distribution prior on the age of the family indicated.  The mean of the distribution will be set so that the median of the lognormal distribution equals the midpoint of the range provided.  The standard deviation will be set to the mean of two values: one with the property that the provided lower bound is at the 5th percentile of the lognormal distribution, and one with the property that the provided upper bound is at the 95th percentile.  The provided interval does not quite end up being a 95% credible interval, but it is roughly so.  Explicitly set the lognormal parameters as shown above if you need more control over the matching than this.

model sections
--------------
{'fraction_non_alphanumeric': 0.07418317119491492, 'fraction_numerical': 0.008991899538777567, 'mean_word_length': 4.237921234267154, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '864705', 'n_tokens_mistral': 7027, 'n_tokens_neox': 6568, 'n_words': 3796}