<commit_before>"""
Global definitions for CTPy
"""

### Internal constants - need no documentation
# Constants tagging different types of dimensions, for DB queries
MODETYPE_EVEN = str("EVEN")
MODETYPE_RANDOM = str('RANDOM')



### Research-level constants

MAXALLELES = 1000000000
"""
simuPOP does infinite-allele models by giving a really large number to the KAllelesMutator,
and ensuring that the combination of popsize * mutationrate * simlength doesn't exceed MAXALLELES
This value is thus very important for defining classifications since it defines an integer-valued
trait space.
"""


NUM_REPLICATES_FOR_RANDOM_DIMENSION_MODES = 10
"""
When creating random dimensions for classification, we want this many random choices
of mode boundaries for each choice of mode count (e.g., 2 modes per dimension, 3 modes...)
"""

DIMENSION_PARTITIONS = [2,3,4,8,16,32]
"""
Classifications with 32 modes per dimension are rare in archaeological practice
when classes are discrete/nominal categories, but we want to see scaling of
observables, and we can imagine cases with random dimensions where we might
have an ability to chop things into chunks but not understand how those map to underlying
variation...
"""

DIMENSIONS_STUDIED = [2,3,4,6,8]
"""
We construct classifications by intersecting dimensions
In the current implementation, we study classifications with
the same number of dimensions as the underlying trait
transmission dimensionality (num_loci).  But we need to study
systems with differing values for num_loci, and thus classifications
with the same number of dimensions
"""

INNOVATION_RATES_STUDIED = [0.0001,0.00025,0.0005,0.001,0.0025,0.005,0.01,0.025]
"""
We examine a range of innovation rates, from slow innovation rates that would imply one innovation
every ten "generations" for a population of 1000 individuals, to rates which imply that at least
10 individuals per generation in a population of 1000 will have a new idea.
"""


SIMULATION_LENGTH_AFTER_STATIONARITY = 10000
"""
We do not begin taking samples from the population until the Markov chain corresponding to the
simulation is fully mixed (i.e., at quasi-stationary equilibrium).  That value is calculated based
upon other simulation parameters.  The simulation continues to run for this many generations
AFTER that stationarity time is reached.  This value should be adjusted for temporal
aggregation studies to give you a sufficient number of samples for your longest TA duration
window.  Making this a derived value from TA would be desirable.
"""


TIME_AVERAGING_DURATIONS_STUDIED = [1000,500,250,125,63,32,16,8,1]
"""
We want to be able to examine the effects of temporal aggregation AND classification, in addition
to just classification.
"""


NUM_SAMPLES_ANALYZED_PER_FINAL_SAMPLE_PATH = 10
"""
Each simulation parameter combination is analyzed with different sample sizes, dimensionality,
classifications, and levels of time averaging.  At the maximum level of TA (1000?), with 10K
raw steps in the simulation post-stationarity, we'll end up wtih 10 samples of the coarsest duration.
We'll want to have the same number of samples at finer durations as well, even though we could have
1250 samples of duration 8, or 10K samples of duration 1.
"""



INITIAL_TRAIT_NUMBER = 10
"""
This constant sets the number of initial traits to which the population is initialized at time zero.
Since we wait for the system to reach stationarity, this value matters only in that it slightly affects
the difference between actual stationary mixing time, and the estimated average value.  I set it
relatively low since the genetic approximations for mixing time usually work with a small number of
alleles at a locus.
"""


SAMPLING_INTERVAL = 100<commit_msg>CTPy-15 Identification of raw data samples to classifications: Added experiment tracking - record completion - after classification identification for all classifications.<commit_after>"""
Global definitions for CTPy
"""

### Internal constants - need no documentation
# Constants tagging different types of dimensions, for DB queries
MODETYPE_EVEN = str("EVEN")
MODETYPE_RANDOM = str('RANDOM')



### Research-level constants

MAXALLELES = 1000000000
"""
simuPOP does infinite-allele models by giving a really large number to the KAllelesMutator,
and ensuring that the combination of popsize * mutationrate * simlength doesn't exceed MAXALLELES
This value is thus very important for defining classifications since it defines an integer-valued
trait space.
"""


NUM_REPLICATES_FOR_RANDOM_DIMENSION_MODES = 10
"""
When creating random dimensions for classification, we want this many random choices
of mode boundaries for each choice of mode count (e.g., 2 modes per dimension, 3 modes...)
"""

DIMENSION_PARTITIONS = [2,3,4,8,16,32]
"""
Classifications with 32 modes per dimension are rare in archaeological practice
when classes are discrete/nominal categories, but we want to see scaling of
observables, and we can imagine cases with random dimensions where we might
have an ability to chop things into chunks but not understand how those map to underlying
variation...
"""

DIMENSIONS_STUDIED = [2,3,4,6,8]
"""
We construct classifications by intersecting dimensions
In the current implementation, we study classifications with
the same number of dimensions as the underlying trait
transmission dimensionality (num_loci).  But we need to study
systems with differing values for num_loci, and thus classifications
with the same number of dimensions
"""

INNOVATION_RATES_STUDIED = [0.0001,0.00025,0.0005,0.001,0.0025,0.005,0.01,0.025]
"""
We examine a range of innovation rates, from slow innovation rates that would imply one innovation
every ten "generations" for a population of 1000 individuals, to rates which imply that at least
10 individuals per generation in a population of 1000 will have a new idea.
"""


SIMULATION_LENGTH_AFTER_STATIONARITY = 10000
"""
We do not begin taking samples from the population until the Markov chain corresponding to the
simulation is fully mixed (i.e., at quasi-stationary equilibrium).  That value is calculated based
upon other simulation parameters.  The simulation continues to run for this many generations
AFTER that stationarity time is reached.  This value should be adjusted for temporal
aggregation studies to give you a sufficient number of samples for your longest TA duration
window.  Making this a derived value from TA would be desirable.
"""


TIME_AVERAGING_DURATIONS_STUDIED = [1000,500,250,125,63,32,16,8,1]
"""
We want to be able to examine the effects of temporal aggregation AND classification, in addition
to just classification.
"""


NUM_SAMPLES_ANALYZED_PER_FINAL_SAMPLE_PATH = 10
"""
Each simulation parameter combination is analyzed with different sample sizes, dimensionality,
classifications, and levels of time averaging.  At the maximum level of TA (1000?), with 10K
raw steps in the simulation post-stationarity, we'll end up wtih 10 samples of the coarsest duration.
We'll want to have the same number of samples at finer durations as well, even though we could have
1250 samples of duration 8, or 10K samples of duration 1.
"""



INITIAL_TRAIT_NUMBER = 10
"""
This constant sets the number of initial traits to which the population is initialized at time zero.
Since we wait for the system to reach stationarity, this value matters only in that it slightly affects
the difference between actual stationary mixing time, and the estimated average value.  I set it
relatively low since the genetic approximations for mixing time usually work with a small number of
alleles at a locus.
"""


SAMPLING_INTERVAL = 1
{'fraction_non_alphanumeric': 0.053135659931776436, 'fraction_numerical': 0.03253739176069273, 'mean_word_length': 5.182481751824818, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '866906', 'n_tokens_mistral': 2162, 'n_tokens_neox': 1916, 'n_words': 1017}