<jupyter_start><jupyter_text>Example usage with Python 3This notebook demonstrates usage of `petab_select` to perform forward selection in a Python 3 script.  Problem setup with initial model Dependencies are imported. A model selection problem is loaded from the specification files. Some helper methods are defined.<jupyter_code>import petab_select
from petab_select import (
    Model,
    ForwardCandidateSpace,
)


# Load the PEtab Select problem.
select_problem = petab_select.Problem.from_yaml('model_selection/petab_select_problem.yaml')
# Fake criterion values as a surrogate for a model calibration tool.
fake_criterion = {
    'M1_0': 200,
    'M1_1': 150,
    'M1_2': 140,
    'M1_3': 130,
    'M1_4': -40,
    'M1_5': -70,
    'M1_6': -110,
    'M1_7': 50,
}


def print_model(model: Model) -> None:
    """Helper method to view model attributes."""
    print(
f"""\
Model subspace ID: {model.model_subspace_id}
PEtab YAML location: {model.petab_yaml}
Custom model parameters: {model.parameters}
Model hash: {model.get_hash()}
Model ID: {model.model_id}
{select_problem.criterion}: {model.get_criterion(select_problem.criterion, compute=False)}
"""
    )


def calibrate(model: Model, fake_criterion=fake_criterion) -> None:
    """Set model criterion values to fake values that could be the output of a calibration tool.
    
    Each model subspace in this problem contains only one model, so a model-specific criterion can
    be indexed by the model subspace ID.
    """
    model.set_criterion(select_problem.criterion, fake_criterion[model.model_subspace_id])


print(
f"""Information about the model selection problem.

YAML path: {select_problem.yaml_path}
Method: {select_problem.method}
Criterion: {select_problem.criterion}
"""
)<jupyter_output>Information about the model selection problem.

YAML path: model_selection/petab_select_problem.yaml
Method: forward
Criterion: AIC<jupyter_text>First iterationNeighbors of the initial model in the model space are identified for testing. Here, no initial model is specified. If an initial model is required for the algorithm, PEtab Select can automatically use a virtual initial model, if such a model is defined. For example, for the forward and backward methods, the virtual initial model defaults to a model with no parameters estimated, and all parameters estimated, respectively. The model candidate space is setup with the initial model. The model space is then used to find neighbors to the initial model. The candidate space is used to calculate distances between models, and whether a candidate model represents a valid move in model space.The in-built `ForwardCandidateSpace` uses the following properties to identify candidate models:- previously estimated parameters must not be fixed;- the number of estimated parameters must increase; and- the increase in the number of estimated parameters must be minimal.The model space keeps a history of identified neighbors, such that subsequent calls ignore previously identified neighbors. This can be disabled by changing usage to `petab_select.ModelSpace.search(..., exclude=False)`, or reset to forget all history with `petab_select.ModelSpace.reset()`.<jupyter_code>candidate_space = petab_select.ui.candidates(problem=select_problem)<jupyter_output><empty_output><jupyter_text>Model IDs default to the model hash, which is generated from hashing the model subspace ID and model parameterization.Here, the model identified is a model with all possible parameters fixed. This is because the default virtual initial model is the same parameterization, and the closest model in the "real" model subspace is the same parameterization. If the initial model was from the "real" model subspace, then candidate models would be true forward steps in the subspace (e.g. an increase in the number of estimated parameters). Each of the candidate models includes information that should be sufficient for model calibration with any suitable tool that supports PEtab.NB: the `petab_yaml` is for the original PEtab problem, and would need to be customized by `parameters` to be the actual candidate model.<jupyter_code>for candidate_model in candidate_space.models:
    print_model(candidate_model)<jupyter_output>Model subspace ID: M1_0
PEtab YAML location: model_selection/petab_problem.yaml
Custom model parameters: {'k1': 0, 'k2': 0, 'k3': 0}
Model hash: 65f94e223024ef684fe3e1a3aa2a54cc3ffd08895fbe4539512522d49d703ceda483aff4aa207b160dc358c458b76b25d88fbd94cacfc78bd0c70f4a46a42191
Model ID: 65f94e223024ef684fe3e1a3aa2a54cc3ffd08895fbe4539512522d49d703ceda483aff4aa207b160dc358c458b76b25d88fbd94cacfc78bd0c70f4a46a42191
AIC: None<jupyter_text>At this point, a model calibration tool is used to find the best of the test models, according to some criterion. PEtab select can select the best model from a collection of models that provide a value for this criterion, or a specific model can be supplied. Here, PEtab Select will be used to select the best model from multiple models. At the end of the following iterations, a specific model will be provided.<jupyter_code># Set fake criterion values that might be the output of a model calibration tool.
for candidate_model in candidate_space.models:
    calibrate(candidate_model)
select_problem.add_calibrated_models(candidate_space.models)
local_best_model = select_problem.get_best(candidate_space.models)
print_model(local_best_model)<jupyter_output>Model subspace ID: M1_0
PEtab YAML location: model_selection/petab_problem.yaml
Custom model parameters: {'k1': 0, 'k2': 0, 'k3': 0}
Model hash: 65f94e223024ef684fe3e1a3aa2a54cc3ffd08895fbe4539512522d49d703ceda483aff4aa207b160dc358c458b76b25d88fbd94cacfc78bd0c70f4a46a42191
Model ID: 65f94e223024ef684fe3e1a3aa2a54cc3ffd08895fbe4539512522d49d703ceda483aff4aa207b160dc358c458b76b25d88fbd94cacfc78bd0c70f4a46a42191
AIC: 200<jupyter_text>Second iterationThe process then repeats. The chosen model is used as the predecessor model, such that neighboring models are identified with respect to the chosen model.<jupyter_code>petab_select.ui.candidates(
    problem=select_problem,
    candidate_space=candidate_space,
    predecessor_model=select_problem.get_best(candidate_space.models),
);
for candidate_model in candidate_space.models:
    print_model(candidate_model)
# Set fake criterion values that might be the output of a model calibration tool.
for candidate_model in candidate_space.models:
    calibrate(candidate_model)
select_problem.add_calibrated_models(candidate_space.models)
local_best_model = select_problem.get_best(candidate_space.models)
print_model(local_best_model)<jupyter_output>Model subspace ID: M1_3
PEtab YAML location: model_selection/petab_problem.yaml
Custom model parameters: {'k1': 'estimate', 'k2': 0.1, 'k3': 0}
Model hash: b7584bfd6f35206dfe32fa0143e53cea808faf965e0c0547bf6ee1cdce7a75cd3ff0aa2bcb1faa27625166454f83e3fcac52cdf43b28e8186fff9a01ac3f8006
Model ID: b7584bfd6f35206dfe32fa0143e53cea808faf965e0c0547bf6ee1cdce7a75cd3ff0aa2bcb1faa27625166454f83e3fcac52cdf43b28e8186fff9a01ac3f8006
AIC: 130<jupyter_text>Third iteration<jupyter_code>petab_select.ui.candidates(
    problem=select_problem,
    candidate_space=candidate_space,
    predecessor_model=select_problem.get_best(candidate_space.models),
);
for candidate_model in candidate_space.models:
    print_model(candidate_model)
# Set fake criterion values that might be the output of a model calibration tool.
for candidate_model in candidate_space.models:
    calibrate(candidate_model)
select_problem.add_calibrated_models(candidate_space.models)
local_best_model = select_problem.get_best(candidate_space.models)
print_model(local_best_model)<jupyter_output>Model subspace ID: M1_6
PEtab YAML location: model_selection/petab_problem.yaml
Custom model parameters: {'k1': 'estimate', 'k2': 'estimate', 'k3': 0}
Model hash: db8700c079c8347123adc89b7f5112256c4aaebd2af0f6e32e7582f398b2c1e5e85e588cdcc56bab054c001b96a9b42b02174266927f879d7f78e8ac5d2c33e6
Model ID: db8700c079c8347123adc89b7f5112256c4aaebd2af0f6e32e7582f398b2c1e5e85e588cdcc56bab054c001b96a9b42b02174266927f879d7f78e8ac5d2c33e6
AIC: -110<jupyter_text>Fourth iteration<jupyter_code>petab_select.ui.candidates(
    problem=select_problem,
    candidate_space=candidate_space,
    predecessor_model=select_problem.get_best(candidate_space.models),
);
for candidate_model in candidate_space.models:
    print_model(candidate_model)
# Set fake criterion values that might be the output of a model calibration tool.
for candidate_model in candidate_space.models:
    calibrate(candidate_model)
select_problem.add_calibrated_models(candidate_space.models)
local_best_model = select_problem.get_best(candidate_space.models)
print_model(local_best_model)<jupyter_output>Model subspace ID: M1_7
PEtab YAML location: model_selection/petab_problem.yaml
Custom model parameters: {'k1': 'estimate', 'k2': 'estimate', 'k3': 'estimate'}
Model hash: 7c105406ec11716473939a0bbb5281066c1014b54e2480ba126030f5c18a597a27a2ca9247aa60d8262f488165079d1c9e040f9d712ec4e19c2d2122a586f3e5
Model ID: 7c105406ec11716473939a0bbb5281066c1014b54e2480ba126030f5c18a597a27a2ca9247aa60d8262f488165079d1c9e040f9d712ec4e19c2d2122a586f3e5
AIC: 50<jupyter_text>Sixth iteration<jupyter_code>petab_select.ui.candidates(
    problem=select_problem,
    candidate_space=candidate_space,
    predecessor_model=select_problem.get_best(candidate_space.models),
);<jupyter_output><empty_output><jupyter_text>The `M1_7` model is the most complex model in the model space (all parameters in the space are estimated), so no valid neighbors are identified for the forward selection method.<jupyter_code>print(f'Number of candidate models: {len(candidate_space.models)}.')<jupyter_output>Number of candidate models: 0.<jupyter_text>At this point, the results of the model calibration tool for the different models can be used to select the best model.<jupyter_code>best_model = select_problem.get_best(select_problem.calibrated_models)
print_model(best_model)<jupyter_output>Model subspace ID: M1_6
PEtab YAML location: model_selection/petab_problem.yaml
Custom model parameters: {'k1': 'estimate', 'k2': 'estimate', 'k3': 0}
Model hash: db8700c079c8347123adc89b7f5112256c4aaebd2af0f6e32e7582f398b2c1e5e85e588cdcc56bab054c001b96a9b42b02174266927f879d7f78e8ac5d2c33e6
Model ID: db8700c079c8347123adc89b7f5112256c4aaebd2af0f6e32e7582f398b2c1e5e85e588cdcc56bab054c001b96a9b42b02174266927f879d7f78e8ac5d2c33e6
AIC: -110<jupyter_text>Seventh iterationNote that there can exist additional, uncalibrated models in the model space, after a single forward algorithm terminates. These additional models can be identified with the brute-force method.<jupyter_code>candidate_space = petab_select.BruteForceCandidateSpace()
petab_select.ui.candidates(
    problem=select_problem,
    candidate_space=candidate_space,
    excluded_models=select_problem.calibrated_models,
);
for candidate_model in candidate_space.models:
    print_model(candidate_model)<jupyter_output>Model subspace ID: M1_4
PEtab YAML location: model_selection/petab_problem.yaml
Custom model parameters: {'k1': 0.2, 'k2': 'estimate', 'k3': 'estimate'}
Model hash: 38c95dd428b3e31da6969a50db4a1ccbcefe6d8824617d27ec2360e57d55647a25f3fdd45e5f0270786698606cbe496cd94be9495986dade4d1f1d166a4bf911
Model ID: 38c95dd428b3e31da6969a50db4a1ccbcefe6d8824617d27ec2360e57d55647a25f3fdd45e5f0270786698606cbe496cd94be9495986dade4d1f1d166a4bf911
AIC: None
{'fraction_non_alphanumeric': 0.061094091903719915, 'fraction_numerical': 0.10678336980306345, 'mean_word_length': 7.438700147710487, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '493879', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 4534, 'n_tokens_neox': 3810, 'n_words': 990, 'path': 'doc/examples/workflow_python.ipynb', 'repo_name': 'PEtab-dev/petab_select', 'size': '11425'}