<jupyter_start><jupyter_text>Smart Selectors<jupyter_code>%load_ext autoreload
%autoreload 2<jupyter_output><empty_output><jupyter_text>Let's start by defining a task.<jupyter_code>from metricx import Metric, Task

task = Task(
    name="task",
    metrics=[
        Metric(name="score", is_higher_better=True),
    ],
)<jupyter_output><empty_output><jupyter_text>Now suppose we have three models whose performance on the task is distributed as follows.<jupyter_code>import numpy as np

models = {
    "model-A": lambda: np.random.normal(loc=0.0, scale=3.0),
    "model-B": lambda: np.random.normal(loc=10.0, scale=1.0),
    "model-C": lambda: np.random.normal(loc=1.0, scale=1.0),
    "model-D": lambda: np.random.normal(loc=-1.0, scale=0.5),
}
for model, func in models.items():
    task.report(model, {"score": func()})<jupyter_output><empty_output><jupyter_text>The selector provides a method to help you choose the next model to evaluate on this task. It allows you to customize the policy used to select models. The default policy is to start by obtaining 3 samples for each model and then transitioning to randomly choosing between several heuristic policies ranging from standard errors to power analysis.<jupyter_code>from copy import deepcopy
from metricx import Selector

selector = Selector(task)
for _ in range(20):
    model = selector.propose()
    task.report(model, {"score": models[model]()})
task.to_figure();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11591220850480109, 'fraction_numerical': 0.01440329218106996, 'mean_word_length': 5.399122807017544, 'pattern_counts': {'":': 6, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '657610', 'license': 'MIT', 'n_tokens_mistral': 478, 'n_tokens_neox': 462, 'n_words': 138, 'path': 'notebooks/2_Smart_Selectors.ipynb', 'repo_name': 'k15z/MetricX', 'size': '1458'}