<jupyter_start><jupyter_text>Non-Iterative OptimizersAKA Level 2 optimizers, are unified 3rd party solutions for random expressions. Look at this space:<jupyter_code>from tune import Space, Grid, Rand

space = Space(a=Grid(1,2), b=Rand(0,1))
list(space)<jupyter_output><empty_output><jupyter_text>`Grid` is for level 1 optimization, all level 1 parameters will be converted to static values before execution. And level 2 parameters will be optimized during runtime using level 2 optimizers. So for the above example, if we have a Spark cluster and Hyperopt, then we can use Hyperot to search for the best `b` on each of the 2 configurations. And the 2 jobs are parallelized by Spark.<jupyter_code>from tune import noniterative_objective, Trial

@noniterative_objective
def objective(a ,b) -> float:
    return a**2 + b**2

trial = Trial("dummy", params=list(space)[0])<jupyter_output><empty_output><jupyter_text>Use DirectlyNotice normally you don't use them directly, instead you should use them through top level APIs. This is just to demo how they work. Hyperopt<jupyter_code>from tune_hyperopt import HyperoptLocalOptimizer

hyperopt_optimizer = HyperoptLocalOptimizer(max_iter=200, seed=0)
report = hyperopt_optimizer.run(objective, trial)

print(report.sort_metric, report)<jupyter_output>1.0000000001665414 {'trial': {'trial_id': 'dummy', 'params': {'a': 1, 'b': 1.2905089873156781e-05}, 'metadata': {}, 'keys': []}, 'metric': 1.0000000001665414, 'params': {'a': 1, 'b': 1.2905089873156781e-05}, 'metadata': {}, 'cost': 1.0, 'rung': 0, 'sort_metric': 1.0000000001665414, 'log_time': datetime.datetime(2021, 10, 6, 23, 30, 51, 970344)}<jupyter_text>Optuna<jupyter_code>from tune_optuna import OptunaLocalOptimizer
import optuna

optuna.logging.disable_default_handler()

optuna_optimizer = OptunaLocalOptimizer(max_iter=200)
report = optuna_optimizer.run(objective, trial)

print(report.sort_metric, report)<jupyter_output>1.0000000003655019 {'trial': {'trial_id': 'dummy', 'params': {'a': 1, 'b': 1.9118105424729645e-05}, 'metadata': {}, 'keys': []}, 'metric': 1.0000000003655019, 'params': {'a': 1, 'b': 1.9118105424729645e-05}, 'metadata': {}, 'cost': 1.0, 'rung': 0, 'sort_metric': 1.0000000003655019, 'log_time': datetime.datetime(2021, 10, 6, 23, 31, 26, 6566)}<jupyter_text>As you see, we have unified the interfaces for using these frameworks. In addition, we also unified the semantic of the random expressions, so the random sampling behavior will be highly consistent on different 3rd party solutions.  Use Top Level APIIn the following example, we directly use the entire `space` where you can mix grid search, random search and Bayesian Optimization.<jupyter_code>from tune import suggest_for_noniterative_objective

report = suggest_for_noniterative_objective(
    objective, space, top_n=1,
    local_optimizer=hyperopt_optimizer
)[0]

print(report.sort_metric, report)<jupyter_output>NativeExecutionEngine doesn't respect num_partitions ROWCOUNT<jupyter_text>You can also provide only random expressions in space, and use in the same way so it looks like a common case similar to the examples<jupyter_code>report = suggest_for_noniterative_objective(
    objective, Space(a=Rand(-1,1), b=Rand(-100,100)), top_n=1,
    local_optimizer=optuna_optimizer
)[0]

print(report.sort_metric, report)<jupyter_output>NativeExecutionEngine doesn't respect num_partitions ROWCOUNT<jupyter_text>Factory MethodIn the above example, if we don't set `local_optimizer`, then the default level 2 optimizer will be used which can't handle a configuration with random expressions.So we have a nice way to make certain optimizer the default one.<jupyter_code>from tune import NonIterativeObjectiveLocalOptimizer, TUNE_OBJECT_FACTORY

def to_optimizer(obj):
    if isinstance(obj, NonIterativeObjectiveLocalOptimizer):
        return obj
    if obj is None or "hyperopt"==obj:
        return HyperoptLocalOptimizer(max_iter=200, seed=0)
    if "optuna" == obj:
        return OptunaLocalOptimizer(max_iter=200)
    raise NotImplementedError

TUNE_OBJECT_FACTORY.set_noniterative_local_optimizer_converter(to_optimizer)<jupyter_output><empty_output><jupyter_text>Now Hyperopt becomes the default level 2 optimizer, and you can switch to Optuna by specifying a string parameter<jupyter_code>report = suggest_for_noniterative_objective(
    objective, Space(a=Rand(-1,1), b=Rand(-100,100)), top_n=1
)[0]  # using hyperopt

print(report.sort_metric, report)

report = suggest_for_noniterative_objective(
    objective, Space(a=Rand(-1,1), b=Rand(-100,100)), top_n=1,
    local_optimizer="optuna"
)[0]  # using hyperopt

print(report.sort_metric, report)<jupyter_output>NativeExecutionEngine doesn't respect num_partitions ROWCOUNT
NativeExecutionEngine doesn't respect num_partitions ROWCOUNT
{'fraction_non_alphanumeric': 0.1008982661374556, 'fraction_numerical': 0.060162941299352415, 'mean_word_length': 6.823529411764706, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 6, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '495581', 'license': 'Apache-2.0', 'n_tokens_mistral': 1748, 'n_tokens_neox': 1517, 'n_words': 451, 'path': 'docs/notebooks/noniterative_optimizers.ipynb', 'repo_name': 'fugue-project/tune', 'size': '4787'}