Table REF  shows the hyperparameter search space of all models for all tasks. To optimize over this hyperparameter space, we used the Optuna [1]} framework with 100 iterations. It is possible to dynamically construct the parameter search space because the framework allows users to formulate hyperparameter optimization as the maximization/minimization process of an objective function that takes a set of hyperparameters as input and returns a validation score. Furthermore, it provides efficient sampling methods, such as relational sampling that exploits the correlations among the parameters.
