One of the most tedious tasks in the application of machine learning is model selection, i.e. hyperparameter selection. Fortunately, recent progress has been made in the automation of this process, through the use of sequential model-based optimization (SMBO) methods. This can be used to optimize a cross-validation performance of a learning algorithm over the value of its hyperparameters. However, it is well known that ensembles of learned models almost consistently outperform a single model, even if properly selected. In this paper, we thus propose an extension of SMBO methods that automatically constructs such ensembles. This method builds on a recently proposed ensemble construction paradigm known as agnostic Bayesian learning. In experiments on 22 regression and 39 classification data sets, we confirm the success of this proposed approach, which is able to outperform model selection with SMBO.
On the other hand, traditional ensemble methods such as , , and @cite_3 require a predefined set of models and are not straightforward to adapt to an infinite set of models.
@cite_3
1534477342 2771061327 2627058984 91160652
Ensemble methods are learning algorithms that construct a set of classifiers and then classify new data points by taking a (weighted) vote of their predictions. The original ensemble method is Bayesian averaging, but more recent algorithms include error-correcting output coding, Bagging, and boosting. This paper reviews these methods and explains why ensembles can often perform better than any single classifier. Some previous studies comparing ensemble methods are reviewed, and some new experiments are presented to uncover the reasons that Adaboost does not overfit rapidly. The superior performance of ensemble methods with infinite models are well known. Most of these methods are based on optimization problems in infinite-dimensional spaces with some regularization, for instance, boosting methods and convex neural networks use @math -regularization with the non-negative constraint. However, due to the difficulty of handling @math -regularization, these problems require early stopping or a rough approximation to solve it inexactly. In this paper, we propose a new ensemble learning method that performs in a space of probability measures, that is, our method can handle the @math -constraint and the non-negative constraint in a rigorous way. Such an optimization is realized by proposing a general purpose stochastic optimization method for learning probability measures via parameterization using transport maps on base models. As a result of running the method, a transport map to output an infinite ensemble is obtained, which forms a residual-type network. From the perspective of functional gradient methods, we give a convergence rate as fast as that of a stochastic optimization method for finite dimensional nonconvex problems. Moreover, we show an interior optimality property of a local optimality condition used in our analysis. Machine improvisation and related style simulation problems usually consider building repre- sentations of time-based media data, such as music, either by explicit coding of rules or applying machine learning methods. Stylistic learning applies such methods to musical sequences in order to capture salient musical features and organize these features into a model. The Stylistic simulation process browses the model in order to generate variant musical sequences that are stylistically consistent with the learned ma- terial. If both the learning process and the simulation process happen in real-time, in an interactive system where the computer “plays” with musicians, then Machine Improvisation is achieved. Improvisation models have to cope with a trade-off between completeness (all the possible patterns and their continuation laws are discovered) and incrementality (the completeness is ensured only asymptotically for infinite sequences). In a previous work we devised a complete and incremental model based on the Factor Oracle Algorithm. In this paper we propose a concurrent constraints model for the Factor Oracle and show how it can be used in a concurrent learning improvisation situation. Our model is based on a non-deterministic concurrent constraint process calculus (NTCC). Such an approach allows the system to respond in a faster and more flexible manner to real-life performance situations. In addition, the declarative nature of constraints greatly simplifies the expansion of the system with improvisation rules at a higher musical level. We also describe the implementation of our model in a NTCC interpreter written in Common Lisp that is capable of real time performance. Monte-Carlo planning algorithms, such as UCT, select actions at each decision epoch by intelligently expanding a single search tree given the available time and then selecting the best root action. Recent work has provided evidence that it can be advantageous to instead construct an ensemble of search trees and to make a decision according to a weighted vote. However, these prior investigations have only considered the application domains of Go and Solitaire and were limited in the scope of ensemble configurations considered. In this paper, we conduct a more exhaustive empirical study of ensemble Monte-Carlo planning using the UCT algorithm in a set of six additional domains. In particular, we evaluate the advantages of a broad set of ensemble configurations in terms of space and time efficiency in both parallel and single-core models. Our results demonstrate that ensembles are an effective way to improve performance per unit time given a parallel time model and performance per unit space in a single-core model. However, contrary to prior isolated observations, we did not find significant evidence that ensembles improve performance per unit time in a single-core model.