This work provides an automatic machine learning (AutoML) modelling architecture called Autostacker. Autostacker improves the prediction accuracy of machine learning baselines by utilizing an innovative hierarchical stacking architecture and an efficient parameter search algorithm. Neither prior domain knowledge about the data nor feature preprocessing is needed. We significantly reduce the time of AutoML with a naturally inspired algorithm - Parallel Hill Climbing (PHC). By parallelizing PHC, Autostacker can provide candidate pipelines with sufficient prediction accuracy within a short amount of time. These pipelines can be used as is or as a starting point for human experts to build on. By focusing on the modelling process, Autostacker breaks the tradition of following fixed order pipelines by exploring not only single model pipeline but also innovative combinations and structures. As we will show in the experiment section, Autostacker achieves significantly better performance both in terms of test accuracy and time cost comparing with human initial trials and recent popular AutoML system. Machine Learning nowadays is the main approach for people to solve prediction problems by utilizing the power of data and algorithms. More and more models have been proposed to solve diverse problems based on the character of these problems. More specifically, different learning targets and collected data correspond to different modelling problems. To solve them, data scientists not only need to know the advantages and disadvantages of various models, they also need to manually tune the hyperparameters within these models. However, understanding thoroughly all of the models and running experiments to tune the hyperparameters involves a lot of effort and cost. Thus, automating the modelling procedure is highly desired both in academic areas and industry.An AutoML system aims at providing an automatically generated baseline with better performance to support data scientists and experts with specific domain knowledge to solve machine learning problems with less effort. The input to AutoML is a cleanly formatted dataset and the output is one or multiple modelling pipelines which enables the data scientists to begin working from a better starting point. There are some pioneering efforts addressing the challenge of finding appropriate configurations of modelling pipelines and providing some mechanisms to automate this process. However, these works often rely on fixed order machine learning pipelines which are obtained by mimicking the traditional working pipelines of human experts. This initial constraint limits the potential of machine to find better pipelines which may or may not be straightforward, and may or may not have been tried by human experts before.In this work, we present an architecture called Autostacker which borrows the stacking Wolpert (1992) BID1 method from ensemble learning, but allows for the discovery of pipelines made up of simply one model or many models combined in an innovative way. All of the automatically generated pipelines from Autostacker will provide a good enough starting point compared with initial trials of human experts. However, there are several challenges to accomplish this:• The quality of the datasets. Even though we are stepping into a big data era, we have to admit that there are still a lot of problems for which it is hard to collect enough data, especially data with little noise, such as historical events, medical research, natural disasters and so on. We tackle this challenge by always using the raw dataset in all of the stacking layers Figure 1 : This figure describes the pipeline architecture of Autostacker. Autostacker pipelines consists of one or multiple layers and one or multiple nodes inside each layer. Each node represents a machine learning primitive model, such as SVM, MLP, etc. The number of layers and the number of nodes per layer can be specified beforehand or they can be changeable as part of the hyperparameters. In the first layer, the raw dataset is used as input. Then in the following layers, the prediction results from each node will be added to the raw dataset as synthetic features (new colors). The new dataset generated by each layer will be used as input to the next layer.while also adding synthetic features in each stacking layer to fully use the information in the current dataset. More details are provided in the Approach section below.• The generalization ability of the AutoML framework. As mentioned above, existing AutoML frameworks only allow systems to generate an assembly line from data preprocessing and feature engineering to model selection where only a specific single model will be utilized by plugging in a previous model library. In this paper, depending on the computational cost and time cost, we make the number of such primitive models a variable which can be changed dynamically during the pipeline generation process or initialized in the beginning. This means that the simplest pipeline could be a single model, and the most complex pipeline could contain hundreds of primitive models as shown in Figure 1 • The large space of variables. The second challenge mentioned above leads to this problem naturally. Considering the whole AutoML framework, variables include the type of primitive machine learning models, the configuration settings of the framework (for instance, the number of primitive models in each stacking layer) and the hyperparameters in each primitive model. One way to address this issue is to treat this as an optimization problem BID3 . Here in this paper, we instead treat this challenge as a search problem. We propose to use a naturally inspired algorithm, Parallel Hill Climbing (PHC), BID10 to effectively search for appropriate candidate pipelines.To make the definition of the problem clear, we will use the terminology listed below throughout this paper:• Primitive and Pipeline: primitive denotes an existed single machine learning model, for example, a DecisionTree. In addition, these also include traditional ensemble learning models, such as Adaboost and Bagging. The pipeline is the form of the output of Autostacker, which is a single primitive or a combination of primitives.• Layer and Node : Figure 1 shows the architecture of Autostacker which is formed by multiple stacking layers and multiple nodes in each layers. Each node represents a machine learning primitive model. During the experiments and research process, we noticed that Autostacker still has several limitations. Here we will describe these limitations and possible future solutions:• The ability to automate the machine learning process for large scale datasets is limited. Nowadays, there are more sophisticated models or deep learning approaches which achieve very good results on large scale datasets and multi-task problems. Our current primitive library and modelling structure is very limited at solving these problems. One of the future solutions could be to incorporate more advanced primitives and to choose to use them when necessary.• Autostacker can be made more efficient with better search algorithms. There are a lot of modern evolutionary algorithms, and some of them are based on the Parallel Hill Climber that we use in this work. We believe that Autostacker could be made faster by incorporating them. We also believe traditional methods and knowledge from statistics and probability will be very helpful to better understand the output of Autostacker, such as by answering questions like: why do was a particular pipeline chosen as one of the final candidate pipelines? In this work, we contribute to automating the machine learning modelling process by proposing Autostacker, a machine learning system with an innovative architecture for automatic modelling and a well-behaved efficient search algorithm. We show how this system works and what the performance of this system is, comparing with human initial trails and related state of art techniques. We also demonstrate the scaling and parallelization ability of our system. In conclusion, we automate the machine learning modelling process by providing an efficient, flexible and well-behaved system which provides the potential to be generalized into complicated problems and is able to be integrated with data and feature processing modules.
Automate machine learning system with efficient search algorithm and innovative structure to provide better model baselines.
{"src": "scitldr-AIC_train", "paper_id": "SyvCD-b0W"}