Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> CASE STUDIES <s> Model-combining (i.e., mixing) methods have been proposed in recent years to deal with uncertainty in model selection. Even though advantages of model combining over model selection have been demonstrated in simulations and data examples, it is still unclear to a large extent when model combining should be preferred. In this work, first we propose an instability measure to capture the uncertainty of model selection in estimation, called perturbation instability in estimation (PIE), based on perturbation of the sample. We demonstrate that estimators from model selection can have large PIE values and that model combining substantially reduces the instability for such cases. Second, we propose a model combining method, adaptive regression by mixing with model screening (ARMS), and derive a theoretical property. In ARMS, a screening step is taken to narrow down the list of candidate models before combining, which not only saves computing time, but also can improve estimation accuracy. Third, we compare ARMS w... <s> BIB001 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> CASE STUDIES <s> Summary 1. Ecologists use statistical models for both explanation and prediction, and need techniques that are flexible enough to express typical features of their data, such as nonlinearities and interactions. 2. This study provides a working guide to boosted regression trees (BRT), an ensemble method for fitting statistical models that differs fundamentally from conventional techniques that aim to fit a single parsimonious model. Boosted regression trees combine the strengths of two algorithms: regression trees (models that relate a response to their predictors by recursive binary splits) and boosting (an adaptive method for combining many simple models to give improved predictive performance). The final BRT model can be understood as an additive regression model in which individual terms are simple trees, fitted in a forward, stagewise fashion. 3. Boosted regression trees incorporate important advantages of tree-based methods, handling different types of predictor variables and accommodating missing data. They have no need for prior data transformation or elimination of outliers, can fit complex nonlinear relationships, and automatically handle interaction effects between predictors. Fitting multiple trees in BRT overcomes the biggest drawback of single tree models: their relatively poor predictive performance. Although BRT models are complex, they can be summarized in ways that give powerful ecological insight, and their predictive performance is superior to most traditional modelling methods. 4. The unique features of BRT raise a number of practical issues in model fitting. We demonstrate the practicalities and advantages of using BRT through a distributional analysis of the short-finned eel ( Anguilla australis Richardson), a native freshwater fish of New Zealand. We use a data set of over 13 000 sites to illustrate effects of several settings, and then fit and interpret a model using a subset of the data. We provide code and a tutorial to enable the wider use of BRT by ecologists. <s> BIB002
All methods discussed above can be applied to simple regression models, while some explicitly rely on a model's likelihood and can thus not be used for non-parametric approaches. We therefore devised two case studies, the first being a rather simple example to illustrate the use of all methods in Table 1 , and the second a more complicated species distribution case study based on a reduced set of methods. Note that we do not include adaptive regression by mixing with model screening (ARMS; Yang 2001) because its more sophisticated variations BIB001 are not readily implemented in R, and the basic ARMS is barely different from AIC model averaging for a preselected set of models. Case study 1: Simulation with Gaussian response, many models and few data points In this first, simulation-based case study, we explore the variability of model-averaging approaches in the common case where several partially nested models are fit (see Data S1 for details and code). The simulation was set up so that several of the fitted models have similar support as explanations for the data. This was achieved by generating the response differently in each of two groups (using similar, but not identical predictors). We simulated 70 data points with 4 predictors yielding 2 4 = 16 candidate models, and another 70 data points for validation. We computed model weights in 19 different ways (Table 1 ) and compared the prediction error of weighted averages, as well as of the individual models to the validation data points. Simulation and analyses were repeated 100 times. Two results emerged from this simulation that are worth reporting. First, prediction error (quantified as RMSE) was similar across the 19 weight-computing approaches, with a few noticeably poor exceptions (the two MBMC approaches, minimal variance and the cos-squared scheme; Fig. 6 ), and most were no better than those of the best nine single model predictions. Second, most averaging approaches gave some weight (w [ 0:01) to 10 or more models (Table 2) , despite models being overlapping and partially nested, so that we have actually only five (more or less) independent models (those containing only one predictor: m2, m3, m5, m9 and intercept-only m1). In real data sets, such spreading of weight is the result of data sparseness or extreme noise, making important effects stand out less; indeed, half of our candidate models are not hugely different, i.e., within DAIC < 4. Case study 2: Real species presence-absence data, many data points and a moderate number of predictors In the second case study, we use data on the real distribution of short-finned eel (Anguilla australis) in New Zealand (from BIB002 . The data are provided in the R package dismo, already split into a 1000-row training and a 500row test data set, and featuring 10 predictors. We ran four different model types (GAM, Random Forest [rF] , artificial neural network [ANN], support vector machine [SVM]) using all 10 predictors, along with two variations of the GLM (best models selected by AIC and BIC from the full model containing the 10 predictors, relevant quadratic terms and all firstorder interactions). For details, see Data S1. The number of averaging approaches that can be used to compute model weights is smaller than in the previous case study, as three of the six models do not report a likelihood or the number of parameters, precluding the use of rjMCMC, Bayes factor, (W)AIC, BIC, and Mallows' Cp. Because we do not know the underlying data-generating model, we evaluate the models on the randomly pre-selected test data provided. One interesting result is that model averaging was effectively a model selection tool in several cases (Table 3) . Stacking, bootstrapping, JMA, and to a lesser degree minimal variance, BMA-EM, and the model-based model combinations yielded non-zero weights for only one (or two) models. Apparently, these approaches yielded suboptimal Table 1 for list of approaches, and case study 1 in Data S1 for list and its of the individual models.