Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> Two separate sets of forecasts of airline passenger data have been combined to form a composite set of forecasts. The main conclusion is that the composite set of forecasts can yield lower mean-square error than either of the original forecasts. Past errors of each of the original forecasts are used to determine the weights to attach to these two original forecasts in forming the combined forecasts, and different methods of deriving these weights are examined. <s> BIB001 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> A number of procedures for forecasting a time series from its own current and past values are surveyed. Forecasting performances of three methodsBox-Jenkins, Holt-Winters and stepwise autoregression-are compared over a large sample of economic time series. The possibility of combining individual forecasts in the production of an overall forecast is explored, and we present empirical results which indicate that such a procedure can frequently be profitable. <s> BIB002 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> It is well known that a linear combination of forecasts can outperform individual forecasts. The common practice, however, is to obtain a weighted average of forecasts, with the weights adding up to unity. This paper considers three alternative approaches to obtaining linear combinations. It is shown that the best method is to add a constant term and not to constrain the weights to add to unity. These methods are tested with data on forecasts of quarterly hog prices, both within and out of sample. It is demonstrated that the optimum method proposed here is superior to the common practice of letting the weights add up to one. <s> BIB003 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> A forecasting comparison is undertaken in which 49 univariate forecasting methods, plus various forecast pooling procedures, are used to forecast 215 U.S. monthly macroeconomic time series at three forecasting horizons over the period 1959 - 1996. All forecasts simulate real time implementation, that is, they are fully recursive. The forecasting methods are based on four classes of models: autoregressions (with and without unit root pretests), exponential smoothing, artificial neural networks, and smooth transition autoregressions. The best overall performance of a single method is achieved by autoregressions with unit root pretests, but this performance can be improved when it is combined with the forecasts from other methods. <s> BIB004 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> A method for improving weather and climate forecast skill has been developed. It is called a superensemble, and it arose from a study of the statistical properties of a low-order spectral model. Multiple regression was used to determine coefficients from multimodel forecasts and observations. The coefficients were then used in the superensemble technique. The superensemble was shown to outperform all model forecasts for multiseasonal, medium-range weather and hurricane forecasts. In addition, the superensemble was shown to have higher skill than forecasts based solely on ensemble averaging. <s> BIB005 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> We compare Bayes Model Averaging, BMA, to a non-Bayes form of model averaging called stacking. In stacking, the weights are no longer posterior probabilities of models; they are obtained by a technique based on cross-validation. When the correct data generating model (DGM) is on the list of models under consideration BMA is never worse than stacking and often is demonstrably better, provided that the noise level is of order commensurate with the coefficients and explanatory variables. Here, however, we focus on the case that the correct DGM is not on the model list and may not be well approximated by the elements on the model list. We give a sequence of computed examples by choosing model lists and DGM's to contrast the risk performance of stacking and BMA. In the first examples, the model lists are chosen to reflect geometric principles that should give good performance. In these cases, stacking typically outperforms BMA, sometimes by a wide margin. In the second set of examples we examine how stacking and BMA perform when the model list includes all subsets of a set of potential predictors. When we standardize the size of terms and coefficients in this setting, we find that BMA outperforms stacking when the deviant terms in the DGM 'point' in directions accommodated by the model list but that when the deviant term points outside the model list stacking seems to do better. Overall, our results suggest the stacking has better robustness properties than BMA in the most important settings. <s> BIB006 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> The control of epidemic malaria is a priority for the international health community and specific targets for the early detection and effective control of epidemics have been agreed. Interannual climate variability is an important determinant of epidemics in parts of Africa where climate drives both mosquito vector dynamics and parasite development rates. Hence, skilful seasonal climate forecasts may provide early warning of changes of risk in epidemic-prone regions. Here we discuss the development of a system to forecast probabilities of anomalously high and low malaria incidence with dynamically based, seasonal-timescale, multi-model ensemble predictions of climate, using leading global coupled ocean-atmosphere climate models developed in Europe. This forecast system is successfully applied to the prediction of malaria risk in Botswana, where links between malaria and climate variability are well established, adding up to four months lead time over malaria warnings issued with observed precipitation and having a comparably high level of probabilistic prediction skill. In years in which the forecast probability distribution is different from that of climatology, malaria decision-makers can use this information for improved resource allocation. <s> BIB007 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> Weed risk assessment has become an accepted methodology for examining the likelihood and consequence of a plant species becoming invasive outside of its native range. Weed risk assessment draws upon biological and ecological information to estimate the likelihood and magnitude of the threats posed by introducing non-indigenous plants. In geographical terms, this has traditionally been understood as within a new country following importation of plant material. However, recent risk assessment development has focused more specifically on intracountry risk posed by already-present invasive plants and is referred to as post-border weed risk management. This form of assessment calls for fine-scale predictions of invasive species habitat suitability. This study applies some of the more popular and widely available habitat prediction models that represent a variety of different statistical approaches (linear regression, logistic regression, Bayesian probability, Classification and Regression Trees, Genetic Algorithm for Rule-set Production) to a single invasive plant, the vertebrate-dispersed, fleshy fruited European olive (Olea europaea L.) in southern Australia. The relationships between the dependant (O. europaea distribution) and independent (soil and climate) variables are used in the models to produce predictive maps for each model. Accuracy was calculated for each model output as well as a combined surface to examine whether recent calls for ensemble modelling of distributions produces improved predictions. Overall, the combined prediction demonstrated superior accuracy compared to any individual model outputs. The combined outputs can be likened to mapped gradations of predicted habitat suitability. The type of output produced in this study should form a critical component of post-border weed risk management but more importantly, the methodology will add to this important discipline. <s> BIB008 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> A variety of predictive models is currently used to map the spatial distribution of earth surface processes and landforms. In this study, we tested statistical consensus methods in order to improve the predictive accuracy of geomorphological models. The distributions of 12 geomorphological formations were recorded at a resolution of 25ha in a sub-arctic landscape in northern Finland. Nine environmental variables were used to predict probabilities of occurrence of the formations using eight state-of-the-art modelling techniques. The probability values of the models were combined using four different consensus methods. The accuracy of the models was calculated using spatially independent test data by the area under the curve (AUC) of a receiver-operating characteristic (ROC) plot. The mean AUC values of the geomorphological models varied between 0.711 and 0.755 based on single-model techniques, whereas the corresponding values based on consensus methods ranged from 0.752 to 0.782. The weighted average consensus method had the highest predictive performance of all methods. It improved the accuracy of 11 predictions out of 12. The results of this study suggest that the consensus methods have clear advantages over single-model predictions. The simplicity of the consensus methods makes it straightforward to implement them in predictive modelling studies in geomorphology. <s> BIB009 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> Multi-model averaging is currently receiving a surge of attention in the atmospheric, hydrologic, and statistical literature to explicitly handle conceptual model uncertainty in the analysis of environmental systems and derive predictive distributions of model output. Such density forecasts are necessary to help analyze which parts of the model are well resolved, and which parts are subject to considerable uncertainty. Yet, accurate point predictors are still desired in many practical applications. In this paper, we compare a suite of different model averaging techniques by their ability to improve forecast accuracy of environmental systems. We compare equal weights averaging (EWA), Bates-Granger model averaging (BGA), averaging using Akaike’s information criterion (AICA), and Bayes’ Information Criterion (BICA), Bayesian model averaging (BMA), Mallows model averaging (MMA), and Granger-Ramanathan averaging (GRA) for two different hydrologic systems involving water flow through a 1950 km2 watershed and 5 m deep vadose zone. Averaging methods with weights restricted to the multi-dimensional simplex (positive weights summing up to one) are shown to have considerably larger forecast errors than approaches with unconstrained weights. Whereas various sophisticated model averaging approaches have recently emerged in the literature, our results convincingly demonstrate the advantages of GRA for hydrologic applications. This method achieves similar performance as MMA and BMA, but is much simpler to implement and use, and computationally much less demanding. <s> BIB010 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> Africa is predicted to be highly vulnerable to 21st century climatic changes. Assessing the impacts of these changes on Africa's biodiversity is, however, plagued by uncertainties, and markedly different results can be obtained from alternative bioclimatic envelope models or future climate projections. Using an ensemble forecasting framework, we examine projections of future shifts in climatic suitability, and their methodological uncertainties, for over 2500 species of mammals, birds, amphibians and snakes in sub-Saharan Africa. To summarize a priori the variability in the ensemble of 17 general circulation models, we introduce a consensus methodology that combines co-varying models. Thus, we quantify and map the relative contribution to uncertainty of seven bioclimatic envelope models, three multi-model climate projections and three emissions scenarios, and explore the resulting variability in species turnover estimates. We show that bioclimatic envelope models contribute most to variability, particularly in projected novel climatic conditions over Sahelian and southern Saharan Africa. To summarize agreements among projections from the bioclimatic envelope models we compare five consensus methodologies, which generally increase or retain projection accuracy and provide consistent estimates of species turnover. Variability from emissions scenarios increases towards late-century and affects southern regions of high species turnover centred in arid Namibia. Twofold differences in median species turnover across the study area emerge among alternative climate projections and emissions scenarios. Our ensemble of projections underscores the potential bias when using a single algorithm or climate projection for Africa, and provides a cautious first approximation of the potential exposure of sub-Saharan African vertebrates to climatic changes. The future use and further development of bioclimatic envelope modelling will hinge on the interpretation of results in the light of methodological as well as biological uncertainties. Here, we provide a framework to address methodological uncertainties and contextualize results. <s> BIB011 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> Stacked generalization is a general method of using a high-level model to combine lower-level models to achieve greater predictive accuracy. In this paper we address two crucial issues which have been considered to be a 'black art' in classification tasks ever since the introduction of stacked generalization in 1992 by Wolpert: the type of generalizer that is suitable to derive the higher-level model, and the kind of attributes that should be used as its input. We find that best results are obtained when the higher-level model combines the confidence (and not just the predictions) of the lower-level ones. ::: ::: We demonstrate the effectiveness of stacked generalization for combining three different types of learning algorithms for classification tasks. We also compare the performance of stacked generalization with majority vote and published results of arcing and bagging. <s> BIB012 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> Abstract Forest management requires prediction of forest growth, but there is no general agreement about which models best predict growth, how to quantify model parameters, and how to assess the uncertainty of model predictions. In this paper, we show how Bayesian calibration (BC), Bayesian model comparison (BMC) and Bayesian model averaging (BMA) can help address these issues. We used six models, ranging from simple parameter-sparse models to complex process-based models: 3PG, 4C, ANAFORE, BASFOR, BRIDGING and FORMIND. For each model, the initial degree of uncertainty about parameter values was expressed in a prior probability distribution. Inventory data for Scots pine on tree height and diameter, with estimates of measurement uncertainty, were assembled for twelve sites, from four countries: Austria, Belgium, Estonia and Finland. From each country, we used data from two sites of the National Forest Inventories (NFIs), and one Permanent Sample Plot (PSP). The models were calibrated using the NFI-data and tested against the PSP-data. Calibration was done both per country and for all countries simultaneously, thus yielding country-specific and generic parameter distributions. We assessed model performance by sampling from prior and posterior distributions and comparing the growth predictions of these samples to the observations at the PSPs. We found that BC reduced uncertainties strongly in all but the most complex model. Surprisingly, country-specific BC did not lead to clearly better within-country predictions than generic BC. BMC identified the BRIDGING model, which is of intermediate complexity, as the most plausible model before calibration, with 4C taking its place after calibration. In this BMC, model plausibility was quantified as the relative probability of a model being correct given the information in the PSP-data. We discuss how the method of model initialisation affects model performance. Finally, we show how BMA affords a robust way of predicting forest growth that accounts for both parametric and model structural uncertainty. <s> BIB013 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> Summary ::: ::: The use of both linear and generalized linear mixed-effects models (LMMs and GLMMs) has become popular not only in social and medical sciences, but also in biological sciences, especially in the field of ecology and evolution. Information criteria, such as Akaike Information Criterion (AIC), are usually presented as model comparison tools for mixed-effects models. ::: ::: ::: The presentation of ‘variance explained’ (R2) as a relevant summarizing statistic of mixed-effects models, however, is rare, even though R2 is routinely reported for linear models (LMs) and also generalized linear models (GLMs). R2 has the extremely useful property of providing an absolute value for the goodness-of-fit of a model, which cannot be given by the information criteria. As a summary statistic that describes the amount of variance explained, R2 can also be a quantity of biological interest. ::: ::: ::: One reason for the under-appreciation of R2 for mixed-effects models lies in the fact that R2 can be defined in a number of ways. Furthermore, most definitions of R2 for mixed-effects have theoretical problems (e.g. decreased or negative R2 values in larger models) and/or their use is hindered by practical difficulties (e.g. implementation). ::: ::: ::: Here, we make a case for the importance of reporting R2 for mixed-effects models. We first provide the common definitions of R2 for LMs and GLMs and discuss the key problems associated with calculating R2 for mixed-effects models. We then recommend a general and simple method for calculating two types of R2 (marginal and conditional R2) for both LMMs and GLMMs, which are less susceptible to common problems. ::: ::: ::: This method is illustrated by examples and can be widely employed by researchers in any fields of research, regardless of software packages used for fitting mixed-effects models. The proposed method has the potential to facilitate the presentation of R2 for a wide range of circumstances. <s> BIB014 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> We estimated the latitudinal velocity (km/decade) of northern and southern boundaries of core distributions for 30 woody taxa over the last 16 000 years (biotic velocities) using networks of fossil pollen records, and compared these with climate velocities estimated from CCSM3 simulations. Biotic velocities were faster during periods of rapid temperature change (i.e. 16 to 7 ka) than times of relative stability (i.e. 7 to 1 ka), with a consistent northward movement of northern and southern boundaries. For most taxa, biotic velocities were faster for northern than for southern boundaries between 12 and 7 ka, resulting in expanding distributions. For individual time periods, biotic velocities were as fast or faster than climate velocities calculated using multivariate approaches. These results indicate that climate change paced the rate of distribution shifts in both northern and southern populations while suggesting that northern populations were more sensitive. A similar sensitivity and pacing is expected under 21st century climate change. <s> BIB015 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> Abstract The ability to map vegetation and in particular individual trees is a key component in forest management and long-term forest monitoring. Here we present a novel approach for mapping individual tree species based on ensemble modeling, i.e. combining the projections of several modeling techniques in order to reduce uncertainty. Using statistical modeling in conjunction with high-resolution aerial imagery (50 cm spatial resolution) and topo-climatic variables (5 m spatial resolution), we map the distributions of six major tree species (3 broadleaf and 3 conifers) in a study area of North-Eastern Switzerland. We also compare the relative predictive power of both topo-climatic and remote-sensing variables for mapping the spatial tree patterns and assess the importance of calibration data quality on model performance. We evaluate our projections using cross-validation as well as with independent data. Overall, the evaluations that we obtain for our vegetation maps are in line with, or higher than, those in similar studies. Depending on the considered tree species, 47.8–85.6% of our samples were correctly predicted, and we obtain an overall CCR (correct classification rate) of 0.72 and a Cohen’s kappa of 0.65. Comparing the predictive power of the different modeling techniques, we find that ensemble modeling (i.e. combining the projections of different individual modeling techniques) generally performs better than individual modeling techniques. <s> BIB016 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> Species distributions models (SDMs) are commonly used to assess potential species’ range shifts or extinction risk under climate change. It has been suggested that the use of ensemble forecasts, where a variety of model algorithms are used to generate consensus predictions, are preferred to individual SDMs by avoiding bias or prediction error inherent in a single modeling approach. Whereas several studies have assessed the performance of ensemble predictions using cross-validation or data-partitioning approaches, few studies have assessed the predictive accuracy of ensemble forecasts under climate change by using temporally independent model validation data. We used five SDM approaches to develop consensus forecasts of distributions of 145 vascular plant species from California in the 1930s and tested their projections against current distributions, a span of approximately 75 years. When evaluated with a portion of the model training data, consensus forecasts were highly accurate with an average AUC value of 0.97. False positive and false negative error rates were also low, exhibiting similar performance to random forest models. However, when evaluated with temporally independent data, the accuracy of consensus forecasts was similar to that of generalized linear and generalized additive models, with an average AUC value of 0.83. Our results suggest that the high levels of predictive accuracy exhibited by consensus forecasts when using data partitioning approaches may not reflect their performance when predicting temporally independent data. We contend that consensus forecasts may not represent the best approach for predicting species distributions under future climatic change, as they may not provide superior predictive accuracy in novel temporal domains compared to traditional modeling approaches that more readily lend themselves to ecological interpretation of model structure. <s> BIB017 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> AIM ::: Conservation planning exercises increasingly rely on species distributions predicted either from one particular statistical model or, more recently, from an ensemble of models (i.e. ensemble forecasting). However, it has not yet been explored how different ways of summarizing ensemble predictions affect conservation planning outcomes. We evaluate these effects and compare commonplace consensus methods, applied before the conservation prioritization phase, to a novel method that applies consensus after reserve selection. ::: ::: ::: LOCATION ::: Europe. ::: ::: ::: METHODS ::: We used an ensemble of predicted distributions of 146 Western Palaearctic bird species in alternative ways: four different consensus methods, as well as distributions discounted with variability, were used to produce inputs for spatial conservation prioritization. In addition, we developed and tested a novel method, in which we built 100 datasets by sampling the ensemble of predicted distributions, ran a conservation prioritization analysis on each of them and averaged the resulting priority ranks. We evaluated the conservation outcome against three controls: (i) a null control, based on random ranking of cells; (2) the reference solution, based on an expert-refined dataset; and (3) the independent solution, based on an independent dataset. ::: ::: ::: RESULTS ::: Networks based on predicted distributions were more representative of rare species than randomly selected networks. Alternative methods to summarize ensemble predictions differed in representativeness of resulting reserve networks. Our novel method resulted in better representation of rare species than pre-selection consensus methods. ::: ::: ::: MAIN CONCLUSIONS ::: Retaining information about the variation in the predicted distributions throughout the conservation prioritization seems to provide better results than summarizing the predictions before conservation prioritization. Our results highlight the need to understand and consider model-based uncertainty when using predicted distribution data in conservation prioritization. <s> BIB018 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> ABSTRACTGeneralized degrees of freedom (GDF), as defined by Ye (1998 JASA 93:120–131), represent the sensitivity of model fits to perturbations of the data. Such GDF can be computed for any statist... <s> BIB019 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> Aim ::: ::: Species distribution models (SDMs) are commonly used to determine threats to biodiversity and opportunities under climate change. Despite SDMs being based on the assumption of complete knowledge of the climate space of the modelled species, truncated occurrence datasets (and hence truncated climate spaces) such as national inventories are often employed. This may lead to prediction errors, which have been proposed to stem from: (1) the degree of climate space truncation and/or (2) instability of the modelling algorithms. Our aim was to explore the potential causes of prediction errors in SDMs using truncated training datasets. ::: ::: ::: ::: Location ::: ::: Europe 11° W–32° E, 34°–72° N. ::: ::: ::: ::: Methods ::: ::: SDMs employing commonly used bioclimatic variables were applied to seven forest tree species. We created two model training datasets covering: (1) Germany only (significantly truncated climate space) and (2) Europe (minimally truncated climate space). Differences between the climate space represented by Germany-only and European data were measured on two-dimensional climate spaces obtained through principal component analysis of the bioclimatic variables. Seven SDM algorithms were run, and the stability of the response function and variable selection for each species and model type were analysed. ::: ::: ::: ::: Results ::: ::: The degree of climate space truncation was less important for model performance than the instability of model algorithms and indiscriminate variable selection. The latter led to irrelevant relationships of species occurrence with bioclimatic variables. These instabilities caused pronounced prediction errors. ::: ::: ::: ::: Main conclusions ::: ::: Our results strongly suggest that erroneous model predictions stem from instability and ecological irrelevance of the statistical functions relating the probability of a species' occurrence to bioclimatic variables, compounded by a lack of consistency in variable selection. Models displaying these characteristics showed lower overall performance when trained with truncated datasets. Further, commonly used ensemble approaches do not compensate for the shortfalls of individual models. Detailed model-by-model and species-by-species analysis of response functions and variable importance is recommended. <s> BIB020 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> Thank you very much for downloading principles of forecasting a handbook for researchers and practitioners. As you may know, people have look numerous times for their chosen books like this principles of forecasting a handbook for researchers and practitioners, but end up in harmful downloads. Rather than enjoying a good book with a cup of coffee in the afternoon, instead they juggled with some malicious bugs inside their laptop. <s> BIB021 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> Distribution models should take into account the different limiting factors that simultaneously influence species ranges. Species distribution models built with different explanatory variables can be combined into more comprehensive ones, but the resulting models should maximize complementarity and avoid redundancy. Our aim was to compare the different methods available for combining species distribution models. We modelled 19 threatened vertebrate species in mainland Spain, producing models according to three individual explanatory factors: spatial constraints, topography and climate, and human influence. We used five approaches for model combination: Bayesian inference, Akaike weight averaging, stepwise variable selection, updating, and fuzzy logic. We compared the performance of these approaches by assessing different aspects of their classification and discrimination capacity. We demonstrated that different approaches to model combination give rise to disparities in the model outputs. Bayesian integration was systematically affected by an error in the equations that are habitually used in distribution modelling. Akaike weights produced models that were driven by the best single factor and therefore failed at combining the models effectively. The updating and the stepwise approaches shared recalibration as the basic concept for model combination, were very similar in their performance, and showed the highest sensitivity and discrimination capacity. The fuzzy-logic approach yielded models with the highest classification capacity according to Cohen's kappa. In conclusion: 1) Bayesian integration, employing the currently used equation, and the Akaike weight procedure should be avoided; 2) the updating and stepwise approaches can be considered minor variants of the same recalibrating approach; and 3) there is a trade-off between this recalibrating approach, which has the highest sensitivity, and fuzzy logic, which has the highest overall classification capacity. Recalibration is better if unfavourable conditions in one environmental factor may be counterbalanced with favourable conditions in a different factor, otherwise fuzzy logic is better. <s> BIB022 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Tactical approaches to computing model weights <s> A multi-model approach to predict phytoplankton biomass and composition was performed in a eutrophic Uruguayan shallow lake which is the second drinking water source of the country. We combined statistical (spectral analysis and Machine learning techniques) and physically based models to generate, for the first time in this system, a predictive tool of phytoplankton biomass (chlorophyll-a) and composition (morphology-based functional groups). The results, based on a 11-year time series, revealed two alternating phases in the temporal dynamics of phytoplankton biomass. One phase is characterized by high inorganic turbidity and low phytoplankton biomass, and the other by low inorganic turbidity and variable (low and high) phytoplankton biomass. A threshold of turbidity (29 TNU), above which phytoplankton remains with low biomass (<15–20 ug/l) was established. The periods of high turbidity, which in total cover 30% of the time series, start abruptly and are related to external forcing. Meteorological conditions associated with the beginning of these periods were modeled through a regression tree analysis. These conditions consist of moderate to high wind intensities from the SW direction, in some cases combined with high antecedent precipitation or low water level. The results from the physically-based modeling indicated that the long decaying time-scale of turbidity and intermediate resuspension events could explain the prolonged length of the high turbidity periods (∼1.5 years). Random Forests models for the prediction of phytoplankton biomass and composition in periods of low turbidity resulted in a proportion of explained variance and a classification error over a test sample of 0.46 and 0.34 respectively. Turbidity, conductivity, temperature and water level were within the most important model predictors. The development and improvement of this type of modeling is needed to provide management tools to water managers in the current water supply situation. <s> BIB023
Methods covered in this section share the "tactical" goal of choosing weights to optimize prediction (e.g., reduce prediction error), without a specific reference to a statistical theory such as Bayesian inference or information theory. The most straightforward approach in this area is to make the averaging weight dependent on an estimate of the predictive error of each model, usually obtained by cross-validation. Cross-validation approximates a model's predictive performance on new data by predicting to a hold-out part of the data (typically between 5 and 20 folds, down to leaveone-out cross-validation, which omits each single data point in turn). The fit to the hold-out can be quantified in different ways. If the data can be reasonably well described by a specific distribution with log-likelihood function ' (even if the model algorithm itself is non-parametric), the log-likelihood of the data in the k folds can be computed and summed BIB013  : where the index ½Ài indicates that the data y ½i in fold i were not used for fitting model m and estimating model parameterŝ h m y ½Ài . It can be shown that leave-one-out cross-validation loglikelihood is asymptotically equivalent to AIC and thus KLdistance (Stone1977), albeit at a higher computational cost. Hence, computing model weights w m CV BIB019 ) creates a weighting scheme very similar to AIC-weights, which implicitly penalizes overfitting. Other measures of model fit to the hold-out folds have been used, largely as ad hoc proxies for a likelihood function (e.g., in likelihood-free models): pseudo-R 2 (e.g., Nagelkerke 1991, Nakagawa and BIB014 , area under the ROC curve (AUC: BIB009 BIB015 BIB020 ), or True Skill Statistic (Diniz-Filho et al. 2009 BIB011 BIB016 BIB018 . In these cases, weights were computed by substituting ' CV in Eq. 13 by the respective measure, or given a value of 1/S for a somewhat arbitrarily defined subset of S (out of M) models, e.g., those above a threshold considered minimal satisfactory performance BIB008 BIB017 BIB015 . Largely ignored by the ecological literature are two other non-parametric approaches to compute model weights: stacking and jackknife model averaging (see Appendix S1.4 for discussion of averaging within machine-learning algorithms). Both are cross-validation based, but unlike simple cross-validation weights, which are based on the performace of each contributing model on hold-out data, stacking and jacknife model averaging explicitly optimize weights to reduce the error of the average on hold-out data. Stacking (Wolpert 1992  BIB012 finds the optimized model weights to reduce prediction error (or maximize likelihood) on a test hold-out of size H. This is, for RMSE and likelihood, respectively:  is the prediction of model m, fitted without using data i, to data i. This procedure is repeated many times, each time yielding a vector of optimized model weights, w m , which are then averaged across repetitions and rescaled to sum to 1.  extend this approach also to Bayesian models to provide a clear prediction-error minimizing goal.  and BIB006 report stacking to generally outperform the cross-validation approach from two paragraphs earlier, and Bayesian model averaging, respectively (see also in Case Studies and Appendix S5). In Jackknife Model Averaging (JMA; Hansen and Racine 2012), each data point is omitted in turn from fitting and then predicted to (thus actually a leave-one-out cross-validation rather than a "jackknife"). Then, weights are optimized so as to minimize RMSE (or maximize likelihood) between the observed and the fitted value across all N "jackknife" samples. The optimization function is the same as for stacking, except that H ¼ N. Thus, in stacking, weights are optimized once for each run, while for the jackknife only one optimization over all N leave-one-out-cross-validations is required (further details and examples with R-code are given in Appendix S1.5.6). The forecasting (i.e., time predictions) literature (reviewed in BIB021 BIB004  ) offers two further approaches. BIB001 's minimal variance approach attributes more weight to models with low-variance predictions. More precisely, it uses the inverse of the variance-covariance matrix of predictions, R À1 , to compute model weights. In the multi-model generalisation BIB002 the weights vector w is calculated as: where 1 is an M-length vector of ones. This is the analytical solution of Eq. 5, assuming no bias and ignoring the problem that weights are random variates, under the weights-sum-toone constraint. Eq. 14 does not ensure all-positive weights, nor is it obvious how to estimate R. One option (used in our case studies) is to base R on the deviation from a prediction to test data in lieu of measure of past performance (following recommendation of BIB001 . Finally, Garthwaite and Mubwandarikwa (2010) devised a rarely used method, called the "cos-squared weighting scheme," designed to adjust for correlation in predictions by different models. It was motivated by (1) giving lower weight November 2018 MODEL AVERAGING IN ECOLOGY 495 to models highly correlated with others (thereby reducing the prediction variance contributed through covariances in Eq. 5), (2) division of weights when a new, near-identical model prediction is added to the set, and (3) reducing all weights when more models are added to the set. Weights are computed as proportional to the amount of rotation the predictions would require to make them orthogonal in prediction space, hence the trigonometric name of their approach. Modelling model weights.-So far, weights were always constant. However, one might also consider making weights dependent on other variables. This approach, which we term "model-based model combinations" (MBMC, also called "superensemble modeling") was first proposed by BIB003 . Here a statistical model f is used to combine the predictions from different models, as if they were predictors in a regression: Fig. 5A ). The regression-type model f can be of any type, such as a linear model or a neural network. We call this regression the "supra-model" in order to distinguish between different modelling levels. A very simple supra-model would compute the median of predictions for each point X i (e.g., BIB009 ). Different models are used in the "average" without requiring any additional parameter estimation. Median predictions imply varying weights, as the one or two models considered for computing the median may change between different X i . An ideal model combination could switch, or gently transition, between models (such as manually constructed by BIB023 . Since the predictions are combined more or less freely in model-based model combinations to yield the best possible fit to the observed data, MBMC should be superior to any constant-weight-per-model approach (see Fig. 5B ), as was indeed found by BIB010 . This advantage comes with a severe drawback: a high proclivity to overfitting, as we fit the same data twice (once to each model, then again to their prediction regression). This does not seem to be widely recognized as a problem (despite being a key message of  ), as most studies we found incorrectly cross-validate the supra-model only, not the entire workflow (if at all; e.g., BIB005 BIB007 BIB010 BIB022 ; but see  . To correctly cross-validate MBMCs, one has to produce hold-outs before fitting the contributing models, and evaluate the MBMC prediction on this hold-out (Fig. 5 , Appendix S5.9 and case studies;  . [Correction added 9 July 2018 after online publication. The work of  was erroneously included in the list of studies. This error was due to a misunderstanding of the method section in their publication.] Note that supra-models may differ substantially in their ability to harness the contributing models. As it is a yet fairly unexplored field in model averaging, analysts are advised to try different supra-model types (Fig. 5) .