Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Information-theoretic model weights <s> We argue that model selection uncertainty should be fully incorporated into statistical inference whenever estimation is sensitive to model choice and that choice is made with reference to the data. We consider different philosophies for achieving this goal and suggest strategies for data analysis. We illustrate our methods through three examples. The first is a Poisson regression of bird counts in which a choice is to be made between inclusion of one or both of two covariates. The second is a line transect data set for which different models yield substantially different estimates of abundance. The third is a simulated example in which truth is known. <s> BIB001 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Information-theoretic model weights <s> Adaptation over different procedures is of practical importance. Different procedures perform well under different conditions. In many practical situations, it is rather hard to assess which conditions are (approximately) satisfied so as to identify the best procedure for the data at hand. Thus automatic adaptation over various scenarios is desirable. A practically feasible method, named adaptive regression by mixing (ARM), is proposed to convexly combine general candidate regression procedures. Under mild conditions, the resulting estimator is theoretically shown to perform optimally in rates of convergence without knowing which of the original procedures work the best. Simulations are conducted in several settings, including comparing a parametric model with nonparametric alternatives, comparing a neural network with a projection pursuit in multidimensional regression, and combining bandwidths in kernel regression. The results clearly support the theoretical property of ARM. The ARM algorithm assigns we... <s> BIB002 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Information-theoretic model weights <s> Ecologists are increasingly applying model selection to their data analyses, primarily to compare regression models. Model selection can also be used to compare mechanistic models derived from ecological theory, thereby providing a formal framework for testing the theory. The Akaike Information Criterion (AIC) is the most commonly adopted criterion used to compare models; however, its performance in general is not very well known. The best model according to AIC has the smallest expected Kullback-Leibler (K-L) distance, which is an information-theoretic measure of the difference between a model and the truth. I review the theory behind AIC and demonstrate how it can be used to test ecological theory by considering two example studies of foraging, motivated by simple foraging theory. I present plausible truths for the two studies, and models that can be fit to the foraging data. K-L distances are calculated for simulated studies, which provide an appropriate test of AIC. Results support the use of a commonly adopted rule of thumb for selecting models based on AIC differences. However, AICc, a corrected version of AIC commonly used to reduce model selection bias, showed no clear improvement, and model averaging, a technique to reduce model prediction bias, gave mixed results. <s> BIB003 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Information-theoretic model weights <s> Model-combining (i.e., mixing) methods have been proposed in recent years to deal with uncertainty in model selection. Even though advantages of model combining over model selection have been demonstrated in simulations and data examples, it is still unclear to a large extent when model combining should be preferred. In this work, first we propose an instability measure to capture the uncertainty of model selection in estimation, called perturbation instability in estimation (PIE), based on perturbation of the sample. We demonstrate that estimators from model selection can have large PIE values and that model combining substantially reduces the instability for such cases. Second, we propose a model combining method, adaptive regression by mixing with model screening (ARMS), and derive a theoretical property. In ARMS, a screening step is taken to narrow down the list of candidate models before combining, which not only saves computing time, but also can improve estimation accuracy. Third, we compare ARMS w... <s> BIB004 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Information-theoretic model weights <s> Statistical thinking in wildlife biology and ecology has been profoundly influenced by the introduction of AIC (Akaike's information criterion) as a tool for model selection and as a basis for model averaging. In this paper, we advocate the Bayesian paradigm as a broader framework for multimodel inference, one in which model averaging and model selection are naturally linked, and in which the performance of AIC-based tools is naturally evaluated. Prior model weights implicitly associated with the use of AIC are seen to highly favor complex models: in some cases, all but the most highly parameterized models in the model set are virtually ignored a priori. We suggest the usefulness of the weighted BIC (Bayesian information criterion) as a computationally simple alternative to AIC, based on explicit selection of prior model probabilities rather than acceptance of default priors associated with AIC. We note, however, that both procedures are only approximate to the use of exact Bayes factors. We discuss and illustrate technical difficulties associated with Bayes factors, and suggest approaches to avoiding these difficulties in the context of model selection for a logistic regression. Our example highlights the predisposition of AIC weighting to favor complex models and suggests a need for caution in using the BIC for computing approximate posterior model weights. <s> BIB005 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Information-theoretic model weights <s> Statistical methods emphasizing formal hypothesis testing have dominated the analyses used by ecologists to gain insight from data. Here, we review alternatives to hypothesis testing including techniques for parameter estimation and model selection using likelihood and Bayesian techniques. These methods emphasize evaluation of weight of evidence for multiple hypotheses, multimodel inference, and use of prior information in analysis. We provide a tutorial for maximum likelihood estimation of model parameters and model selection using information theoretics, including a brief treatment of procedures for model comparison, model averaging, and use of data from multiple sources. We discuss the advantages of likelihood estimation, Bayesian analysis, and meta-analysis as ways to accumulate understanding across multiple studies. These statistical methods hold promise for new insight in ecology by encouraging thoughtful model building as part of inquiry, providing a unified framework for the empirical analysis of theoretical models, and by facilitating the formal accumulation of evidence bearing on fundamental questions. <s> BIB006 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Information-theoretic model weights <s> This paper considers the problem of selection of weights for averaging across least squares estimates obtained from a set of models. Existing model average methods are based on exponential Akaike information criterion (AIC) and Bayesian information criterion (BIC) weights. In distinction, this paper proposes selecting the weights by minimizing a Mallows criterion, the latter an estimate of the average squared error from the model average fit. We show that our new Mallows model average (MMA) estimator is asymptotically optimal in the sense of achieving the lowest possible squared error in a class of discrete model average estimators. In a simulation experiment we show that the MMA estimator compares favorably with those based on AIC and BIC weights. The proof of the main result is an application of the work of Li (1987). Copyright The Econometric Society 2007. <s> BIB007 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Information-theoretic model weights <s> Summary In medical research, there is great interest in developing methods for combining biomarkers. We argue that selection of markers should also be considered in the process. Traditional model/variable selection procedures ignore the underlying uncertainty after model selection. In this work, we propose a novel model-combining algorithm for classification in biomarker studies. It works by considering weighted combinations of various logistic regression models; five different weighting schemes are considered in the article. The weights and algorithm are justified using decision theory and risk-bound results. Simulation studies are performed to assess the finite-sample properties of the proposed model-combining method. It is illustrated with an application to data from an immunohistochemical study in prostate cancer. <s> BIB008 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Information-theoretic model weights <s> Behavioural ecologists often study complex systems in which multiple hypotheses could be proposed to explain observed phenomena. For some systems, simple controlled experiments can be employed to reveal part of the complexity; often, however, observational studies that incorporate a multitude of causal factors may be the only (or preferred) avenue of study. We assess the value of recently advocated approaches to inference in both contexts. Specifically, we examine the use of information theoretic (IT) model selection using Akaike’s information criterion (AIC). We find that, for simple analyses, the advantages of switching to an IT-AIC approach are likely to be slight, especially given recent emphasis on biological rather than statistical significance. By contrast, the model selection approach embodied by IT approaches offers significant advantages when applied to problems of more complex causality. Model averaging is an intuitively appealing extension to model selection. However, we were unable to demonstrate consistent improvements in prediction accuracy when using model averaging with IT-AIC; our equivocal results suggest that more research is needed on its utility. We illustrate our arguments with worked examples from behavioural experiments. <s> BIB009 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Information-theoretic model weights <s> Model averaging or combining is often considered as an alternative to model selection. Frequentist Model Averaging (FMA) is considered extensively and strategies for the application of FMA methods in the presence of missing data based on two distinct approaches are presented. The first approach combines estimates from a set of appropriate models which are weighted by scores of a missing data adjusted criterion developed in the recent literature of model selection. The second approach averages over the estimates of a set of models with weights based on conventional model selection criteria but with the missing data replaced by imputed values prior to estimating the models. For this purpose three easy-to-use imputation methods that have been programmed in currently available statistical software are considered, and a simple recursive algorithm is further adapted to implement a generalized regression imputation in a way such that the missing values are predicted successively. The latter algorithm is found to be quite useful when one is confronted with two or more missing values simultaneously in a given row of observations. Focusing on a binary logistic regression model, the properties of the FMA estimators resulting from these strategies are explored by means of a Monte Carlo study. The results show that in many situations, averaging after imputation is preferred to averaging using weights that adjust for the missing data, and model average estimators often provide better estimates than those resulting from any single model. As an illustration, the proposed methods are applied to a dataset from a study of Duchenne muscular dystrophy detection. <s> BIB010 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Information-theoretic model weights <s> Statistical modeling is a powerful tool for developing and testing theories by way of causal explanation, prediction, and description. In many disciplines there is near-exclusive use of statistical modeling for causal ex- planation and the assumption that models with high explanatory power are inherently of high predictive power. Conflation between explanation and pre- diction is common, yet the distinction must be understood for progressing scientific knowledge. While this distinction has been recognized in the phi- losophy of science, the statistical literature lacks a thorough discussion of the many differences that arise in the process of modeling for an explanatory ver- sus a predictive goal. The purpose of this article is to clarify the distinction between explanatory and predictive modeling, to discuss its sources, and to reveal the practical implications of the distinction to each step in the model- ing process. <s> BIB011 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Information-theoretic model weights <s> Aim ::: ::: We developed a set of statistical models to improve spatial estimates of mangrove aboveground biomass (AGB) based on the environmental signature hypothesis (ESH). We hypothesized that higher tidal amplitudes, river discharge, temperature, direct rainfall and decreased potential evapotranspiration explain observed high mangrove AGB. ::: ::: ::: ::: Location ::: ::: Neotropics and a small portion of the Nearctic region. ::: ::: ::: ::: Methods ::: ::: A universal forest model based on site-level forest structure statistics was validated to spatially interpolate estimates of mangrove biomass at different locations. Linear models were then used to predict mangrove AGB across the Neotropics. ::: ::: ::: ::: Results ::: ::: The universal forest site-level model was effective in estimating mangrove AGB using pre-existing mangrove forest structure inventories to validate the model. We confirmed our hypothesis that at continental scales higher tidal amplitudes contributed to high forest biomass associated with high temperature and rainfall, and low potential evapotranspiration. Our model explained 20% of the spatial variability in mangrove AGB, with values ranging from 16.6 to 627.0 t ha−1 (mean, 88.7 t ha−1). Our findings show that mangrove AGB has been overestimated by 25–50% in the Neotropics, underscoring a commensurate bias in current published global estimates using site-level information. ::: ::: ::: ::: Main conclusions ::: ::: Our analysis show how the ESH significantly explains spatial variability in mangrove AGB at hemispheric scales. This finding is critical to improve and explain site-level estimates of mangrove AGB that are currently used to determine the relative contribution of mangrove wetlands to global carbon budgets. Due to the lack of a conceptual framework explicitly linking environmental drivers and mangrove AGB values during model validation, previous works have significantly overestimated mangrove AGB; our novel approach improved these assessments. In addition, our framework can potentially be applied to other forest-dominated ecosystems by allowing the retrieval of extensive databases at local levels to generate more robust statistical predictive models to estimate continental-scale biomass values. <s> BIB012
In the information-theoretic perspective, models closer to the data, as measured by the Kullback-Leibler divergence, should receive more weight than those further away. There are several approximations of the KL-divergence, most famously Akaike's Information Criterion (AIC; Akaike 1973, Burnham and  . AIC and related indices can be computed only for likelihood-based models with known number of parameters (p m ), restricting the information-theoretic approach to GLM-like models (including GAM): where ' m is the log-likelihood of model m. In the ecological literature, AIC (and its sample-size corrected version AIC c , and its adaptations to quasi-likelihood models such as QIC; Pan 2001, Claeskens and  is by far the most common approach to determine model weights (for recent examples see, e.g.,  BIB012 , despite the fact that the reasoning behind this choice is not entirely clear. AIC-weights (Eq. 11) have been interpreted as Bayesian model probabilities Anderson 2002:75, Link and BIB005 , assuming a specific, model complexity and sample size-dependent, "savvy prior"  302, see also Hooten and Hobbs 2015:16 , for reformulation as regularization prior). An alternative interpretation is the proportion of times a model would be chosen as the best model under repeated sampling BIB006 , but such an interpretation is contentious BIB003   . In an anecdotal comparison, Burnham and Anderson (2002:178) showed that AIC weights are substantially different from bootstrapped model weights. The latter were proposed by BIB001 and represent the proportion of bootstraps a model is performing best in terms of AIC: see case study 1 below. In simulations, AIC weights did not reliably identify the model with the known lowest KL-divergence or prediction error BIB003 BIB009 . Instead, Mallows' model averaging (MMA) has been shown to yield the lowest mean squared error for linear models BIB007 BIB010 . Mallows' C p penalizes model complexity equivalent to À2' m À n þ 2p m (for n data points; rather than AIC's À2' m þ 2p m , Eq. 11). Schwartz' Bayesian Information Criterion was derived to find the most probable model given the data  BIB011 , equivalent to having the largest Bayes factor (see previous section). BIC uses logðnÞ rather than AIC's "2" as penalization factor for model complexity (Appendix S1.5.3). A particularly noteworthy modification of the AIC exist, where the model fit is assessed with respect to a focal predictor value, e.g., a specific age or temperature range, yielding the Focussed Information Criterion (FIC;  . We are not aware of a systematic simulation study comparing the performance of these model averaging weights, but AIC's dominance should not indicate its superiority (see also case study 1 below). The weighting procedure can additionally be wrapped into a cross-validation and model pre-selection, which leads to the ARMS-procedure (Adaptive Regression by Mixing with model Screening; BIB002 BIB004 BIB008 . We shall not present details on ARMS here (for cross-validation see next section), because we regard model pre-selection as an unresolved issue (see 494 DORMANN Vol. 88, No. 4 REVIEW Validation-based weighting or validation-based pre-selection of models).