Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Process models are no different <s> Specification of an appropriate model is critical to valid statistical inference. Given the true model for the data is unknown, the goal of model selection is to select a plausible approximating model that balances model bias and sampling variance. Model selection based on information criteria such as AIC or its variant AIC c , or criteria like CAIC, has proven useful in a variety of contexts including the analysis of open-population capture-recapture data. These criteria have not been intensively evaluated for closed-population capture-recapture models, which are integer parameter models used to estimate population size (N), and there is concern that they will not perform well. To address this concern, we evaluated AIC, AIC c , and CAIC model selection for closed-population capture-recapture models by empirically assessing the quality of inference for the population size parameter N. We found that AIC-, AIC c -, and CAIC-selected models had smaller relative mean squared errors than randomly selected models, but that confidence interval coverage on N was poor unless unconditional variance estimates (which incorporate model uncertainty) were used to compute confidence intervals. Overall, AIC and AIC c outperformed CAIC, and are preferred to CAIC for selection among the closed-population capture-recapture models we investigated. A model averaging approach to estimation, using AIC, AIC c , or CAIC to estimate weights, was also investigated and proved superior to estimation using AIC-, AIC c -, or CAIC-selected models. Our results suggested that, for model averaging, AIC or AIC c should be favored over CAIC for estimating weights. <s> BIB001 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Process models are no different <s> A method for improving weather and climate forecast skill has been developed. It is called a superensemble, and it arose from a study of the statistical properties of a low-order spectral model. Multiple regression was used to determine coefficients from multimodel forecasts and observations. The coefficients were then used in the superensemble technique. The superensemble was shown to outperform all model forecasts for multiseasonal, medium-range weather and hurricane forecasts. In addition, the superensemble was shown to have higher skill than forecasts based solely on ensemble averaging. <s> BIB002 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Process models are no different <s> The ‘‘reliability ensemble averaging’’ (REA) method for calculating average, uncertainty range, and a measure of reliability of simulated climate changes at the subcontinental scale from ensembles of different atmosphere‐ ocean general circulation model (AOGCM) simulations is introduced. The method takes into account two ‘‘reliability criteria’’: the performance of the model in reproducing present-day climate (‘‘model performance’’ criterion) and the convergence of the simulated changes across models (‘‘model convergence’’ criterion). The REA method is applied to mean seasonal temperature and precipitation changes for the late decades of the twenty-first century, over 22 land regions of the world, as simulated by a recent set of nine AOGCM experiments for two anthropogenic emission scenarios (the A2 and B2 scenarios of the Intergovernmental Panel for Climate Change). In the A2 scenario the REA average regional temperature changes vary between about 2 and 7 K across regions and they are all outside the estimated natural variability. The uncertainty range around the REA average change as measured by 6 the REA root-mean-square difference (rmsd) varies between 1 and 4 K across regions and the reliability is mostly between 0.2 and 0.8 (on a scale from 0 to 1). For precipitation, about half of the regional REA average changes, both positive and negative, are outside the estimated natural variability and they vary between about 225% and 130% (in units of percent of present-day precipitation). The uncertainty range around these changes (6 rmsd) varies mostly between about 10% and 30% and the corresponding reliability varies widely across regions. The simulated changes for the B2 scenario show a high level of coherency with those for the A2 scenario. Compared to simpler approaches, the REA method allows a reduction of the uncertainty range in the simulated changes by minimizing the influence of ‘‘outlier’’ or poorly performing models. The method also produces a quantitative measure of reliability that shows that both criteria need to be met by the simulations in order to increase the overall reliability of the simulated changes. <s> BIB003 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Process models are no different <s> Models represent our primary method for integration of small-scale, process- level phenomena into a comprehensive description of forest-stand or ecosystem function. They also represent a key method for testing hypotheses about the response of forest ecosystems to multiple changing environmental conditions. This paper describes the eval- uation of 13 stand-level models varying in their spatial, mechanistic, and temporal com- plexity for their ability to capture intra- and interannual components of the water and carbon cycle for an upland, oak-dominated forest of eastern Tennessee. Comparisons between model simulations and observations were conducted for hourly, daily, and annual time steps. Data for the comparisons were obtained from a wide range of methods including: eddy covariance, sapflow, chamber-based soil respiration, biometric estimates of stand-level net primary production and growth, and soil water content by time or frequency domain reflectometry. Response surfaces of carbon and water flux as a function of environmental drivers, and a variety of goodness-of-fit statistics (bias, absolute bias, and model efficiency) were used to judge model performance. A single model did not consistently perform the best at all time steps or for all variables considered. Intermodel comparisons showed good agreement for water cycle fluxes, but considerable disagreement among models for predicted carbon fluxes. The mean of all model outputs, however, was nearly always the best fit to the observations. Not surprisingly, models missing key forest components or processes, such as roots or modeled soil water content, were unable to provide accurate predictions of ecosystem responses to short-term drought phenomenon. Nevertheless, an inability to correctly capture short-term physiolog- ical processes under drought was not necessarily an indicator of poor annual water and carbon budget simulations. This is possible because droughts in the subject ecosystem were of short duration and therefore had a small cumulative impact. Models using hourly time steps and detailed mechanistic processes, and having a realistic spatial representation of the forest ecosystem provided the best predictions of observed data. Predictive ability of all models deteriorated under drought conditions, suggesting that further work is needed to evaluate and improve ecosystem model performance under unusual conditions, such as drought, that are a common focus of environmental change discussions. <s> BIB004 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Process models are no different <s> Reducing overfishing and recovering overfished fish stocks is a challenging and important global problem. Rebuilding targets are essential ingredients for guiding overfished stocks to recovery. Having robust estimates of re- building targets is likely a necessary condition for effective long-term management of fishery resources. In this paper, we show how Bayesian model averaging can be applied to estimate rebuilding targets under alternative hypotheses about stock-recruitment dynamics. Using alternative hypotheses about stock-recruitment dynamics and environmental variation is important because using only a single hypothesis can lead to substantially different reference points and policy implications. The alternative hypotheses are augmented with prior information collected from meta-analyses of stock-recruitment data to construct a set of age-structured production models. We illustrate our approach using three overfished New England groundfish stocks. We fit alternative model likelihoods to observed data using Bayesian infer- ence techniques. The Schwarz goodness-of-fit criterion was used to calculate model probabilities. Bayesian model aver- aging was used to estimate rebuilding targets that were robust to model selection uncertainty. Model-averaged estimates suggested that rebuilding targets for overfished stocks can be reasonably well determined if adequate prior information on stock productivity is available. Nonetheless, results had wide confidence intervals that reflected the underlying un- certainty in rebuilding targets. <s> BIB005 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Process models are no different <s> Bayesian estimation methods are used to fit an age- and sex-structured population model to available data on abundance and stag eproportions (i.e. calves/mature animals in the population) for the Bering-Chukchi-Beaufort Seas stock of bowhead whales ( Balaena mysticetus). The analyses consider three alternative population modelling approaches: (1) modelling the entire population trajectory from 1848, using the ‘backwards’ method where the trajectory is back-calculated based on assigning a prior distribution to recent abundance ; (2) modelling only the recent population trajectory, using the ‘forwards from recent abundance’ method, where the population is projected forwards from a recent year and the abundance in that year is not assumed to be at carrying capacity; and (3) a version of (2) that ignores density-dependence. The ‘backwards’ method leads to more precise estimates of depletion level. In contrast, the ‘forwards from r ecent abundance’ method provides an alternative way of calculating catch-related quantities without having to assume that the catch record is known exactly from 1848 to the present, or having to assume that carrying capacity has not changed since 1848. Not only are all three models able to fit the abundance data well, but each is also able to remain consistent with available estimates of adult surviv al and age of sexual maturity. Sensitivity to the stage-proportion data and the prior distributions for the life history parameters indicates that use of the 1985 stage-proportion data has the greatest effect on the results, and that those data are less consistent with data on trends in abundance and age of sexual maturity. The analyses indicate that the population has approximately doubled in size since 1978, and the ‘backwards’ analyses suggest that the population may be approaching carrying capacity, although there is no obvious sign in the data that the popula tion growth rate has slowed. Bayes factors are calculated to compare model fits to the data. However, there is no evidence for selecting on e model over another, and furthermore, the models considered in this study result in different posterior distributions for quantities of int erest to management. Posterior model probabilities are therefore calculated and used as weights to construct Bayesian model-averaged pos terior distributions for outputs shared among models to take this ambiguity into account. This study represents the first attempt to e xplicitly quantify model uncertainty when conducting a stock assessment of bowhead whales. <s> BIB006 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Process models are no different <s> Abstract Model selection based on information theory is a relatively new paradigm in biological sciences with several advantages over the classical approaches. The aim of the present study was to apply information theory in the area of modelling fish growth and to show how model selection uncertainty may be taken into account when estimating growth parameters. The methodology was applied for length–age data of four species of fish, taken from the literature. Five-candidate models were fitted to each dataset: von Bertalanffy growth model (VBGM), generalized VBGM, Gompertz growth model, Schnute–Richards growth model, and logistic. In each case, the ‘best’ model was selected by minimizing the small-sample, bias-corrected form of the Akaike information criterion (AIC). To quantify the plausibility of each model, given the data and the set of five models, the ‘Akaike weight’ w i of each model was calculated. The average model was estimated for each case based on w i . Following a multi-model inference (MMI) approach, the model-averaged asymptotic length L ¯ ∞ for each species was estimated, using all five models, by model-averaging estimations of L∞ and weighting the prediction of each model by w i . In the examples of this study, model selection uncertainty caused a magnification of the standard error of the asymptotic length of the best model (up to 3.9 times) and thus in all four cases estimating L∞ from just the best model would have caused overestimation of precision of the asymptotic length. The VBGM, when used for inference, without being the best model, could cause biased point estimation and false evaluation of precision. Model selection uncertainty should not be ignored even if VBGM is the best model. Multi-model inference by model-averaging, based on Akaike weights, is recommended for making robust parameter estimations and for dealing with uncertainty in model selection. <s> BIB007 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Process models are no different <s> Fisheries scientists habitually consider uncertainty in parameter values, but often neglect uncertainty about model structure, an issue of increasing importance as ecosystem models are devised to support the move to an ecosystem approach to fisheries (EAF). This paper sets out pragmatic approaches with which to account for uncertainties in model structure and we review current ways of dealing with this issue in fisheries and other disciplines. All involve considering a set of alternative models representing different structural assumptions, but differ in how those models are used. The models can be asked to identify bounds on possible outcomes, find management actions that will perform adequately irrespective of the true model, find management actions that best achieve one or more objectives given weights assigned to each model, or formalize hypotheses for evaluation through experimentation. Data availability is likely to limit the use of approaches that involve weighting alternative models in an ecosystem setting, and the cost of experimentation is likely to limit its use. Practical implementation of an EAF should therefore be based on management approaches that acknowledge the uncertainty inherent in model predictions and are robust to it. Model results must be presented in ways that represent the risks and trade-offs associated with alternative actions and the degree of uncertainty in predictions. This presentation should not disguise the fact that, in many cases, estimates of model uncertainty may be based on subjective criteria. The problem of model uncertainty is far from unique to fisheries, and a dialogue among fisheries modellers and modellers from other scientific communities will therefore be helpful. <s> BIB008 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Process models are no different <s> The common practice among researchers who study fish growth is to a priori adopt the von Bertalanffy growth model (VBGM), which is the most used and ubiquitous equation in the fisheries literature. However, in many cases VBGM is not supported by the data and many species seem to follow different growth trajectories. The information theory approach frees the researcher from the limiting concept that a ‘true’ growth model exists. Multi-model inference (MMI) based on information theory is proposed as a more robust alternative to study fish growth. The proposed methodology was applied to 133 sets of length-at-age data. Four candidate models were fitted to each data set: von Bertalanffy growth model (VBGM), Gompertz model, Logistic and the Power model; the three former assume asymptotic and the latter non-asymptotic growth. In each case, the ‘best’ model was selected by minimizing the small-sample, bias-corrected form of the Akaike information criterion (AICc). To quantify the plausibility of each model, the ‘Akaike weight’wi of each model was calculated. Following a MMI approach, the model averaged asymptotic length for each case was estimated, by model averaging estimations of interpreting Akaike weights as a posterior probability distribution over the set of candidate models. The VBGM was not selected as the best model in 65.4% of the cases. Most often VBGM was either strongly supported by the data (with no other substantially supported model) or had very low or no support by the data. The estimation of asymptotic length was greatly model dependent; as estimated by VBGM was in every case greater than that estimated by the Gompertz model, which in turn was always greater than that estimated by the Logistic model. The percentage underestimation of the standard error of , when ignoring model selection uncertainty, was on average 18% with values as high as 91%. Ignoring model selection uncertainty may have serious implications, e.g. when comparing the growth parameters of different fish populations. Multi-model inference by model averaging, based on Akaike weights, is recommended as a simple and easy to implement method to model fish growth, for making robust parameter estimations and dealing with model selection uncertainty. <s> BIB009 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Process models are no different <s> Recent coordinated efforts, in which numerous general circulation climate models have been run for a common set of experiments, have produced large datasets of projections of future climate for various scenarios. Those multimodel ensembles sample initial conditions, parameters, and structural uncertainties in the model design, and they have prompted a variety of approaches to quantifying uncertainty in future climate change. International climate change assessments also rely heavily on these models. These assessments often provide equal-weighted averages as best-guess results, assuming that individual model biases will at least partly cancel and that a model average prediction is more likely to be correct than a prediction from a single model based on the result that a multimodel average of present-day climate generally outperforms any individual model. This study outlines the motivation for using multimodel ensembles and discusses various challenges in interpreting them. Among these challenges are that the number of models in these ensembles is usually small, their distribution in the model or parameter space is unclear, and that extreme behavior is often not sampled. Model skill in simulating present-day climate conditions is shown to relate only weakly to the magnitude of predicted change. It is thus unclear by how much the confidence in future projections should increase based on improvements in simulating present-day conditions, a reduction of intermodel spread, or a larger number of models. Averaging model output may further lead to a loss of signal— for example, for precipitation change where the predicted changes are spatially heterogeneous, such that the true expected change is very likely to be larger than suggested by a model average. Last, there is little agreement on metrics to separate ‘‘good’’ and ‘‘bad’’ models, and there is concern that model development, evaluation, and posterior weighting or ranking are all using the same datasets. While the multimodel average appears to still be useful in some situations, these results show that more quantitative methods to evaluate model performance are critical to maximize the value of climate change projections from global models. <s> BIB010 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Process models are no different <s> Hollowed, A. B., Bond, N. A., Wilderbuer, T. K., Stockhausen, W. T., A’mar, Z. T., Beamish, R. J., Overland, J. E., and Schirripa, M. J. 2009. A framework for modelling fish and shellfish responses to future climate change. – ICES Journal of Marine Science, 66: 000–000. A framework is outlined for a unified approach to forecasting the implications of climate change on production of marine fish. The framework involves five steps: (i) identification of mechanisms underlying the reproductive success, growth, and distribution of major fish and shellfish populations, (ii) assessment of the feasibility of downscaling implications of climate scenarios derived from Intergovernmental Panel on Climate Change (IPCC) models for regional ecosystems to select and estimate relevant environmental variables, (iii) evaluation of climate model scenarios and select IPCC models that appear to provide valid representations of forcing for the region of study, (iv) extraction of environmental variables from climate scenarios and incorporation into projection models for fish and shellfish, and (v) evaluation of the mean, variance, and trend in fish and shellfish production under a changing ecosystem. This framework was applied to forecast summer sea surface temperature in the Bering Sea from 2001 to 2050. The mean summer surface temperature was predicted to increase by 28C by 2050. The forecasting framework was also used to estimate the effects of climate change on production of northern rock sole (Lepidopsetta polyxystra) through projected changes in cross-shelf transport of larvae in the Bering Sea. Results suggest that climate change will lead to a modest increase in the production of strong year classes of northern rock sole. <s> BIB011 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Process models are no different <s> We quantify the impact of climate model uncertainty upon predictions of future vegetation carbon stocks, for the period up to 2100, generated by a dynamic global vegetation model (DGVM) under a particular emissions scenario (SRES A2). Deterministic simulations are generated from the Lund-Potsdam-Jena (LPJ) model using climatic inputs derived from nine general circulation models (GCMs). Simulated changes between 1961–1990 and 2070–2099 range from +26 to +133 gtC, and are in broadly good agreement with those obtained in a recent previous study using the LPJ model. Simulated values for the 20th century are also obtained by running LPJ with observed climate data, and this provides a baseline against which the other runs can be compared. Time series regression models are used to analyse the discrepancies between each of the GCM-based simulations and the baseline simulation, and a novel form of model averaging—in which we average not only across GCM-based simulations but also across models for each discrepancy—is then used to combine these into a single probabilistic projection for global stocks of vegetation carbon. Weights for the regression models are estimated in a simple post hoc way using the Bayesian information criterion (BIC), and the weights for the GCMs are either estimated in the same way or else fixed to be equal. Estimating the GCM weights leads the predictions to be dominated by a single climate model and hence produces narrow predictive distributions. If GCMs are weighted equally then the predictive distributions are much more diffuse and span the full range of simulated values. Copyright © 2008 John Wiley & Sons, Ltd. <s> BIB012 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Process models are no different <s> Abstract Model–data fusion is defined as matching model prediction and observations by varying model parameters or states using statistical estimation. In this paper, we review the history of applications of various model–data fusion techniques in studies of terrestrial carbon fluxes in two approaches: top-down approaches that use measurements of global CO 2 concentration and sometimes other atmospheric constituents to infer carbon fluxes from the land surface, and bottom-up approaches that estimate carbon fluxes using process-based models. We consider applications of model–data fusion in flux estimation, parameter estimation, model error analysis, experimental design and forecasting. Significant progress has been made by systematically studying the discrepancies between the predictions by different models and observations. As a result, some major controversies in global carbon cycle studies have been resolved, robust estimates of continental and global carbon fluxes over the last two decades have been obtained, and major deficiencies in the atmospheric models for tracer transport have been identified. In the bottom-up approaches, various optimization techniques have been used for a range of process-based models. Model–data fusion techniques have been successfully used to improve model predictions, and quantify the information content of carbon flux measurements and identify what other measurements are needed to further constrain model predictions. However, we found that very few studies in both top-down and bottom-up approaches have quantified the errors in the observations, model parameters and model structure systematically and consistently. We therefore suggest that future research will focus on developing an integrated Bayesian framework to study both model and measurement errors systematically. <s> BIB013 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Process models are no different <s> We show how model averaging can be applied to estimate the probable status of a fishery resource under assessment scenario uncertainty. This approach is applied to North Pacific striped marlin (Tetrapturus audax), an apex predator that may be vulnerable to recruitment overfishing in pelagic longline fisheries targeting tunas. In the current striped marlin assessment, two assessment scenarios were used to account for different hypotheses about the steepness of the stock–recruitment dynamics. Estimates of spawning stock and recruitment from these scenarios are used post hoc to fit age-structured production models that represent alternative hypotheses about the degree of compensation in stock–recruitment dynamics and the degree of serial correlation of environmental forcing. Model-averaged estimates of target spawning biomass to produce maximum sustainable yield (SMSY) and the associated limit fishing mortality (FMSY) characterize relative stock status (S/SMSY and F/FMSY) under each scenario. Scenario-weight... <s> BIB014 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Process models are no different <s> With the growing interest in estimating carbon stocks in forests, available allometric equations have been compiled. These compilations often reveal contrasting models for the same species and site. Rather than choosing a model with the risk of not selecting the best available one, Bayesian model averaging (BMA) offers a way to combine different allometric equations into a single predictive model. In the deterministic version of BMA, existing models with known coefficients are combined. In the statistical version, competing models are at the same time fitted and combined. Using the BMA of deterministic models, we combined three existing multispecies pan-tropical biomass equations for tropical moist forests. The resulting model brought a relatively minor although consistent improvement of the predictions of the aboveground dry biomass of trees. These three models were particular cases of a family of models that were subsequently combined using the BMA of statistical models. Again, the resulting model was able to capture features in the biomass response to diameter that no single model was able to fit. BMA thus is an alternative to model selection that allows integrating the biomass response from different models. FOR .S CI. 58(1):15-23. <s> BIB015 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Process models are no different <s> Advances in numerical weather prediction represent a quiet revolution because they have resulted from a steady accumulation of scientific knowledge and technological advances over many years that, with only a few exceptions, have not been associated with the aura of fundamental physics breakthroughs. Nonetheless, the impact of numerical weather prediction is among the greatest of any area of physical science. As a computational problem, global weather prediction is comparable to the simulation of the human brain and of the evolution of the early Universe, and it is performed every day at major operational centres across the world. <s> BIB016
In fishery science, averaging process models is relatively common (Brodziak and Piner 2010), as it is in weather and November 2018 MODEL AVERAGING IN ECOLOGY 499 climate science BIB002 BIB010 BIB016 . There are at least two connected challenges such enterprises face: validation and weighting. Often process models are tuned/calibrated on all sets of data available, in the sensible attempt to describe all relevant processes in the best possible way. That means, however, that no independent validation data are available, so that we cannot use the prediction accuracy of different models to compute model weights. Consequently, all models receive the same weight (e.g., in IPCC reports or for economic models), or some reasonable but statistically ad-hoc construction of weights is employed (e.g., BIB003 . In recent years, hind-casting has gained in popularity, i.e., evaluating models by predicting to past data. This will only be a useful approach if historic data were not already used to derive or tune model parameters, and if hindcasting success is related to prediction success (which it need not be, if processes or drivers change). Cross-validation is often infeasible for large models, as run-times are prohibitively long. However, the greatest obstacle to averaging process models is the absence of truly equivalent alternative models, which predict the same state variable. Fishery science is one of the few areas of ecology in which commensurable models exist and are being averaged in a variety of ways (e.g., BIB001 BIB005 BIB006 BIB007 BIB008 BIB009 BIB011  BIB014 . Carbon and biomass assessments are also moving in that direction BIB004 BIB012 BIB013 BIB015 ). These fields could profit from exploring averaging methods such as minimal variance and cos-squared, which do not require cross-validation and may perform better than either equal weights or BMA-EM, and probably better than MBMC's potentially overfitted supra-models. Finally, irrespective of the approach chosen, model averaging studies should report model weights, and predictions should be accompanied by estimates of prediction uncertainty.