Modeling Good Research Practices—Overview <s> Related Materials <s> Agent-based modeling and simulation (ABMS) is a new approach to modeling systems comprised of autonomous, interacting agents. ABMS promises to have far-reaching effects on the way that businesses use computers to support decision-making and researchers use electronic laboratories to support their research. Some have gone so far as to contend that ABMS is a third way of doing science besides deductive and inductive reasoning. Computational advances have made possible a growing number of agent-based applications in a variety of fields. Applications range from modeling agent behavior in the stock market and supply chains, to predicting the spread of epidemics and the threat of bio-warfare, from modeling consumer behavior to understanding the fall of ancient civilizations, to name a few. This tutorial describes the theoretical and practical foundations of ABMS, identifies toolkits and methods for developing ABMS models, and provides some thoughts on the relationship between ABMS and traditional modeling techniques. <s> BIB001 </s> Modeling Good Research Practices—Overview <s> Related Materials <s> In financially constrained health systems across the world, increasing emphasis is being placed on the ability to demonstrate that health care interventions are not only effective, but also cost-effective. This book deals with decision modelling techniques that can be used to estimate the value for money of various interventions including medical devices, surgical procedures, diagnostic technologies, and pharmaceuticals. Particular emphasis is placed on the importance of the appropriate representation of uncertainty in the evaluative process and the implication this uncertainty has for decision making and the need for future research. This highly practical guide takes the reader through the key principles and approaches of modelling techniques. It begins with the basics of constructing different forms of the model, the population of the model with input parameter estimates, analysis of the results, and progression to the holistic view of models as a valuable tool for informing future research exercises. Case studies and exercises are supported with online templates and solutions. This book will help analysts understand the contribution of decision-analytic modelling to the evaluation of health care programmes. ABOUT THE SERIES: Economic evaluation of health interventions is a growing specialist field, and this series of practical handbooks will tackle, in-depth, topics superficially addressed in more general health economics books. Each volume will include illustrative material, case histories and worked examples to encourage the reader to apply the methods discussed, with supporting material provided online. This series is aimed at health economists in academia, the pharmaceutical industry and the health sector, those on advanced health economics courses, and health researchers in associated fields. <s> BIB002 </s> Modeling Good Research Practices—Overview <s> Related Materials <s> Based on the National Aeronautics and Space Administration's (NASA's) work in developing a standard for models and simulations (M&S), the subject of credibility in M&S became a distinct focus. This is an indirect result from the Space Shuttle Columbia Accident Investigation Board (CAIB), which eventually resulted in an action, among others, to improve the rigor in NASA's M&S practices. The focus of this action came to mean a standardized method for assessing and reporting results from any type of M&S. As is typical in the standards development process, this necessarily developed into defining a common terminology base and common documentation requirements (especially for M&S used in critical decision making), along with a method for assessing the credibility of M&S results. What surfaced in the development of the NASA Standard was the various dimensions of credibility to consider when accepting the results from any model or simulation analysis. The eight generally applicable factors of credibility chosen in the NASA Standard proved only one aspect in the dimensionality of M&S credibility. At the next level of detail, the full comprehension of some of the factors requires an understanding along a couple of dimensions as well. Included in this discussion are the prerequisites for the appropriate use of a given M&S, the choice of factors in credibility assessment with their inherent dimensionality, and minimum requirements for fully reporting M&S results. <s> BIB003 </s> Modeling Good Research Practices—Overview <s> Related Materials <s> Many still view simulation models as a black box. This paper argues that perceptions could change if the systematic design of experiments (DOE) for simulation research was fully realized. DOE can increase (1) the transparency of simulation model behavior and (2) the effectiveness of reporting simulation results. Based on DOE principles, we develop a systematic procedure to guide the analysis of simulation models as well as concrete templates for sharing the results. A simulation model investigating the performance of learning algorithms in an economic mechanism design context illustrates our suggestions. Overall, the proposed systematic procedure for applying DOE principles complements current initiatives for a more standardized simulation research process. <s> BIB004 </s> Modeling Good Research Practices—Overview <s> Related Materials <s> To date, decision trees and Markov models have been the most common methods used in pharmacoeconomic evaluations. Both of these techniques lack the flexibility required to appropriately represent clinical reality. In this paper an alternative, more natural, way to model clinical reality — discrete event simulation — is presented and its application is illustrated with a real world example.A discrete event simulation represents the course of disease very naturally, with few restrictions. Neither mutually exclusive branches nor states are required, nor is a fixed cycle. All relevant aspects can be incorporated explicitly and efficiently. Flexibility in handling perspectives and carrying out sensitivity analyses, including structural variations, is incorporated and the entire model can be presented very transparently. The main limitations are imposed by lack of data to fit realistic models.Discrete event simulation, though rarely employed in pharmacoeconomics today, should be strongly considered when carrying out economic evaluations, particularly those aimed at informing policy makers and at estimating the budget impact of a pharmaceutical intervention. <s> BIB005
For more information on the ISPOR-SMDM Task Force, visit the Web site at http://www.smdm.org/ispor_smdm.shtml. See also ''Building Better Models: If We Build Them, Will Policy Makers Use Them? Toward Integrating Modeling into Health Care Decisions,'' by Jeanne Mandelblatt, Clyde Schechter, David Levy, Ann Zauber, Yaojen Chang, and Ruth Etzioni, published in this issue on pages 656-659. in a health economic evaluation framework. Therefore, while these papers focus on modeling (drawing broadly on general methods) and apply beyond health economic assessment, the series touches on many aspects pertaining to economic evaluation. Although the use of models to inform policy decision about the use of health technologies has been increasing, 7 there remain strong concerns with their credibility  9 -a concern that is not unique to our field. BIB003 BIB004  To help allay these concerns, several guidelines for good practices in modeling have been issued. 13 In 2000, the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Task Force on Good Research Practices in Modeling Studies was created, and after an extensive process of consultation, it issued its report in 2003. This report defined a model and its purpose, laid out the approach to evaluating a model, and described the task force's consensus regarding the attributes that characterize a good model, in terms of structure, data, and validation. 14 In the intervening years, the range of modeling techniques for medical and economic decision modeling has advanced substantially,  16 as modelers in our discipline have become acquainted with more sophisticated modeling techniques. The relative simplicity of cohort-based models is still an attraction for many modelers and decision makers. Nevertheless, there are situations when the decision problem demands taking history into account and individual-level microsimulation methods are required. 17 Adaptation of methods borrowed from engineering and operations research has led to another way to implement individual-level simulation models, by framing the problem in terms of the states that individuals can be in and the events that can happen and their consequences. BIB005 19 These individual-level and stochastic techniques present additional challenges and require somewhat different approaches to modeling. Infectious disease modeling is a further approach that can handle interaction between individuals, and this dynamic form of modeling has developed its own set of challenges and techniques. 20 The methods for simultaneously handling multiple parameters of a model and addressing uncertainty have also progressed significantly, and the approach to validation of models has received increasing attention. To ensure that the guidelines for good practices in modeling remain current, effective, and helpful, ISPOR judged it necessary to update them to accord with the newer methods being used in practice. As a result, a new Good Research Practices in Modeling Task Force was constituted to build on the excellent work done by the initial one in 2000-2003. To bring to bear the broadest expertise in this area, the Society for Medical Decision Making (SMDM) was invited to join the effort, resulting in a joint working group tasked with providing guidelines for designing the approach, selecting a technique, implementing and validating the model, parameterizing the inputs and assessing uncertainty, and using the resulting tool to inform decision making. Early in 2010, the cochairs and members of the task force were appointed with consent from the ISPOR and SMDM boards. The task force assembled expert developers and experienced users of models from academia, industry, and government, with representation from many countries. Given the breadth of the field at this point, a decision was made to divide the topic into 6 components, and leads were appointed for each. Three of these topics covered the aspects felt to be general to all models in our field: conceptualization of a model, estimation of model parameters and handling of uncertainty, and validation of models and concerns for transparency. The other 3 dealt with specific techniques in common use: state-transition modeling, discrete event simulation (DES), and dynamic transmission models. While there are undoubtedly topics of interest that do not fit into these 6 papers, it was felt that these would cover the major areas and were at a stage of development appropriate for issuing guidelines. The task force held its first meeting via teleconference on May 7, 2010, and hosted information sessions during 2010 at the ISPOR 15th Annual International Meeting, in Atlanta, Georgia; at the 32nd Annual Meeting of the Society for Medical Decision Making, in Toronto, Ontario; and at the ISPOR 13th Annual European Congress, in Prague. Over numerous teleconferences and occasional in-person meetings, the working groups produced draft reports for each section. Although the groups referred to the literature frequently, there was no systematic attempt to review it. Though substantiated as much as possible, the recommendations that emerged represent the opinions of the experts in the task force. These were not forced to consensus, and had substantial differences of opinion remained, they would have been documented as such. The draft recommendations were discussed by the task force as a whole in a meeting held in Boston in March 2011 and subsequently edited and circulated to the task force members in the form of a survey, where each was asked to agree or disagree with a recommendation, and, if the latter, to provide the reason (or reasons). Each group received the results of the survey and endeavored to address all rejections. In the end, there were no dissenting positions. The final drafts of the reports were posted on the ISPOR and SMDM websites for comment by the general membership of the societies. A second group of experts-again, with broad representation of modelers and users of models-was commissioned to formally review the papers. The comments received were addressed by each working group, and revised drafts of each paper were circulated to the task force as a whole. After receiving any additional comments and considering any further revisions, the final version of each paper was prepared. (A copy of the original draft of this paper, as well as the reviewer comments and author responses, is available at the ISPOR website: http://www.ispor .org/workpaper/Modeling-Good-Research-Practices-Overview.asp.) A summary of these papers was presented at a plenary session of the ISPOR 16th Annual International Meeting, in Baltimore, Maryland, in May 2011 and again at the 33rd Annual Meeting of the Society for Medical Decision Making in Chicago, Illinois, in October 2011. The final versions of the papers were then submitted simultaneously to Value in Health and Medical Decision Making. The audience for this set of papers encompasses both the researchers who develop models and those who use models to inform decisions. Investigators charged with reviewing others' models should find the guidelines helpful in their assessments. Even those affected by the decisions informed by models and those who report on the results of modeling analyses should find these recommendations useful. It is important to note, however, that these papers are not intended as primers on their subjects-they are not step-by-step ''how-to'' manuals. General textbooks and tutorial articles covering these techniques already exist, BIB002 BIB001  [24]  and references to specific publications that do address the methods are provided throughout. By the same token, these papers are not methodological treatises that address every aspect of a particular topic. Instead, they propose a set of best practices for modeling. They focus on the types of models and approaches taken today, not on nascent ones nor even on those whose use is currently being debated (e.g., model averaging 26 ). Rather, we have taken the view that these guidelines should reflect current best practice and that expected further development of the methods will require that these guidelines be updated in due course. Although it may not be possible to follow the entire set of recommendations in every modeling exercise, these do represent what the task force felt to be the best practices for modeling today, and each recommendation should be given serious consideration. Nevertheless, the guidelines are not intended for use as a checklist to be followed unthinkingly. We encourage modelers who believe that they should not or cannot follow a particular recommendation to document this divergence, its rationale and likely consequences for their model, its results, and the inferences that will guide decision makers. This overview paper presents the process and methods of the task force and gives an orientation to the contents of each detailed paper. It also provides all the recommendations of the task force but without their detailed rationales and caveats. General quandaries and gaps in knowledge not covered in the other papers are addressed in the final section, along with some thoughts on developments in this area.