Measuring the Commercial Outcomes of Serious Games in Companies – A Review <s> Evidence of the effectiveness of SGs in companies <s> Management games and teaching simulations have obtained widespread adoption in business schools and management development programs. While a number of summary evaluations have demonstrated the efficacy of the management game as a learning environment, little is known about what behavior players exercise within the game situation. Using the critical incident technique, the author collected 1,453 behavior citations of effective and ineffective performance by 211 students playing one of two different management games. The reported behaviors fell into seven broad performance groupings covering the students' pregame preparation, their company's personality and operating characteristics, Board meeting atmosphere and interpersonal climate, the team's use of forecasting, analytical, and planning techniques, and their decision making mechanics. It was found that the management games rewarded rational policy and decision making practices and that chance played no consistent part in company success. As a teaching device, the management game appears to provide many of the elements needed for education in business policy and decision making. <s> BIB001 </s> Measuring the Commercial Outcomes of Serious Games in Companies – A Review <s> Evidence of the effectiveness of SGs in companies <s> With the intensive use of computers in business schools' curriculums the last ten years, many computer games have been developed. This paper is concerned with such a project. The computer game developed in this paper simulates the operation of an exc'usive agency propertyliability insurance company. The paper is divided into sections describing the regulatory environment, insurance operations, investment operations, and evaluation of the gaming process. In the appendix a description of how to interpret game results is shown and explained, based on a simple computer printout. Throughout the paper examples are given where students have obtained valuable information from operating the game. Although many computer games in management, marketing, and finance have been written, few are available in the insurance area. The computer game with which this paper is concerned was developed to increase insurance students' understanding of property-liability insurer operations and to introduce them to some of the problems of insurance company management. The game depicts the operations of an exclusive agency property-liability insurance company that sells only homeowners insurance. The game is organized so that the operations of the insurance company are divided into two major components-insurance operations and investments. Insurance operations include all functions associated with the insurance side of the business: selling, underwriting, and the paying of losses. In the game there are five decision variables the student controls that affect insurance operations: price, underwriting, marketing, commissions, and financial rating. The investment component involves the management of the insurance company's investment portfolio which consists of three assets: cash, bonds, and common stocks. The major interdependence between the two components is through the surplus account. The manner in which the game is played is discussed under the following headings: regulation, insurance operations, investment, evaluation, and summary. James S. Trieschman is Associate Professor of Risk Management and Insurance, University of Georgia. He was a winner of an award for a feature article in the Journal of Risk and Insurance in 1973, and was a Harry J. Loman Fellow in 1974. This paper was submitted for publication, July 1974. <s> BIB002 </s> Measuring the Commercial Outcomes of Serious Games in Companies – A Review <s> Evidence of the effectiveness of SGs in companies <s> Undertaking a review of the literature is an important part of any research project. The researcher both maps and assesses the relevant intellectual territory in order to specify a research question which will further develop the knowledge hase. However, traditional 'narrative' reviews frequently lack thoroughness, and in many cases are not undertaken as genuine pieces of investigatory science. Consequently they can lack a means for making sense of what the collection of studies is saying. These reviews can he hiased by the researcher and often lack rigour. Furthermore, the use of reviews of the available evidence to provide insights and guidance for intervention into operational needs of practitioners and policymakers has largely been of secondary importance. For practitioners, making sense of a mass of often-contrad ictory evidence has hecome progressively harder. The quality of evidence underpinning decision-making and action has heen questioned, for inadequate or incomplete evidence seriously impedes policy formulation and implementation. In exploring ways in which evidence-informed management reviews might be achieved, the authors evaluate the process of systematic review used in the medical sciences. Over the last fifteen years, medical science has attempted to improve the review process hy synthesizing research in a systematic, transparent, and reproducihie manner with the twin aims of enhancing the knowledge hase and informing policymaking and practice. This paper evaluates the extent to which the process of systematic review can be applied to the management field in order to produce a reliable knowledge stock and enhanced practice by developing context-sensitive research. The paper highlights the challenges in developing an appropriate methodology. <s> BIB003 </s> Measuring the Commercial Outcomes of Serious Games in Companies – A Review <s> Evidence of the effectiveness of SGs in companies <s> Abstract Some game users have contended that students can be successful in games even if they “know little” about the rules and the underlying business theories driving them. Thus, if teams are uninvolved, it is possible that successful game play may not impart the learning typically found in experiential exercises nor that designed into games by their creators. In this study, the authors examined the performance of companies led by varying levels of involvement in decision-making strategies. Firms with less involvement performed poorly relative to engaged simulated “ghost” firms and engaged real-student firms. <s> BIB004
In the field of medical education, SGs are relatively new. de Wit-Zuurendonk and Oei (2011) made a literature review on the effectiveness of SGs as a training method for future doctors. They argue that simulations have long been considered as effective in the medical field, and that games could also be effective because learning takes place within contexts that are meaningful to the student. Nevertheless, results of their study show that SGs effectiveness has not been conclusively demonstrated in this particular area, when compared with military training. When measuring the training effectiveness of SGs, Oprins and Korteling (2012) used a control group receiving conventional on-the-job training. They measured both performance and competence by observation and self-assessment. Results showed that both variables were higher for the group playing a SG. In order to clarify the question a systematic literature review was conducted in order to identify the evidence of the effectiveness of serious games used in companies. We want to find studies, or evaluations, of serious games that have been carried out in companies, or at the very least with employees. Thus we ruled out papers which were simply descriptions of business related games or reflections on the potential of serious games. We also excluded from consideration games for business which have been evaluated in an educational context or with student participants -as we are looking for evidence of the impact of serious games on companies. According to  , systematic literature review has been found to provide the high quality and most efficient method for identifying and evaluating extensive literatures. The current theoretical and empirical literature is evaluated in order to provide recommendations for future research directions for scholars in serious games, training and development, human resource management, computer science and social psychology. The first step in the literature review is to define the set of keywords to use for searching the databases. The process of identifying the search terms and keywords was an iterative process, several meetings were held within the review team to decide on the search strings that are most appropriate for the review. After discussions, five search terms were selected: 'company', 'evaluation', 'study', and 'business game' or 'business simulation'. During the search, we excluded 'game theory' because a large number of articles using game theory as theory or experimental studies were found which are irrelevant to the current research. These five key search terms were selected because we were interested in the evaluation of business games used in companies, and these five key search terms can best reflect the parameters of this review. In order to ensure the comprehensive coverage of the literature search, we did not refine the type of game, such as serious game, video game etc., but rather, we included any business relevant game which has been used or evaluated in companies. We focused on peer-reviewed full-text English-language scholarly journal articles when conducting the literature search. According to Armstrong and Wilkinson (2007), journal articles are likely to have the highest impact on the field and can be considered validated knowledge. We also included conference proceedings, given that serious games is an emerging field so there might be relevant conference papers which have not yet been published in journals. There is also a tendency for computer science related research to be published in conference proceedings rather than journals (eg. ACM's HCI conference -CHI). We began with a keyword search using several electronic databases, including Science Direct, ProQuest/ABI, ACM Digital Library (DL), IEEE XExplore Digital Library and the Academy of Management database. These databases were selected to cover the following disciplines: social science, business and management, computer science in order to max-imise the chances of finding relevant articles. The specific reason for selecting Science Direct is that this is a leading scientific database in both science and social science areas. More than 2,500 journals and almost 20,000 books can be found from Science Direct. ProQuest/ABI is one of the most comprehensive business databases. It includes in-depth coverage for over 3,730 publications, with more than 2,670 available in full text. We also selected the ACM and IEEE XExplore digital libraries, since they have extensive coverage of the databases in computer science and information technology. The ACM Digital Library is the most comprehensive collection of full-text articles and bibliographic records covering the fields of computing and information technology, it also indexes the Springer collection. The full-text data-base currently consists of more than 44 high impact Journals as well as more than 275 Conference Proceedings. The IEEE XExplore digital library database provides full text access to more than 140 technical journals and approximately 900 annual conference proceedings published by the Institute of Electrical and Electronics Engineers (IEEE) and the Institution of Engineering and Technology (formerly the Institution of Electrical Engineers). We also included the database of the Academy of Management, since it publishes leading journals in the business and management field and provides the highest quality papers. The reason for us to include this database is to identify if there are papers on serious games so that we can learn from their best research practice. The Academy publishes five journals: articles published in the Academy of Management Journal (AMJ) empirically examine theory-based knowledge; the Academy of Management Review (AMR) provides a forum to explicate theoretical insights and developments; The Academy of Management Learning and Education (AMLE) provides a forum to examine learning processes and management education; the Academy of Management Perspectives (AMP) publishes accessible articles about important issues concerning management and business; and the Academy of Management Annals provides up-to-date, comprehensive examinations of the latest advances in various management fields. Each volume features critical research reviews written by leading management scholars. After the literature search of the databases, a total of 137 articles were found (see Table 3 above). Following the rigorous methodology used by BIB003 , the reviewer reviewed all the 137 articles in-depth, articles that met all the inclusion criteria and which manifested none of the exclusion criteria were selected. For example, to be included in the review, an article had to address games/ simulations used for education, training or learning purposes in companies. Other studies, for example, on simulations (especially with equations) which are used for modelling real organisational processes and games that are used for student education purposes (except executive education) were not included. This resulted in a total of 29 articles. As the decision regarding inclusion and exclusion remain relatively subjective (Tranfield, Denyer & Smart, 2003), after the first selection process, a number of discussions were held within the review team to further discuss the criteria for selecting articles. Finally a total number of six articles were identified. A summary of the studies is in Table 4 . As can be seen in Table 4 , serious games have been found to be an effective teaching tool, as, for example, it has increased learners' participation and interest in the study of insurance company operations BIB002 . The findings of Ben-Zvi (2007) demonstrated the potential for using business games as an educational tool for teaching management information systems (MIS) and Decision Support Systems (DSS). In Pourabdollahian, Taisch & Kerga's (2012) study, a high level of engagement among learners was exhibited during game play based on the evaluation framework adopted. In addition, BIB001 found that management games could reward rational policy and decision making practices. Seven broad groupings' of performance behaviours were also identified in BIB001 research. Of these six studies, four studies used executive education students, one study used a combination of both students and company employees and only one study was carried out with engineers and project managers in a company in Italy. The type of game used ranged from an insurance game, a management game, a concurrent engineering (new product development) game and other business games. The application domain, therefore, included insurance, HRM, business policy and decision making, engineering and information systems. The levels of analysis/operation in the games included the individual, team and firm levels. The evaluation methods employed included experiments, for example,  , which simulated the operations of a multi-firm, one-product industry and BIB004 , who conducted quasi-experiments with senior managers in companies and made comparisons between the engaged firms and unengaged firms in terms of their returns on equity and assets and earnings per share. Four studies used questionnaires, to measure participants' attitudes  , the level of engagement among learners (Pourabdollahian, Taisch, & Kerga, 2012) and the use and contribution of information systems (Ben-Zvi, 2007). The limited number of evaluation studies found in the literature may be due to several reasons. First, the fact that companies who adopt serious games would not wish to disseminate the evaluation information to their competitors, so it remains confidential and hence difficult for scholars to access. In addition, it seems difficult to develop appropriate measures for the learning outcomes, especially for measuring soft skills outcomes (for example, interpersonal skills, leadership and negotiation). Management and HR researchers need to develop validated measures for these so that they can be applied in a standard way to evaluate serious games. There is also the lack of evaluation opportunity problem -serious games can be developed and deployed without evaluations being performed due to the lack of evaluation experts/researchers being there at the time to evaluate the impact. For the studies identified in the literature, although they provide interesting evidence for the use of serious games in companies, a number of methodological issues have been found as well. For example, the performance measures used in BIB004 seem to be weakly formulated. They used in-game indicators, i.e. the return on equity and assets and earnings to evaluate participants' game performance. The use of in-game measures is problematic because we need to be sure that the game generates the correct measures -that is the fidelity and validity of the game's algorithms needs to be high. Separate studies of this validity of the game would need to be carried out, without learning outcome indicators; however, it is difficult to identify the learning effects from the game. Other methodological problems were present in the studies too, in BIB002 research, no sample size was reported and it is thus difficult to evaluate the validity of study. Experiment (This is inferred from the text, as there is no explicit mention of methodology)