When Students Design Their Own Games: A Failed Experiment In a First Year Seminar
This paper compares indicators of student engagement across different sections of a firstyear seminar taught in Fall 2017. As part of an active learning pedagogy, students in the author's sections of the course were clustered into teams that designed and played games on refugee migration, aid, and resettlement. Students in seminar sections taught by other faculty members experienced traditional forms of instruction that did not include game design. Data from a survey administered to students in different seminar sections did not indicate an association between game design and student engagement. Further investigation revealed substantial declines in the results of student evaluations of the author's teaching from the previous year, despite only minor differences in course content. Colleagues anecdotally reported a marked decrease in the academic orientation and performance of first year students in 2017, suggesting that pre existing characteristics may be a greater influence on student engagement than an active learning pedagogy involving games. The Active Learning Pedagogy Games and simulations exemplify an active learning instructional methodology. The rationale for their use as teaching tools starts with the premise that they actively engage students in the direct experience of a lived reality, an environment that allows for efficient comprehension of complex knowledge and development of skills. This environment is believed to facilitate learning because it permits rapid and continuous empirical observation, hypothesis formulation, and hypothesis testing (Enterline and Jepsen 2009, 58) Traditional pedagogies, in contrast, require that students passively receive information from teachers, work to understand it, and then, after some delay, attempt to correctly apply it. When a delay exists between encountering knowledge and the opportunity to apply it, it is believed to become less relevant to the learner, resulting in decreased interest and less learning (Dorn 1989, 6) Although active learning seems to have several advantages over traditional teaching methods, its effectiveness is not as clear as is generally assumed. Andrews et al. (2011, 394) found "no association between student learning gains and the use of active learning instruction" in college biology courses. Meta analyses of problem based learning, an active learning methodology frequently used in medical education, found no evidence that it contributed to superior performance on standardized exams (Gijbels et al. 2005, 31 32) In an introductory statistics course, students who experienced the active learning method of a flipped classroom were less satisfied with how the technique "oriented them to the learning tasks in the course" (Strayer 2012, 171) Cobb (2016, 13) found no difference in student performance across traditional, flipped, and online versions of an American government course. In a review of literature on the use of active learning techniques in the political science classroom, Ishiyama (2013, 124) found "remarkably little empirical evidence in the top SOTL journals in US political science and international relations that demonstrates such techniques are effective in promoting student learning." Teaching With Gaming and Simulation Reported benefits of games and simulations include improvements in students' domain knowledge, analytical thinking skills, and exam scores (Shellman and Turan 2006, 30; Frederking 2005, 391; Baranowski 2006, 40) their intellectual and emotional engagement (Weidenfeld and Fernandez 2017, 54; Crocco, Offenholley, and Hernandez 2016, 418) their desire to learn about the practices and beliefs of different ethno cultural groups (Sales et al. 2013, 3) and their empathy toward others (Cruz and Patterson 2005, 43; Chen et al. 2015, 3; Zappile, Beers, and Raymond 2017, 11) A study by Crocco et al. (2016, 419) found that games were "most effective when used to foster higher order thinking, such as application, analysis, and evaluation." In a meta analysis of published literature, Baranowski and Weir (2015, 399) concluded that "a small but growing body of evidence lends support to the contention that students who participate in simulations do in fact learn more than students not taking part in such exercises." However, other studies have not replicated these findings. Raymond (2010, 57) found that no statistically significant improvements in exam scores among students who participated in a simulation, while Powner and Allendoerfer (2008, 85) concluded that there were no statistically significant differences in the overall performance of students who participated in either a role play activity or classroom discussion. I became interested in whether the design of games by students, in contrast to only gameplay, might have a positive effect on students' engagement with a course's academic content. A review of the literature produced only one journal article on the subject, "What Happens When Students Design and Run Their Own Simulations?" by Ken Jones, published in 1998 in Simulation Gaming. In the article, Jones presents a classroom exercise in which groups of students designed simulations by answering four questions: "What's the problem (or issue, or situation) Who are the participants (roles, duties) What do they have to do (talk, plan, negotiate) What do they have to do it with (previous knowledge, documents, match sticks)" (Jones 1998, 393) Jones focuses the exercise's ability to "confer power to the participants" and "encourage teachers to share power and responsibility" (Jones 1998, 342) He does not discuss the activity's usefulness in achieving specific cognitive or affective outcomes. The Course I decided to test my hypothesis that game design might improve student engagement in UNV 101 University Seminar I, a three credit course taken in the first college semester by all entering undergraduates at my university. Launched in Fall 2014 as a part of a revision to the university's general education requirements, its stated purpose to provide students with an opportunity to "begin to develop college level analytical and communication skills to prepare them for academic success" (Salve Regina University 2017) Specific course content is left up to the instructor. I have taught UNV 101 since its inception and emphasize three of the student learning outcomes in the university's general education curriculum (Salve Regina University Core Curriculum Task Force 2013) Speak and write about significant issues in a cogent, analytical and persuasive manner. Analyze and evaluate the interaction among diverse populations around the world and within the United States. Demonstrate awareness of the challenges facing society and of the need for merciful and just responses to them. This last learning outcome provided me with a justification for building a critical, social justice oriented agenda into UNV 101 in 2016. The study of refugees and immigration a topic made all the more relevant by the outcome of that year's U.S. presidential election acting as a framework for this agenda. My intent was not to organize the course around the traditional assessment of students' recall of factual or conceptual knowledge, but to "inspire the adoption of new frames for understanding and, possibly, changing the world" (Barab et al. 2007, 266) My hope was to introduce students to alternative modes of thinking about people who are frequently disparaged and mistreated. Multi week, collaborative game design projects served as an instrument for implementing this agenda in 2016 and 2017. These projects involved students in design thinking, an iterative, feedback driven process of problem definition, understanding the complexity and uncertainty of systems, decision making, and teamwork. Design thinking, when embedded in a project based learning environment, emphasizes the application of knowledge as a means of making the problem at hand understandable (Dym et al. 2005, 104 105; Gaydos 2015, 478) Games happen to represent aspects of design thinking in a manner that is easy for students to comprehend. For example, any game reflects a set of deliberate choices made by the designer that were intended to benefit the game's end users. Games also illustrates the principle that the suitability of a solution often depends on the context of the problem. "[A] game that succeeds in one environment may not succeed in another" (Crocco et al. 2016, 407) During the semester, students in my UNV 101 section participated in three game design projects. Each project included a preparatory memo writing assignment to familiarize students with the game's real world context, a game design and construction phase, play testing of the completed game by another team of students who scored it against a rubric of design criteria, and a performance evaluation in which students ranked the contributions to the project of all team members. The memo, rubric score, and performance evaluation all contributed to students' final course grades. For each project, I provided students with a problem definition of the game they were about to design, specifying the role assumed by the game's players, the game's setting, and its educational objectives. The problem definition for the first game, which placed players in the role of a young South Sudanese woman who flees her village and becomes a refugee, is contained in Appendix A. In the second game, players acted as members of a humanitarian aid organization that constructed and operated a camp for Rohingya refugees in Thailand. For the third game, players decided how a community in the U.S. could best assimilate a group of resettled refugees from Afghanistan. Collectively the three projects represented approximately twenty five percent of the final course grade. In Fall 2017, students also completed twenty three written responses to readings, worth fifty seven percent of the final course grade. Another portion of the c