Libro_Bastiaens_Education and Information Technology 2012 a Selection of AACE Award Papers
Uploaded by Astro Carrasquilla
saveSave Libro_Bastiaens_Education and Information Technolo... For Later
CPD Social Media Police
inte 6750 trends paper-final
Robotics for Education
Reconstructing Classroom Routines Through on-Line Instructional
article_4
research paper - social media
improve-formal-learning
Chapter Four Charts
Home Access Grants
AB Cover Letter
Ohio State University Medical Center Social Media Participation Guidelines
Twitter Rhetoric and Natural Disasters
Annotated+Bibliography
Teaching of Sanskrit and the Use of ICT at Degree Level
Sociology.pdf
Professional Development Learning Module - Digital Access Syllabus
Social Media Power Point Achmad Zihad
grant s p1d3
A Selection of AACE Award Papers
Theo J. Bastiaens, Ph.D.
Gary H. Marks, Ph.D.
Association for the Advancement of
Computing in Education
aace.org
Education and Information Technology Annual 2012
AACE--Association for the Advancement of Computing in Education
Education and Information Annual - 2012
(ISBN # 1-880094-97-5) is published by AACE, PO Box 1545, Chesapeake, VA 23327-1545, USA
757-366-5606; Fax: 703-997-8760; E-mail: info@aace.org
Copyright 2012 by AACE
www.aace.org
Available at http://www.aace.org/bookshelf.htm
The Association for the Advancement of Computing in Education (AACE), http://AACE.org, founded in 1981, is an
international, not-for-prot, educational organization with the mission of advancing Information Technology in Educa-
tion and E-Learning research, development, learning, and its practical application.
AACE serves the profession with international conferences, high quality publications, leading-edge Digital Library
(http://EdITLib.org), Career Center, and other opportunities for professional growth.
We are proud to present to you this selection of 31 award winning papers from AACEs conferences (http://AACE.
org/conf). This year's selection includes papers from the annual conference of the Society for Information Technology
& Teacher Education (SITE) in Nashville (TN), the Global Learn Asia Pacic conference in Melbourne (Australia), the
World Conference on Educational Multimedia, Hypermedia and Telecommunications (Ed-Media) in Lisbon (Portugal)
and the World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education (E-Learn) in
Honolulu (HI). The decision to nominate a conference paper for an award was made by peer reviewers. All authors were
honored during the conference and received a certicate that serves as testimony to their outstanding research and contri-
bution to the conference.
This AACE nest of 2012 book groups the award winning papers in seven parts. These seven parts provide a timely
overview and record of topics that are of primary interest in educational technology this year.
We hope that the reader enjoys this selection as much as we enjoyed working with these cutting-edge scholars. It
is our intention to establish a new annual tradition with this publication. We look forward to our future selections of
AACEs award papers.
Thank you very much for your support and participation in AACE events and activities.
Theo J. Bastiaens
Gary H. Marks
SOCIAL MEDIA OPPORTUNITIES
The growing use of the social media phenomenon is synonymous with words such as Facebook, Twitter, Web 2.0
and blogging. Teenagers today embraced social media through their active use of multiple social network sites and what
makes social media social and attractive to users is that users are encouraged to share as much information as possible
with other their peers of the site. This growing trend in the use of social media opens up a possibility for researchers to
harness information from this media. In chapter 1, Lee, Kim et al. describe how to use status updates from an exchange
student during his time abroad to create a narrative of his experience aboard.
Being able to design interactive media is an important capacity for young people to develop in order to understand
and negotiate our modern media landscape. Authors Brennan, Valverde et all, describe in chapter 2 a programming en-
vironment that enables young people to create their own interactive media and share their creations within an online
community. They explain the ways in which young peoples development as creators of interactive media is supported by
others, using the context of the online community.
The aforementioned spread of the internet and the evolution of the Web signicantly changes the way how we learn.
Technology brings enhancements for learning almost on a daily basis. In chapter 3 imko, Barla et all, discuss the role
and importance of content annotations in a domain of web-based learning. The authors focus on collaboration support,
continuous improvements of the content quality and increase of students motivation. They introduce two basic types
of activity ows within an adaptive web-based educational system and aim at annotations in both conceptual as well as
practical point of view.
Nowadays podcasts are used in many university courses and often viewed as an effective way to augment undergrad-
uate education. Kushnir, Berry et al. conducted research on the use of podcasts in four disciplines (Art, Dentistry, Design
and Psychology). They surveyed 386 students on their opinions, perceptions, and use of podcasts and found that students
had preconceived notions that the use of podcasts would help them learn. But in spite of students perceptions, the au-
thors found that the use of podcasts did not have a positive impact on their learning. Variables that explain how podcasts
inuence learning outcomes are reviewed in chapter 4 and theories that contribute to the understanding of instructor and
student misconceptions around the use of podcasts are discussed.
Chapter 5 focuses on an important and very often criticized aspect in Net Geners learning, that is, multi-modal/
multi-task learning. The ndings of author Zheng reveal that multi-tasks learning can induce high cognitive load and
with that lower performance in learning. The chapter reveals the benets and at the same time the constraints associated
with multi- modal and multi-task learning.
PART 2 NEW PHENOMENA IN LEARNING:
MOBILE LEARNING/ GAMING / VIRTUAL WORLDS
Mobile applications for teaching and learning are becoming an increasingly popular method of content delivery.
Universities are being confronted with a rapid surge in the penetration of mobile devices amongst students. The chal-
lenge that universities are facing today is how to design and develop device independent mobile applications which can
be easily implemented and integrated and are effective for teaching and learning, regardless of the mobile device in use.
In chapter 6, Khaddage & Knezek discuss mobile applications (Apps) and technologies. Further more design issues and
implications are described and current methods of some proposed applications are reviewed.
In chapter 7, Moldovan & Hava Muntean discuss the educational multimedia content consumption in mobile learn-
ing as one of the key factors driving m-learning acceptance. The authors proposes an original solution for enabling mul-
timedia content personalization based on learners device screen resolution, as well as multimedia content adaptation
based on the available network bandwidth.
Another very popular new phenomenon are Massively Multiplayer Online Games (MMOGs). In Chapter 8 Kong
and Kwok aim to empirically examine four modes of collaborative knowledge creation in Massively Multiplayer Online
Games; game-play, their dynamic relationship with players engagement level, duration and perceived addictiveness. Al-
though an increasing number of researchers explore the use of MMOG as a new generation of educational platform, the
study for examining the empirical evidence of the occurrence and dynamics of collaborative learning behavior in MMOG
is still under-researched. The authors explore the aforementioned collaborative knowledge creation processes in MMOG
grounded on Nonakas dynamic theory of organization creation.
A mystery game, the Lost City, was trialed in chapter 9 as a component of a national on-line problem-solving com-
petition. The game was created largely from free web resources, using low-bandwidth clues released on a schedule to
maximize their use in participants preparing video and stage documentaries of their virtual explorations. Author Ben-
jamins ndings show positive results comparable to previous online mysteries but at a cost in time and resources low
enough to be attainable by individual teachers without additional funding or help.
Author Amory argues in chapter 10 that computer video games support learning when function as tools to medi-
ate knowledge construction. His study makes use of two intrinsic cases as part of a collective instrumental case study to
advance the theory of the use of video games in learning and teaching. Two groups of pre-service teachers participated
in the study. While playing a video game on the biology of diseases groups of undergraduate students developed theoreti-
cal and practical knowledge in the use of video games in teaching and learning. Postgraduate students overcame some to
their misconceptions related to genetics by playing a video game that addresses such learning problems. The introduction
of game puzzles into a learning activity acted as an extrinsic mediator, while the discussions between players intrinsically
mediated their understanding.
Immersive experiences are debated to be one of the main advantages for using virtual worlds for educational simula-
tion, however further research is required to understand how learners become immersed and how to encourage this im-
mersion. Meaningful involvement in a simulation is argued to be one factor determining immersion. In chapter 11, nine
users experiences of immersion within a virtual world role play simulation are described by Cram, Hedberg et al. Their
results demonstrate that user involvement is an important factor in determining immersion.
In Chapter 12 the concept of a Virtual Research Arena is introduced as a framework for creating awareness about
educational and research activities, promoting cross-fertilization between different environments and engaging general
public. Initial results of an explorative case study are presented. Fominykh and Prasolovas study includes a practical ex-
ercise in a cooperation technology course and the rst Virtual Science Fair in Trondheim, Norway.
THE FORMAL CURRICULUM & TECHNOLOGY
Studying abroad has become an important asset in learning languages and acquiring intercultural competences. This
results in a growing need for transparency of curricula. But not only international students need to get a fast overview of
study programs in order to make well-informed decisions. This is why highly usable curriculum visualizations should
complement textual documents and support students and lecturers in navigating through curricula for the sake of fast ori-
entation. Kriglstein and Motschnig-Pitrik present in chapter 13 a case study on a human centered approach to developing
a tool for curriculum visualization.
Armstrong, in chapter 14, describes undergraduate students experiences and perceptions of online courses based
on interviews, observations, and online focus groups. In the chapter the author explains the motivational and learner
characteristics within online classes, the positive and negative aspects of online courses as experienced by students, what
instructors can do to improve the teaching of online courses, and how undergraduate students perceptions of the online
learning environment affects the selection of their approach to learning.
In Chapter 15 the main focus is on discussion forums. After witnessing students delay in making required discussion
board posts until just before the deadline, the authors, Herrick, Lin et al. postulate that a two-deadline solution, one in
which initial responses were due days before peer comments, will serve to better incite dialogue among the students. Sig-
nicant ndings are presented and discussed.
Fluck, Ranmuthugala et al. document in chapter 16 an investigation in four Australian schools to train pupils aged
ten to twelve years how to solve problems using integral calculus with computer algebra system software. After eleven
lessons the students completed a test constructed from items at the level of a rst year engineering degree calculus ex-
amination. The average achievement was at the credit level, and students showed good understanding of the applications
of integral calculus.
Twitchell, Seal et al., reports the success of the U.S. Department of Veteran AffairsVeterans Health Administra-
tions efforts to share training on a large scale. In chapter 17 they report on the growth of sharing among federal health-
care providing government agencies from 2004 to present and discloses cost avoidance gures through 3rd Quarter
2011. A description and architecture is proposed to optimize sharing of existing training and for all levels of aggregation
(i.e., lesson, objects, and assets). They conclude that successful reuse and sharing is is a culture or frame of mind that can
be cultivated and grown in any organization.
The implementation of authentic learning elements at education institutions in ve countries, eight online courses in
total, is examined in chapter 18. Leppisaari, Herrington et al. applied elements of authentic learning as criteria to evalu-
ate authenticity. The results indicate multiple roles and perspectives and scaffolding were the most strongly implemented
elements. Collaborative construction of knowledge was implemented weakly.
The effectiveness of any educational technology depends upon teachers and learners perception of the functional
utility of that medium for teaching, learning, and assessment. In chapter 19 the autors Code, Clarke-Midura et al. studied
the feasibility of immersive technology to develop performance assessments of middle school students scientic inquiry.
Their chapter is a report of the initial ndings of a study conducted with middle school students and their perceptions of
the use of immersive virtual environments for assessment.
In Chapter 20 training systems that use modern surveillance technology and that can automatically track and assess
students exercising physical-motion skills are introduced. Rowe, Houde et al. describe two systems, one with inexpensive
nonimaging sensors and one with multi-camera fusion, to track U.S. Marines during training and assess how well they
are performing. The key challenge here was to visually summarizing the data and their results show that interesting phe-
nomena can be seen in the visualizations that are otherwise often difcult to see.
Chapter 21 presents an empirical study grounded in the Community of Inquiry framework and employs quantitative
content analysis of student discourse and other artifacts of learning in online courses in an effort to enhance and improve
the framework and offer practical implications for online education. As a theoretical framework the purpose of the model
is to describe, explain, and predict learning in online environments. The major question addressed here is whether the
model adequately explains effective learner behavior in fully online courses and to articulate a new conceptual element
learning presence. The Author, Shea, identies instances and conditions under which learning presence is evident in
online courses.
LITARACY
In chapter 22 qualitative research examines the impact of multimedia web-authoring tools on the composing pro-
cesses of literacy learners. The authors, OByrne, Bailey et al. propose that the multimodal features of those tools facili-
tate and extend the range of literacy processes and products and that the shifting between and among the different tools
enriches the construction of knowledge. Two interventions demonstrating learner engagement with multiple tools were
investigated. Using a case study method, some early markers were identied that demonstrated the impact of web-author-
ing tools on the composing processes of grade two and grade eleven students.
With the increasing number of online programs offered by institutions of higher education, it is important to nd
ways to provide information literacy instruction to support off-campus students and to help them succeed in such pro-
grams. Chapter 23 is concerned with the aim of designing instruction appropriate to the needs and existing skills of
online graduate students. The authors, Kumar & Ochoa, conducted a needs assessment of students information literacy
skills before beginning the online program. The results were used to design library instruction intended to ll the gaps in
graduate students information literacy skills and research abilities.
ROLES, COMPETENCIES & ATTITUDE
The rapid growth of online education in higher education is calling for further research on the knowledge, skills and
competencies required for successful online teaching. In chapter 24, the authors Baran, Correia and Thompson critize
teaching approaches that tend to organize teachers roles without making connections between content, pedagogy, tech-
nology and the contextual dynamics that teachers engage in their teaching practices. A literature review on online teach-
ing focused on online teaching roles and competencies, and exemplary online teaching was conducted and limitations of
existing research on online teacher roles, competencies, and paths to exemplary online teaching were explored.
Researchers across disciplines seek to discover common themes, extraordinary patterns, causal relationships, and
ultimately any evidence that will give insight into how the Internet is changing the world. While many researchers are
searching for answers, their ndings paint a conicting picture. The author of chapter 25, Trombley, describes an un-
derlying glitch in studies of Internet use and that is the fact that scholars use the terms sex and gender interchangeably.
Because Internet usage is a specic human behavior gender should play a more essential role than sex in predicting that
behavior. With this in mind, her study provides support for the idea that gender (i.e., a persons psychological identity) is
a better predictor of Internet use than sex.
Meeting International Society for Technology in Education (ISTE) competencies creates a challenge for teachers. In
chapter 26 authors Skoretz and cottle provide a problem-based video framework that guides teachers in enhancing 21st
century skills to meet these competencies. Focusing on content and teaching the technology skills only at the point the
students need to use them are suggested. Furthermore an example is provided on how the problem-based video frame-
work was applied in a higher education setting and the benets and challenges for using video within a problem-based
learning context are discussed.
Chapter 27 is a report on the validation of a new version of the Computer Attitude Questionnaire (CAQ) which is an
instrument for measurement of student attitudes toward computers (comfort and learning with), empathy, creativity, and
school. The new version was developed by the authors, Mills, Wakeeld et al. in response to a request for a more brief
instrument to be used for a National Science Foundations Innovative Technology Experience for Students and Teachers
program research project. The instrument was found to have strong internal consistency reliability, content validity, and
criterion-related validity. It was judged to be acceptable to very good for measurement of student attitudes toward
learning with computers in middle and high school years.
Web-based digital video tools enable learners to access video sources in more constructive ways. To leverage these
affordances teachers need to integrate their knowledge about the potentials of a technology with their professional
knowledge about teaching. In Chapter 28, Krauskopf, Zahn et al. suggest that in a rst step this is a cognitive process,
which is strongly connected to a teachers mental model of the tools affordances. Considering the TPCK-framework the
authors investigated aspects of pedagogical knowledge (PK) in a sample of German pre-service teachers as a predictor
for their mental models of YouTube and how these affect the potential instructional use of this technology. They describe
those mental models of YouTube and present quantitative analyses revealing PK as predictor for the participants intend-
ed and ideal instructional use of YouTube with students. Additionally, this relation is mediated by the mental models for
ideal instructional use.
Despite the continued growth of distance education, many institutions have not considered how best to support and
develop online adjunct faculty. As numbers of online adjunct faculty increase, forecasting their employment needs is
essential. Author Larcara studies, in chapter 29, online adjuncts perceptions of what is important in their work. Her re-
search questions examine perceptions of online adjuncts in nding and retaining work, motivation to teach online, and
professional development. Her study gives voice to an important stakeholder in distance education who directly serves
students, impacts the student experience, and facilitates student learning and achievement.
E-LEARNING ADAPTION
E-learning has been integrated into the university programmes for some time. Although universities have invested in
resources as well as provided training and support teams to encourage teachers to adopt e-learning in their professional
practice, it is according to authors Huan and McKay not welcomed by all teachers in Taiwan. Between the interaction of
universities and their teaching staff, there are many emergent factors that they academic staff to fully embrace e-learning.
In Chapter 30 their aim is to investigate Taiwanese teachers experience in adopting e-learning and then, to explore the
factors affecting their adoption behaviour in Taiwan higher education.
In the nal chapter the authors Fischer and Khler introduce empirical ndings of a research project Adoption of
e-learning innovations in Higher Education. The aim of the project is the characterization of teachers as potential adopt-
ers of e-learning innovations. Therefore the needs, motivation and attitudes of teachers in Higher Education toward e-
learning innovations were analyzed. By using multivariate analysis (cluster analysis, factor analysis) four adopter types
could be identied: young professionals, experienced explorers, preservers and reward seekers. Their empirical ndings
can be used for designing target group-specic change management strategies during introducing e-learning innovations
in academic teaching.
ED-MEDIA World Conference on Educational Multimedia,
Hypermedia & Telecommunications
E-Learn World Conference on E-Learning in Corporate,
Healthcare, Government, and Higher Education
SITE Society for Information Technology and Teacher
Education International Conference
Adding soon!
Global Learn Asia Pacific Global Conference on Learning
Global TIME Global Conference on Technology,
Innovation, Media & Education
www.EdITLib.org
Free access to abstracts so you can try the Digital Library at no cost!
Conduct research Keep current on the latest research and publications in your field
Access and fully search publications Create and store personal collections of articles by topic
Receive table of contents alerts prior to journal publication
Share article abstracts and search results with colleagues via email and include your comments
Export citations to BibTex, EndNote, and RefWorks
EdITLib is your source for 15+ years (with your subscription) of peer-reviewed,
published articles (20,000+) and papers on the latest research, developments, and
applications related to all aspects of Educational Technology and E-Learning.
Special Topic Books
Conference Invited Speaker Talks
Conference Presentation Slides
Journal of Educational Multimedia and Hypermedia
International Journal on E-Learning
(Corporate, Government, Healthcare, & Higher Education)
Journal of Computers in Mathematics and Science Teaching
Journal of Interactive Learning Research
Journal of Technology and Teacher Education
AACE Journal
Contemporary Issues in Technology & Teacher Education
Ten (10)
Email: info@aace.org Phone: 757-366-5606 Fax: 703-997-8760
Does Your Library
PART 1 SOCIAL MEDIA OPPORTUNITIES
1 Creating the intercultural learning narrative using social network sites status updates: An innovative approach in
Jason Wen Yau Lee, Beaumie Kim, and Mi Song Kim.................................................................................................15
2 More than code: The signicance of social interactions in young peoples development as interactive media creators
Karen Brennan, Amanda Valverde, Joe Prempeh, Ricarose Roque, and Michelle Chung............................................23
3 Supporting Collaborative Web-Based Education via Annotations
Marin imko, Michal Barla, Vladimr Mihl, Maro Unk, and Mria Bielikov....................................................33
4 Lecture Capture: Good Student Learning or Good Bedtime Story? An Interdisciplinary Assessment of the Use
of Podcasts in Higher Education
Lena Paulo Kushnir & Kenneth Berry, University of Toronto, Canada; Jessica Wyman, Florin Salajan.....................43
5 Findings from Net Geners Multi-Modal and Multi-Task Learning
Robert Zheng.................................................................................................................................................................55
PART 2 NEW PHENOMENA IN LEARNING: MOBILE LEARNING/ GAMING/ VIRTUAL WORLDS
6 Device Independent Mobile Applications for Teaching and Learning: Challenges, Barriers and Limitations
Ferial Khaddage and Gerald Knezek............................................................................................................................61
7 Towards Personalised and Adaptive Multimedia in M-learning Systems
Arghir-Nicolae Moldovan and Cristina Hava Muntean..................................................................................................67
8 Knowledge Creation in MMOG: An Empirical Study
Joseph Siu-Lung Kong and Ron Chi-Wai Kwok..........................................................................................................79
9 The Lost City: Development of a National-Level On-Line Mystery Game Using Freeware and Low Budget
Tom Benjamin.................................................................................................................................................................9
10 Play games to learn: Pre-service teacher development
Alan Amory.................................................................................................................................................................101
11 Beyond Immersion Meaningful Involvement in Virtual Worlds
Andrew Cram, John Hedberg and Maree Gosper.........................................................................................................111
12 Virtual Research Arena: Presenting Research in 3D Virtual Environments
Mikhail Fominykh and Ekaterina Prasolova-Frland.................................................................................................121
PART 3 THE FORMAL CURRICULUM & TECHNOLOGY
13 The Curriculum as an Ontology - A Human Centered Visualization Approach
Simone Kriglstein and Renate Motschnig-Pitrik..........................................................................................................131
14 Students Perceptions of Online Learning and Instructional Tools: A Qualitative Study of Undergraduate
Students Use of Online Tools
David Armstrong.........................................................................................................................................................141
15 Online Discussions: The Effect of Having Two Deadlines
Michael Herrick, Meng-Fen Grace Lin and Charlotte Huei-Wen...............................................................................147
16 Calculus in elementary school: an example of ICT-based curriculum transformation
Andrew Fluck, Dev Ranmuthugala, Chris Chin and Irene Penesis..............................................................................155
17 Green Training: Chronicling the Reuse of Government Healthcare Instruction
David Twitchell, Mitchell Seal and Christopher Lynch................................................................................................163
18 Authentic e-Learning in a Multicultural Context: Virtual Benchmarking Cases from Five Countries
Irja Leppisaari, Jan Herrington, Leena Vainio, and Yeonwook Im.............................................................................173
PART 4 ASSESMENT
19 Student Perceptions of Immersive Virtual Environments for the Meaningful Assessment of Learning
Jillianne Code, Jody Clarke-Midura, Nick Zap, and Chris Dede................................................................................185
20 Visualizing summaries of performance for instructors assessing physical-motion skills
Neil Rowe, Jeff Houde, Rey Osoteo, Riqui Schwamm, Cory Kirk, Ahren Reed, Saad Khan, Chris Broaddus
and Chris Meng...........................................................................................................................................................195
21 Learning Presence in the Community of Inquiry Model: Towards a Theory of Online Learner Self- and
Co-regulation
Peter Shea....................................................................................................................................................................207
PART 5 LITERACY
22 Literacy in Multimedia Environments: Preliminary Findings
Barbara OByrne, Diana Bailey and Stacey Murrell...................................................................................................215
23 Student-Centered Library Instruction: An Assessment of Online Graduate Students Information Literacy Skills
and Needs
Swapna Kumar and Marilyn Ochoa............................................................................................................................221
PART 6 ROLES, COMPETENCIES & ATTITUDES
24 Paths to Exemplary Online Teaching: A Look at Teacher Roles, Competencies and Exemplary Online Teaching
Evrim Baran, Ana-Paula Correia and Ann Thompson..................................................................................................229
25 Virtual Gender Roles: Is Gender a Better Predictor of Internet Use than Sex?
Amy Trombley.............................................................................................................................................................237
26 Meeting ISTE Competencies with a Problem-Based Video Framework
Yvonne Skoretz and Amy Cottle.................................................................................................................................243
27 Validating the Computer Attitude Questionnaire NSF ITEST (CAQ N/I)
Leila Mills, Jenny Wakeeld, Anjum Najmi, Dean Surface, Rhonda Christensen and Gerald Knezek......................249
28 Leveraging the Affordances of YouTube: Pedagogical Knowledge and Mental Models of Technology
Affordances as Predictors for Pre-Service Teachers Planning for Technology Integration
Karsten Krauskopf, Carmen Zahn and Friedrich W. Hesse..........................................................................................257
29 Supporting the Online Adjunct in Higher Education: A Delphi Study
Marie Larcara..............................................................................................................................................................265
PART 7 E-LEARNING ADAPTION
30 The Factors Inuencing E-learning Adoption by Academic Staff: A Case Study in Taiwan
Kuo-Tung Huan and Elspeth McKay............................................................................................................................273
31 Adopter Types of E-Learning Innovations in Higher Education. Empirical Findings
Helge Fischer and Thomas Khler................................................................................................................................285
AUTHORS AND EDITORS CONTACT INFORMATION
Chapter 1 Creating the intercultural learning narrative using social network sites status updates: An innovative approach
in using social media
Jason Wen Yau Lee, Beaumie Kim & Mi Song Kim, National Institute of Education, Singapore
E-mail: jasonleewy@gmail.com
Chapter 2 More than code: The signicance of social interactions in young peoples development as interactive media
Karen Brennan, MIT Media Lab, USA; Amanda Valverde & Joe Prempeh, Harvard University, USA; Ricarose
Roque & Michelle Chung, MIT Media Lab, USA
E-mail: kbrennan@media.mit.edu
Chapter 3 Supporting Collaborative Web-based Education via Annotations
Marin imko, Michal Barla, Vladimr Mihl, Maro Unck & Mria Bielikov, Slovak University of Technology,
E-mail: {simko, barla}@it.stuba.sk
Chapter 4 Lecture Capture: Good Student Learning or Good Bedtime Story? An Interdisciplinary Assessment of the Use
Lena Paulo Kushnir & Kenneth Berry, University of Toronto, Canada; Jessica Wyman, OCAD University, Canada;
Florin Salajan, North Dakota State University, USA
E-mail: lena.paulo.kushnir@utoronto.ca
Chapter 5 Findings from Net Geners Multi-Modal and Multi-Task Learning
Robert Zheng, University of Utah, USA
E-mail: NO EMAIL AVAILABLE
Chapter 6 Device Independent Mobile Applications for Teaching and Learning: Challenges, Barriers and Limitations
Ferial Khaddage, Deakin University School of Information Technology, Australia; Gerald Knezek, University of
Northern Texas Department of Learning Technologies, USA
E-mail: fayekhaddage@gmail.com
Chapter 7 Towards Personalised and Adaptive Multimedia in M-learning Systems
Arghir-Nicolae Moldovan & Cristina Hava Muntean, National College of Ireland, Ireland
E-mail: amoldovan@student.ncirl.ie
Chapter 8 Knowledge Creation in MMOG: An Empirical Study
Joseph Siu-Lung Kong & Ron Chi-Wai Kwok, Department of Information Systems, City University of Hong Kong,
E-mail: slkong@student.cityu.edu.hk
Chapter 9 The Lost City: Development of a National-Level On-Line Mystery Game Using Freeware and Low Budget
Tom Benjamin, New South Wales Department of Education & Training, Australia
E-mail: thomas.benjamin@det.nsw.edu.au
Chapter 10 Play games to learn: Pre-service teacher development
Alan Amory, University of Johannesburg, South Africa
E-mail: aamory@uj.ac.za
Chapter 11 Beyond Immersion Meaningful Involvement in Virtual Worlds
Andrew Cram, John Hedberg & Maree Gosper, Macquarie University, Australia
E-mail: andrew.cram@students.mq.edu.au
Chapter 12 Virtual Research Arena: Presenting Research in 3D Virtual Environments
Mikhail Fominykh & Ekaterina Prasolova-Frland, Program for learning with ICT, Norwegian University of Science
and Technology, Norway
E-mail: mikhail.fominykh@svt.ntnu.no
Chapter 13 The Curriculum as an Ontology - A Human Centered Visualization Approach
Simone Kriglstein & Renate Motschnig-Pitrik, University of Vienna, Faculty of Computer Science, Austria E-mail:
simone.kriglstein@univie.ac.at
Chapter 14 Students Perceptions of Online Learning and Instructional Tools: A Qualitative Study of Undergraduate
David Armstrong, University of San Francisco, USA
E-mail: DArmstrong@scu.edu
Chapter 15 Online Discussions: The Effect of Having Two Deadlines
Michael Herrick, Meng-Fen Grace Lin & Charlotte Huei-Wen, University of Hawaii, USA
E-mail: mherrick@hawaii.edu
Chapter 16 Calculus in elementary school: an example of ICT-based curriculum transformation
Andrew Fluck, Dev Ranmuthugala, Chris Chin & Irene Penesis, University of Tasmania, Australia
E-mail: Andrew.Fluck@utas.edu.au
Chapter 17 Green Training: Chronicling the Reuse of Government Healthcare Instruction
David Twitchell, Department of Veterans Affairs, USA; Mitchell Seal & Christopher Lynch, US Navy, USA
E-mail: david.twitchell1@va.gov
Chapter 18 Authentic e-Learning in a Multicultural Context: Virtual Benchmarking Cases from Five Countries
Irja Leppisaari, Central Ostrobothnia University of Applied Sciences, Finland; Jan Herrington, Murdoch University,
Australia; Leena Vainio, HAMK University of Applied Sciences, Finland; Yeonwook Im, Hanyang Cyber University,
E-mail: irja.leppisaari@cou.
Chapter 19 Student Perceptions of Immersive Virtual Environments for the Meaningful Assessment of Learning
Jillianne Code, University of Victoria, Canada; Jody Clarke-Midura, Harvard University, USA; Nick Zap, Simon
Fraser University, Canada; Chris Dede, Harvard University, USA
E-mail: jcode@uvic.ca
Chapter 20 Visualizing summaries of performance for instructors assessing physical-motion skills
Neil Rowe & Jeff Houde, U.S. Naval Postgraduate School, USA; Rey Osoteo, U.S. Naval Posgraduate School, USA;
Riqui Schwamm, Cory Kirk & Ahren Reed, U.S. Naval Postgraduate School, USA; Saad Khan, Chris Broaddus &
Chris Meng, Sarnoff Laboratories, USA
E-mail: ncrowe@nps.edu
Chapter 22 Learning Presence in the Community of Inquiry Model: Towards a Theory of Online Learner Self- and Co-
regulation Peter Shea, University at Albany, State University of New York, USA
Chapter 22 Literacy in Multimedia Environments: Preliminary Findings
Barbara OByrne, Diana Bailey & Stacey Murrell, Marshall University Graduate College, USA
Chapter 23 Student-Centered Library Instruction: An Assessment of Online Graduate Students Information Literacy
Skills and Needs
Swapna Kumar & Marilyn Ochoa, University of Florida, Gainesville, USA
E-mail: pshea@uamail.albany.edu
Chapter 24 Paths to Exemplary Online Teaching: A Look at Teacher Roles, Competencies and Exemplary Online Teaching
Evrim Baran, Ana-Paula Correia & Ann Thompson, Iowa State University, USA
E-mail: evrimb@gmail.com
Chapter 25 Virtual Gender Roles: Is Gender a Better Predictor of Internet Use than Sex?
Amy Trombley, The University of North Texas, USA
E-mail: amytrombley@my.unt.edu
Chapter 26 Meeting ISTE Competencies with a Problem-Based Video Framework
Yvonne Skoretz & Amy Cottle, Marshall University, USA
Chapter 27 Validating the Computer Attitude Questionnaire NSF ITEST (CAQ N/I)
Leila Mills, University of North Texas, USA; Jenny Wakeeld, Anjum Najmi & Dean Surface, UNT, USA; Rhonda
Christensen & Gerald Knezek, University of North Texas, USA
E-mail: leilamills@yahoo.com
Chapter 28 Leveraging the Affordances of YouTube: Pedagogical Knowledge and Mental Models of Technology Af-
fordances as Predictors for Pre-Service Teachers Planning for Technology Integration. Karsten Krauskopf, Carmen
Zahn & Friedrich W. Hesse, Knowledge Media Research Center, Germany
E-mail: k.krauskopf@iwm-kmrc.de
Chapter 29 USA Supporting the Online Adjunct in Higher Education: A Delphi Study
Marie Larcara, Canisius College
Chapter 30 The Factors Inuencing E-learning Adoption by Academic Staff: A Case Study in Taiwan Kuo-Tung Huan &
Elspeth McKay, SBITL RMIT University, Australia
E-mail: kuotung.huan@rmit.edu.au
Chapter 31 Adopter Types of E-Learning Innovations in Higher Education. Empirical Findings.
Helge Fischer & Thomas Khler, Prof. Dr, Media Center, Technical University Dresden, Germany
E-mail: helge.scher@tu-dresden.de
Theo Bastiaens is professor of Educational Technology at the Fernuniversitt in Hagen, Germany and part time professor
at the Open University, The Netherlands. He is a member of the AACE board of directors
E-mail: Theo.Bastiaens@fernuni-hagen.de
Gary Marks is CEO and founder of AACE.
E-mail: info@aace.org
Creating the Intercultural Learning Narrative Using Social Network Sites Status Updates 15
Creating the Intercultural Learning Narrative Using Social Network Sites Status Updates:
An Innovative Approach in Using Social Media
JASON WEN YAU LEE*, BEAUMIE KIM, AND MI SONG KIM
In recent years, there has been a push by the software industry as a whole in the development of social software that
has enabled people to express themselves freely and collaboratively. This has caused the use of social media applications
to steadily increase over the past years, coupled with the increasing number of such applications developed and pushed
across the Internet. Even major news portals such as CNN and the BBC have integrated social media applications into
their site. For example, readers of such sites are able to easily share any news articles that they nd interesting on their
social bookmarks such as Delicious, StumbleUpon and Facebook; or they can leave comments on the articles that others
can see. Humans are essentially social beings and technology has enabled these social applications to be integrated into
our daily lives. It is quite inconceivable that the average urban teenagers today in developed nations not have some form
of their social presence publically viewable online.
With this growing trend of the usage of social media, it is very likely that teenagers will be actively using these
social media in their daily lives. The amount of time spent by teenagers on social network sites has been steadily incre-
asing over the past few years (Lenhart, Purcell, Smith, & Zickuhr, 2010) and it is not surprising that there is a wealth of
information being posted on their daily lives in these sites. Due to the informal nature of these sites, a person is likely to
be feeling more free and more comfortable with expressing him or herself as compared with a more formal environment
such as interviews, eld observation or to an extent, journal writing.
In this paper, we described how we use the status updates to create the intercultural learning story of a participant on
an intercultural exchange program. Traditionally, researchers wanting to understand the intercultural learning experience
of participants on exchange programs used blogs, journal writing, surveys, and interviews (e.g., Cushner & Karim, 2003;
Hansel, 2008; Koskinen & Tossavainen, 2004; ONeill, 2008). In this paper, we propose a more naturalistic digital ethno-
graphy approach in trying to understand the learning experience. This is achieved by using a collection of status update
over the period of intercultural exchange abroad and we created a narrative of the experience using these status updates.
2.0 Situating the study
Ethnography is a study where the researcher goes to the eld and returns to report their stories. Essentially, the
ethnographic approach study people in everyday setting with attention to the participants meaning-making process (An-
derson-Levitt, 2006). People make meanings and learn by interacting with others, interpret their experience, and generate
behaviors based on what they learn (Spradley, 1980). This means that our ways of behaving are acquired through social
interactions and may change according to the situations and people that we encounter. This shared culture of how to in-
teract with each other in a community is what ethnographers are interested in their studies.
As we move into the digital age, researchers have begun to leverage on social network technologies in their re-
search methods (e.g., Ellison, Steineld, & Lampe, 2007; Lee, Kim, & Kim, 2010; Mazer, Murphy, & Simonds, 2007;
Raja Hussain & Ng, 2010). Despite this change, the epistemology underpinning digital ethnography remains the same,
which is to understand the behavior of people but in a digital environment. With the availability of new media, ethno-
graphers can leverage the use of these technologies as avenues for understanding the meaning making process of study
participants. Blogs, online questionnaires, forums and social networking can be thought as digital spaces where people
can mingle around and hang out in a virtual environment. With these technologies, ethnography has gone digital, which
changes how the participants are observed in this borderless environment.
16 Lee, Kim & Kim
Social media for research
Figure 1 is a compilation of social media technologies that are popularly used today. These technologies are plot-
ted along the asynchronous-synchronous technologies vs. private-public space. Synchronous technologies allows for real
time communication such as instant messaging while asynchronous communication does not occur in real time such as
forums or emails. We further divided these technologies to public space and private space. Technologies that are classi-
ed to be in the public space can be viewed by everyone while those in the private space are restricted to those privy to
the information. For example, blogs can be both public and private depending on the privacy level set by the user. SNSs
provide an interesting case as it contains both asynchronous (e.g., wall message, private message) and synchronous com-
munication (Facebook chat) functions within the site. With the privacy control afforded in most SNS, users can choose to
have the information within their prole be public or restricted to friends but a direct connection to friends is required in
order to have full access to a persons social network prole.
Figure 1. Social media in the public-private and synchronous-asynchronous dimensions. (Adapted from Hernndez-Ramos,
The term social media has come to encompass a wide variety of applications such as blogs, wikis, Twitter, and social
network sites. Although these technologies had humble beginnings once mainly accessible only to the technologically sa-
vvy, it has now become widely available to the mass. Blogs are perhaps one of the oldest technologies that have evolved
to become part of the social media. It is social as readers are able to leave comments or receive feedback from their rea-
ders and able to interact in an asynchronous manner. Blogs are also been used by researchers (e.g., Hernndez-Ramos,
2004; Lin & Yuan, 2006) as a source of insights into student reection during the learning process.
However, as we move towards a more instantaneous world, the use of technologies that allows for quick access to
information has become increasingly popular. While blogs were useful for one to write down some reective thoughts,
microblogging allows for a person to quickly broadcast what he thinks in a short sentence burst. Microblogging is a form
of asynchronous communication that has gained popularity to broadcast ideas, thoughts and even as a marketing tool for
companies where these short messages are archived on a microblogging site similar to how blogs are archived. What dis-
tinguishes these microblogs from blogs are the amount of words able to be published, typically under 500 characters thus
requiring the users to be concise with their postings. Among the popular microblog services are Twitter, Tumblr, Jaiku,
and Sixapart but these features are not limited exclusively to these microblogging sites. Social network sites also have
their own built in microblog services, such as status updates in Facebook that can seamlessly connect with each other
through certain protocols. A user only needs to update one of these services and the content can be aggregated over mul-
tiple services.
Although blogs and microblogs provide an avenue for people to express their feelings and thoughts, they do not
provide the ability for people to easily connect with each other as microblogs are developed mainly to publish content.
In parallel to the increasing popularity of microblogs, social network sites began to develop as an avenue for people to
connect with each other. One of the earliest popular SNS was MySpace whose helm is now taken over by Facebook. For
active users of such SNS, a considerable amount of time would have gone into the content creation process of their pro-
le page, which is an extension of the users online social presence. The vast amount of information that is being posted
on the users SNS page creates the possibility to give an understanding into what the person is experiencing in their life.
For researchers, the term social network site has been gaining much attention. There has been also a steady and growing
empirical research being published to investigate the use of social network sites (e.g., boyd & Ellison, 2008; Lee, et
al., 2010; Raja Hussain & Ng, 2010; Zywica & Danowski, 2008). However, it was not until the early 2000s when Web
2.0 technologies (OReilly, 2005) such as AJAX and other lightweight programming languages that powered SNS has
reached maturity to enable the growth of SNS as we know today.
While the journal writing process is often a private affair, a more recent alternative is to use blogs as tools for re-
ective writing (Jonassen, 2008). Blogs are a collection of entries of commentary written by individuals or groups of
individuals and displayed in reverse chronological order. It gained mainstream popularity as a way for people to express
their views and thoughts online. Blogs have shown to be useful in motivating students to actively reect on their learning
experiences and expressed positive responses to their learning experience (Lin & Yuan, 2006). Park (2003) called blogs
as a successor to the learning journal as it serves as a vehicle for individual reection. While blogs seems to be a promi-
sing avenue for students to reect on their experiences, Hall and Davison (2007) cautioned that some students may not be
as enthusiastic or encouraged when using blogs to reect on their experiences.
3.0 Using SNS to create the intercultural story
We center the discussion of this paper on a popular SNS called Facebook. Facebook is one of the most popular SNS
which has over 500 million users are logged in on a daily basis with a total of 700 billion minutes spent per month on
Facebook (Facebook, 2010). The term friend is loosely dened on SNS. In a SNS, friends are people whom one share
a virtual social connection. For example, family members, acquaintances, co-workers or even strangers can be considered
as friends on SNS. To make adding friends easier, one of the features within Facebook is the system would suggest
friends to users based on mutual friends that they share. However, adding friends on Facebook is a bi-directional process
where conrmation is required in order for such a social connection to be built. Once another user has been approved as
a friend, they will have more access to information that would otherwise be restricted.
3.1 Facebook Status Updates
Status update is a feature within Facebook that lets users post short messages of up to 420 characters. Status updates
are essentially a short shout-out or emotional expression that a person wishes to share with friends. On Facebook, users
are encouraged to post their current status where prompted by the question Whats on your mind? when they log into
their Facebook page. When a status update is posted, it is published and appears on the users wall and on his friends
news feed, which is the main landing page upon login to Facebook . Friends can also leave comments on the status up-
dates.
These status updates and corresponding comment are archived with a timestamp and respondents name, thus pro-
viding researchers an insight on the interaction that has taken place within this space. These postings within the SNS
are the digital equivalent of a short conversation between friends. Unlike an asynchronous chat or forum posting, status
update postings and responses are limited by a word count thus making the conversation more concise. If users post a
status update that is interesting, a rich conversation can sometimes ourish from this status update as illustrated in Figure
3. This unstructured conversation between friends in an informal environment gives the users freedom to express them-
selves. For the researcher, this is important as exchanges of meaning in this environment occur in a naturalistic yet virtual
3.2 Analyzing Facebook Status update postings
To illustrate how we can use SNS in intercultural research, we take a closer look at Nick, a 17-year old from Malay-
sia who is on exchange to the United States for a 6-month program. Nick is a regular user of Facebook who updates his
status from several times a week to several times a day. During his time abroad to the United States, he posted 102 status
updates to his wall.
We found that one of the most common purpose of his status updates were to share his experiences with his friends.
Occasionally, we also observed that the status updates are sometimes used to brag to friends on anything they nd excit-
ing to them. These experiences of differences (such as culture, weather or people) can be a useful source of information
for the researcher to understand what a person is going through in their intercultural learning process. As status updates
are often short burst of expressions or experiences, a single status update may not be able to provide a rich account of the
experience. However, a collection of status updates over a longer period of time can show a trajectory of the experience a
person values and nds important.
For example, in the following ve status updates, we demonstrate Nicks experience with the weather. Nick rst re-
ferred to the weather in a negative manner when he ranted that typing in the cold makes his hand numb (turn 1) and how
the cold weather is wrecking his skin (turn 3). Subsequent status updates were more positive when he refers to how he
is loves seeing the snow cascading down (turn 2) and how he marvels at the snow storm (turn 4). Being from a tropical
climate, he feels such an experience perhaps exciting and worthwhile sharing. As the seasons changed, he continued to
update his status to describe his feeling of the weather. In turn 5 he updated his status describing how he is enjoying the
weather in spring.
Table 1: Status updates on experiences valued by Nick
Turn Date Status Update
Nick says typing in the cold is not fun with your ngers going numb :(
Nick loves looking at the snow cascading down from above :)
Nick hates how the cold is wrecking his skin :[
Nick marvels at the snow storm taking place outside
Nick just laid on the grass at the park shirtless. Ahh... spring :)
3.3 Creating the intercultural narration
Using the status updates collected in the previous section, we can use these status updates to create a story of the
participants intercultural learning experience. While the status updates may not mean much by itself, analyzing the sta-
tus updates and the corresponding responses over a period of time can yield a more important account on the experience.
However, this will require the researcher to interpret these status updates to create a meaningful story on the experience.
In this section, we will status updates similar to those in Table 1 to create a short narrative on Nicks experience abroad
with a focus on his adjustment back to his home country after the program.
Nick is a 17-year old Malaysian who was selected to participate on an exchange program to the United States for 6
months. He was excited about his exchange when he updated his status that he was ready to take on America. Several
days later, he updated his status to indicate he has received his host family placement. To show his excitement on this
new development, he ended the status update with a *dances* to describe the action dancing. During the rst few days
upon arrival to the United States, he updated his status several times on his safe arrival to his host family and that they
were super nice.
Being from a tropical climate, Nick may not be used to the cold weather he was experiencing at that time. In his sta-
tus update postings, Nick referred to the cold weather several times during the initial part of his stay. For example, Nick
described the cold weather negatively by posting a status update describing how his ngers were going numb from the
cold weather and that it was not fun typing that way and that how the cold weather was wrecking his skin. On the other
hand, the cold weather was also something that Nick found interesting when he updated his status to describe how he
marvels at the snow cascading from above and how he marvels the snow storm that was taking place outside his home.
One of the biggest challenges that Nick will face is to adjust to his new environment. While the weather was some-
thing different for Nick, the host family plays an important role in the acculturation and adjustment process by making
Nick feel welcomed to his new environment. This was observed in his Facebook status update after arriving two weeks
when Nick updated his status to say that he loves his host family and several days later that he is tting in just ne.
The process of assimilating and learning a new culture is something that occurs over a period of time. It is important
is that the learner reects on their experience and the realization that changes are happening within themselves is a pow-
erful tool in learning. Nick made this important reection a month after he arrived when he updated his status asking if
he does really speak with an American accent. Although he speaks English well, his reection that he speaks with an
American accent shows that he realized that he has unknowingly begun the assimilation process into his new host cul-
ture.
It is common that sojourns develop a sense of homesickness when they start to reminisce memories of home or feel
like a stranger in the new environment. Sojourns are known to miss things that are familiar to them such as food, people
or the environment. Nick was no different that the other exchange students when he also gone through a period of home-
sickness. His rst status update reected this in mid February when he asks if it is wrong to feel this way and he wants to
feel familiar. As this status update was posted in the initial stages during his exchange, it is likely that this is the begin-
ning of his cultural assimilation process when he begins to realize the subtle cultural differences that were not apparent
earlier on. A month later, we also observe a similar statement when he updated his status to indicate that he craves for
cheesecake from a caf back home and asks why he feels so. He ended the status update with a :( emoticon to indicate
his unhappiness with feeling that way.
Nick also experienced many positive experiences during his exchange that he wanted to share with his friends. Dur-
ing his exchange, he used his status updates inform his friends on the places he visited such as the Mall of America,
watched a basketball game, and traveling to New York and to Los Angeles. In a way, these status updates serve as a way
for Nick to brag to his friends about his experiences and is a form of social comparison where he is letting his friends
know that he is doing well and also able to travel around during his time abroad.
Three more months into the middle of his exchange program, Nick began to have conicting feelings about return-
ing home. This was indicated where he ended both of the status updates with an emoticon that indicates sadness. Despite
feeling homesick two weeks ago, Nick updated his status to indicate that he was not ready to return home just yet. In his
next status update, he shows conicting emotions about returning home when he post a rhetoric question to indicate that
he misses home but does not really want to return home. It a contrast as compared to when he rst arrived in January
where he was feeling homesick but the thought of an impending end to his exchange in 3 more months created this con-
icting feeling of missing home but not ready to return home.
Although there were no further status updates referring to his departure in April and May, Nick started making sev-
eral references in June as he approached closer to the end of the exchange program. He clearly had very conicting
emotions where he felt that he did not want to leave his American family (See turn 6). He reected this in a status update
when he posted before I came, I was afraid. Now as my time here draws to a close end, my legs are trembling and my
lips quiver (turn 7). It is common to see exchange students having a strong sense of not wanting to return home as they
approach closer to their return back home as they have started to grown accustomed to their new environment.
Table 2: Status updates towards the end of Nicks stay abroad
Nick mood took a 360 degrees turn. I dont wanna go home yet.
Nick before I came, I was afraid. Now as my time here draws to a close
end, my legs are trembling and my lips quiver.
Nick I dont want to leave. I really dont want to leave right now.
Nick misses his american home already. Not joking. All this waiting is
killing me. Gah!
Upon returning home to Malaysia, Nick continued using Facebook to reect on his recent experience abroad by
posting his thoughts on Facebook. He was having mixed feelings when he realizes that his exchange has come to an end
and that the task of re-assimilating back to his home culture is something daunting. In Nicks next status update, he made
a very important realization that the task of re-assimilation is not as effortless as he thought it would be. Such reversal of
feelings is something that most exchange students experience upon returning back home and the reentry process back to
the home culture can be as difcult as the initial assimilation to the foreign culture.
And thus, this concludes the story of Nicks intercultural learning experience abroad of 6 months. While the intercul-
tural learning experience does not end after his return We can see how Nicks thoughts have changed and how he has de-
veloped intercultural competence over time. Initial status update postings such as those in turn 1 through turn 15 relates
to supercial cultural observations such as the weather, his feelings on how great it was to be there and places that he was
to visit. Towards the end of the program, his status update postings reected heavily on his experiences during his time
abroad. Although the statements made had negative emotions related to it, it was an important learning process that Nick
had to experience before any intercultural learning can occur. This is based on an intercultural learning perspective where
the ability to recognize cultural differences and make sense of the new knowledge is the most important aspect on how
learning can occur.
The central aim of this paper is to create an account of the intercultural learning experience through Facebook status
updates. To achieve this, we created the story through a careful selection of status updates posted by a participant on a
6-month intercultural exchange program called Nick. The story that was created is by no means a full account of the en-
tire experience but we demonstrated how a researcher could use these status update postings to interpret the intercultural
learning experience in a more naturalistic manner.
The creations of these digital artifacts in an informal space such as a SNS gives an alternative insight into a partici-
pants feelings and emotions that was once limited to using written accounts such as blogs and journals or oral accounts
such as interviews. As researchers interested in understanding the intercultural learning process, we view these digital
artifacts as a digital extension of their experience during their time abroad on the intercultural exchange program. Similar
to journals or blogs, postings on SNS can be used as an avenue for a participant to express their feelings. By analyzing
these digital artifacts over a period of time, researchers are able to gain insights into emotions and critical incidences that
are otherwise difcult to obtain through interviews or surveys. In the previous section, we described an example how we
created a quick snapshot into an exchange students experience of traveling to the United States, adjusting to the culture
there and adjusting to the culture back home by collecting and analyzing his status updates postings over a period of 7
From our analysis of over 3000 status update postings by 16 participants in another study, we observed that there is
a vast amount of information that is contained within Facebook and does not include other features such as notes, wall
comments or photo comments that could possibly yield a richer account of the intercultural learning experience. Al-
though status updates are short messages of up to 420 characters, we have demonstrated that it is possible for researchers
to obtain rich amount of information through the analysis of short text collected over a period of time. Comments from
friends and the interaction between them can further provide information that is similar to a recorded conversation. The
researcher can then use the research questions to focus on these status updates and comments to triangulate any ndings
that they may obtain from interviews, questionnaires or any other sources.
For example, researchers interested in understanding how social support affects the adjustment process in a foreign
culture for example our previous study (See Lee, Kim and Kim (2010)) where we described using status updates to gain
an understanding into critical emotional related incidences that may affect the intercultural learning process. A researcher
interested in understanding what role this social support play can use SNS as an avenue for their observation. Due to the
informal nature of this environment, a person is more likely to speak candidly about their problems and the ability to
crowdsource their problem to many people simultaneously makes it an attractive avenue for people to seek social sup-
While the use of SNS remains largely popular among the Net savvy generation, we would like to caution that not
everyone will be an active user of SNS. Although an individual may have an account on multiple SNS, it does not mean
that they would be actively updating the site with what they are doing. These passive users do not produce contents such
as status updates, sharing links or write notes. Such individuals use SNS to casually keep in contact with friends and
not use such sites to disseminate information. Due to the lack of updates or comments to friends, they may seem to be an
inactive user of SNS. A researcher using SNS should be cautious of the fact that although technology is an enabler, it can
be a barrier for those who are not technologically inclined as they may require more time to familiarize themselves with
the technology.
Despite us advocating the use of SNS as a research methodology, there are several concerns that researchers may
have to consider when using this new media. In the next section, we will discuss some of the issues researchers should
consider when using SNS as a tool for research.
In this paper, we described a novel approach in using social network sites to gain insights into the intercultural
learning process of participants on an exchange program. This study contributes to the growing literature on SNS that
we hope will have methodological implications in both the eld of computer mediated communication and intercultural
learning. We took into account an extensive discussion by boyd and Ellison (2008) on social network sites and proposed
a methodology for using SNS status updates to provide a rich ethnographic account using intercultural learning as a con-
text how this methodology can be applied,
The examples described in Section 3 of this paper provides several examples of the types of information a resear-
cher may obtain from analyzing the status updates. As compared to journal writing or interviews, the informal nature of
SNS also allows for the users of such sites to express themselves in a more relaxed manner and also provides a different
perspective. When triangulated with other sources such as journals or interviews, we can use these artifacts from SNS to
corroborate any ndings.
However, we observed that there are two groups of SNS users. One group are individuals who are not too comforta-
ble with sharing information online and on the other hand are individuals who have no reservations with sharing anything
and everything that comes into their minds. Therefore, we suggest that it is still important that analysis of SNS be used in
conjunction interviews, surveys or questionnaires in order to have a clearer understanding on the learning process of the
intercultural learner.
While the use of social media is relatively new, we attempt to propose the use of social network sites as a non intru-
sive research tool rather than as an intervention. With the increasing interest in the use of social media in research, we
have shown in this study how social network sites can be used as a source for gaining insights into intercultural learning
for participants on a student exchange program. The affordances within social network sites allow researchers the op-
portunity to tap into a resource that can potentially be a powerful research tool. This study demonstrated just one of the
many potential from analyzing status updates and comments posted by participants on an intercultural exchange pro-
gram. We also see SNS as one of the sources for gaining a better understanding on the intercultural development process.
There are also many other affordances within a social network site that a researcher can further tap into for their own
research purposes. What is clear is that researchers will be nding more novel approaches in using social media in their
research in the near future.
Anderson-Levitt, K. (2006). Ethnography. In J. Green, G. Camilli, P. Elmore, A. Skukauskaite & E. Grace (Eds.), Handbook of
complementary methods in education research (pp. 279-296). Washington DC: US: Lawrence Erlbaum Associates.
boyd, d., & Ellison, N. (2008). Social network sites: Denition, history and scholarship. Journal of Computer-Mediated Com-
munication, 13(1), 210-230.
Cushner, K., & Karim, A. U. (2003). Study abroad at the university level. In D. Landis, J. M. Bennett & M. J. Bennett (Eds.),
Handbook of intercultural training (3rd ed.). London:UK: Sage.
Ellison, N. B., Steineld, C., & Lampe, C. (2007). The Benets of Facebook Friends: Social capital and college students use
of online social network sites. Journal of Computer Mediated Communication, 12(4), 1143-1168.
Facebook. (2010). Facebook | Statistics. Retrieved November 29, 2010, from http://www.facebook.com/press/info.
php?statistics
Hall, H., & Davison, B. (2007). Social software as support in hybrid learning environments: The value of the blog as a tool for
reective learning and peer support. Library & Information Science Research, 29(2), 163-187.
Hansel, B. (2008). AFS long term impact study: Looking at intercultural sensitivity, anxiety, and experience with other cultures.
New York: AFS International.
Hernndez-Ramos, P. (2004). Web logs and online discussions as tools to promote reective practice. The Journal of Interactive
Online Learning, 3(1), 1-16.
Jonassen, D. (2008). Meaningful learning with technology (3rd ed.). Upper Saddle River, NJ: Pearson Education.
Koskinen, L., & Tossavainen, K. (2004). Study abroad as a process of learning intercultural competence in nursing. Interna-
tional journal of nursing practice, 10, 111-120.
Lee, J. W. Y., Kim, B., & Kim, M. S. (2010). An exploration on the use of Facebook as a tool for social support on an intercul-
tural exchange program. Paper presented at the Global Learn Asia Pacic 2010, Penang, Malaysia.
Lenhart, A., Purcell, K., Smith, A., & Zickuhr, K. (2010). Social media & mobile internet use among teens and adults. Retrieved
from http://pewinternet.org/Reports/2010/Social-Media-and-Young-Adults.aspx
Lin, H. L., & Yuan, S. M. (2006). Taking blog as a platform of learning reective journal Advances in Web Based Learning -
ICWL 2006 (pp. 38-47): Springer.
Mazer, J. P., Murphy, R. E., & Simonds, C. J. (2007). Ill see you on Facebook: The effects of computer-mediated teacher
self-disclosure on student motivation, affective learning, and classroom climate. Communication Education, 56(1), 1 - 17.
ONeill, E. J. (2008). Intercultural competence development: Implementing international virtual elementary classroom activities
into public schools in the U.S. and in Korea. University of Virginia, Virginia.
OReilly, T. (2005). What is Web 2.0: Design patterns and business models for the next generation of software. Retrieved Feb
20, 2009, from http://www.oreillynet.com/pub/a/oreilly/tim/news/2005/09/30/what-is-web-20.html
Park, C. (2003). Engaging students in the learning process: the learning journal. Journal of Geography in Higher Education,
27(2), 183-199.
Raja Hussain, R. M., & Ng, H. Z. (2010). A case-study: The adoption, adaptation and transformation of Facebook as eportfolio
in Higher Education. Paper presented at the Proceedings of Global Learn Asia Pacic 2010, Penang, Malaysia.
Spradley, J. P. (1980). Participant observation. New York: US: Holt, Rinehart and Winston
Zywica, J., & Danowski, J. (2008). The faces of Facebookers: Investigating social enhancement and social compensation hy-
potheses; predicting Facebook and ofine popularity from sociability and self-esteem, and mapping the meanings of popu-
larity with semantic networks. Journal of Computer-Mediated Communication, 14(1).
More than code: The signicance of social interactions in young people's development as interactive media creators 23
More than Code: The Signicance of Social Interactions in Young Peoples Development
as Interactive Media Creators
KAREN BRENNAN, AMANDA VALVERDE*, JOE PREMPEH*, RICAROSE ROQUE, AND MICHELLE CHUNG
INTRODUCTION: INTERACTIVE MEDIA CREATION AND SCRATCH
Think about all of the forms of interactive media that surround young people: video games, online social networks,
wikis, etc. Although they are engaged with these media forms, young people participate primarily as consumers of inter-
active media, rather than as producers of interactive media. Young people play video games, but dont design their own
games. They provide content for social network platforms, but dont construct their own infrastructure for exchanging
social information. They use services like Wikipedia and even sometimes contribute content to wikis, but dont question
the mechanisms that enable them to share their work online. The use of interactive media is important, certainly but
understanding and designing interactive media will increasingly become capacities necessary for understanding and ne-
gotiating our interactive media landscape.
Scratch (http://scratch.mit.edu) is a programming environment that enables young (and not-so-young) people to
create their own interactive media. Unlike most programming languages, which require the creator to type text-based
program instructions (such as Java, Python, C++), Scratch presents program instructions as visual blocks. Thus, just as
one can build physical creations using LEGO bricks, a creator can build interactive media by snapping together Scratch
blocks in stacks. Arbitrarily complex and highly diverse programs can be built up this way a choose-your-own-adven-
ture story, a maze game, a simulation of the effects of gravity on objects (Fig. 1).
Figure 1: The Scratch programming environment, screenshots from four sample projects (clockwise from upper left: an
interactive navigation through the layers of the earth, a LEGO model creator, a digital logic toolkit, a rescue game), and an
example of a stack of Scratch blocks.
In addition to the Scratch programming environment, there is a website the Scratch online community where
young people are able to share their interactive creations (Fig. 2). The online community, launched in May 2007, has
more than 680,000 registered members sharing, discussing, and remixing one anothers Scratch projects (Resnick et al.,
2009). Each day, members (mostly ages 8 to 16) upload approximately 1500 new Scratch projects to the website on
average, a new project every minute.
24 Brennan, Valverde Prempeh, Roque and Chung
Figure 2: The home page of the Scratch online community, where young people can share projects and interact with other
creators and creations.
Beyond enabling people to upload their projects, the site was designed with features typical of community-based
content-creation sites like Flickr and YouTube. Members can leave comments on projects, annotate projects with tags,
indicate admiration of projects by clicking the Love It link, and bookmark others projects in a list of favorites. Members
can also mark other members as friends, create galleries or collections of projects with others, and participate in discus-
sion forums. Each member has a prole page that displays their alias and country, as well as their contributions and inter-
actions lists of projects, favorites, friends, and galleries.
More than code: Studying interactive media creators development
Given the considerable number of young people who are engaged as interactive media creators with Scratch, we
were interested in exploring two key questions related to their engagement: How are young people participating and de-
veloping as interactive media creators? How can we, as researchers, represent this participation and development?
Our rst approach to responding to these two questions was to examine projects posted on the website. Each Scratch
members prole page is a portfolio of projects and we considered measuring development by examining the elements
of code (the programming blocks) that make up each project. By looking at projects over time, we could examine the
blocks that a Scratcher was using and determine whether they were creating more programmatically sophisticated arti-
facts. But we soon abandoned this approach, recognizing that a Scratch project is more than the sum of its programming
elements. With code as the primary unit of analysis, we would have ignored other important aspects of interactive media
creations, including diversity of creation, experimenting with aesthetics and genres.
We then considered a more holistic approach. Inspired by the work of Barron (2007) and of Plaisant (1996), we
worked on developing individual Scratcher timelines or biographies. We hoped that a timeline/biography approach would
better represent the process of interactive media creation, by examining creators project notes and discussions around
the artifacts, in addition to more quantitative, product-oriented measures of participation. But as we studied projects,
notes, comments, and forum posts, it became clear that an individual Scratcher was too narrow as a unit of analysis. The
complex interactions between Scratchers, which were signicant for supporting each others development, made the ex-
amination of individual Scratchers challenging from a research perspective.
The central problem was that our code-only and our more holistic portfolio approaches both insufciently acknowl-
edged the social nature of participation on the Scratch website. Inspired by Paperts Mindstorms, the website draws on a
samba school model, where people of all ages come together with shared goals, to support each others learning and col-
laborate on endeavors that are more substantial than could be achieved individually (Papert, 1993). More broadly, theo-
ries about communities of practice and situated learning have provided ways of thinking about how community settings
can support the learning of a practice by providing learners access to others and opportunities to explore the activities,
artifacts, and ideals of the practice (Brown, Collins, & Duguid, 1989; Lave & Wenger, 1991). Given the social intercon-
nections, trying to trace an individuals development as a Scratcher became an increasingly complex task - so much of an
individual Scratchers development was dependent on how others helped them, and how they in turn helped others. Other
members were so central in the development of Scratchers who participate in the Scratch online community that our key
research question became: How do others support an individuals development as a creator of interactive media with
Scratch?
The role of others: Case studies of participation
In responding to our research question, we adopted a predominantly qualitative approach. The study took place over
a four-month period. For two months, the ve members of our research team conducted ethnographic observations of the
online community. We rst looked at Scratch as a whole, taking note of how users interacted with various aspects of the
website, mapping an individuals potential navigation pathways. These general observations allowed us to see the various
ways that each Scratcher could interact with the site, including (but not limited to) learning skills from posted tutorials,
reading posts on the discussion forums, and browsing projects highlighted on the homepage.
Based on these initial ethnographic observations, we segmented the Scratch population (based on age, geographic
location, duration of participation in the community, magnitude of participation, visibility of participation, and project
genre and sophistication) and selected a sample of 10 Scratchers to study in greater depth to achieve coverage of these
different segments.
We created user proles for each of these Scratchers, which served as comprehensive summaries of their different
modes of participation. We looked at a broad spectrum of activities for each user, cataloging what types of projects they
made and with what frequency. Additionally, in order to have a more complete view of each user, we expanded our view
beyond an individuals creations to see how they connected with other users. We observed the channels of communica-
tion available on the site, including feedback in comment postings, as well as forum discussions where users asked for
programming assistance, got to know other users with similar interests, and interacted with others to form collaborative
work groups.
Over the next several weeks, we conducted hour-long interviews by phone or Skype with six of the ten Scratchers
that we had proled. These six were selected based on two factors: (1) achieving satisfying coverage based on our initial
coding of the ethnographic observations and proles, and (2) availability of particular Scratchers to participate in the
interviews. The interviews were conducted in researcher pairs and were semi-structured. Examples of questions that we
asked in the interviews include: How did you get started with Scratch? What do you do with Scratch? Why do you use
Scratch? Throughout the four months of the study, the research team met on a weekly basis, to share and discuss our data
collection and analysis experiences. These meetings served as inter-coder reliability checks, making sure that the analysis
was approved by all research team members. Approval for this study was granted by the universitys ethics committee
and all interview participants gave informed consent.
Through the process of coding our observations, proles, and interviews, we identied particular roles of participa-
tion that were recurrent in our data. There was not one way in which others supported a Scratchers development, but
rather a cluster of symmetric roles (roles in which the individual and others were more equal, such as collaborator) and
asymmetric roles (roles in which the individual and others had differentiated roles, such as teacher or moderator) that
members transitioned between at various points in time during their development as interactive media creators.
We present our analysis of roles here as a series of six cases studies. Each case study represents an archetype of par-
ticipation: The Newcomer, The Remixer, The Collaborator, The Teacher, The Moderator, and The Contributor. The case
studies are centered on individual Scratchers, but emphasize their relationships in the social context of the Scratch online
community and how that social context has supported their development as interactive media creators.
The Newcomer: Kylie, 9, Australia
Well, its just that there are endless possibilities. Its not like you can just make this project, or this project, and
thats all that you can make.
Kylie is a precocious 9-year-old home-schooled by her mother, passionate about her four pet guinea pigs, and
dreams of one day owning a caf or being a piano teacher. She is an avid Scratcher. Although Kylie is a relative new-
comer (having only been a member of the Scratch community for 15 weeks), she has been a prolic creator of projects,
uploading more than 240 projects to share with others in the community. She visits the Scratch website every day to
browse through the messages left for her by other Scratchers, to interact with new projects that others have posted, and to
share her latest Scratch creations.
Kylie was shown the basics of Scratch programming by a friend, and she was excited about the possibilities of
expressing herself creatively in this new medium. This creative excitement is demonstrated in the diversity of her proj-
ects: an interactive random-number-generating dragon, a role playing game about virtual pets, an interactive encyclope-
dia project that lists facts about guinea pigs, and dozens of community-organized coloring contest entries. She regularly
explores others projects, downloading and learning from the work created by other Scratchers. She values access to the
collective experience of other users, saying that you can search for what it is you are stuck on and have a look at other
peoples scripts.
A few weeks after joining the community, one of Kylies projects was selected by a Scratch community administra-
tor to be featured on the website home page. Being featured enabled Kylie to be introduced to a greater number of com-
munity members and to get more feedback on her work, both of which supported her deepening engagement with the
community and her development as an interactive media creator. Kylie appreciates these benets, but she is not primarily
motivated by peer recognition or acknowledgment from site administrators although she recognizes that others are.
When asked why some Scratchers abandon the site after only a brief period of time, Kylie suggested that it was due to a
lack of visibility: Some people [leave] for popularity but I dont think thats a good reason to quit.
Having her project featured allowed her to become connected with a wider array of users and added to her enjoy-
ment, but has not ultimately dened why she is such an involved creator. She is passionate about her ideas and about
creatively expressing those ideas with Scratch; she is enthusiastic in her explorations and continually seeks out new ex-
periences. It is not important to her to be popular, but she recognizes the value of being connected to others. When asked
whether she thought that it would be as fun to work on Scratch if she could not communicate with her peers on the site,
Kylie invoked the Scratch online community motto probably not because its imagine, program, share. You have to
share what you think.
The Remixer: John, 46, Eastern Europe
For my 9-year-old daughter, I know the need for such projects. Thank you for your support.
John rst joined Scratch to nd out why it was so appealing to his daughter. 249 projects, 1598 comments, and 111
friends later, he has established himself as a contributing member of the Scratch community. In his rst month of using
Scratch, John familiarized himself with the software and learned how to make simple projects. He realized that he could
combine his love of art, math, and science using Scratch. He created projects that demonstrated patternmaking based on
fractals and symmetry, and his colorful, interactive creations attracted positive reactions from Scratchers of all ages. John
responded graciously to all of the positive feedback, taking the time to respond appreciatively to almost every comment,
saying thanks, thank you for stopping by and commenting and you are welcome.
Over time, in addition to interacting with Scratchers through comments, John interacted with Scratchers through
remixing, a practice of downloading a project and editing it. When he learned how to animate characters in stories and
games, many Scratchers downloaded and remixed his interactive projects. He supported remixing by exchanging proj-
ects with his daughter and by remixing other Scratchers projects, encouraging it as interesting to see the same game
done so differently by so many authors. Scratchers acknowledged and reciprocated Johns sincere and caring approach.
One of his projects was nominated by a young Scratcher to be featured on the front page of the website. Another project
achieved Top Viewed status on the website, with 1407 views from other Scratchers, and John was honored to receive
such recognition from the community.
John uses remixing not only as a way to create projects, but as a powerful metaphor for the way he thinks about
learning that people construct knowledge based on what they already know and through explorations of the world. His
approach is to not teach by transferring them your knowledge, instead you should help them to explore the dynamic
world by themselves. He designs projects that support this type of intellectual remixing through interactivity:
I could not resist making this project after reading an article on Magnetic Resonance Imaging (MRI).
The intended audience is those who want to learn the physics of MRI. I have MRI images, but I did not
want to use pictures in the project with no interactivity.
Young members are not the only Scratchers John interacts with other adult Scratchers interested in using Scratch
as a pedagogical tool have also made connections with him. Together, they collaborate on group projects via remixing
and they exhibit their work in galleries on Scratch. With these connections, John hopes to spread Scratch to his local
community, and he is working to integrate Scratch into his daughters school by creating an online gallery to showcase
students projects.
The Collaborator: Rachel, 14, Northeastern United States
Im trying to work with more people online to learn more from them, and see if we can put our work together.
For some time, Rachel had been intrigued by programming, and was thinking about pursuing a career in game de-
sign. When her father heard about Scratch, he introduced Rachel to the Scratch programming environment and she im-
mediately dove into project creation. Her rst project, a simple cartoon, inspired her to create a whole series of ani-
mations. Rachel received positive comments and helpful suggestions on the projects in her animation series, and this
feedback motivated her to want to improve her skills and create more elaborate projects. But this was difcult to do by
herself, and she began searching for possibilities to work with other Scratchers.
Rachel found inspiration in PowerCo., a small, international group of Scratch members, ages 9 to 15, who had col-
laborated to design games with Scratch. PowerCo members promoted themselves as a Scratch company and assigned
themselves to different company roles, such as president and lead programmer. Rachel saw how the PowerCo group was
able to create projects of greater complexity than any individual member would have been able to create, as each member
brought a unique perspective and set of capacities. Rachel thought that starting her own Scratch company would be an
exciting way to further develop as a creator of interactive media.
Rachel recruited two other young Scratch members to start their own Scratch company, which they named Green
Flower Productions (GFP). GFP posted their plan to the Scratch forums and invited others to participate in the cre-
ation of a project to celebrate Halloween. Another Scratcher suggested creating an interactive project that has the player
navigating a spooky old mansion. Rachel and the other members of GFP loved the idea and the group started working
on the plot of the story. They created an initial draft of the story and posted a link to the project in a forum thread. Other
Scratchers were excited about the project and volunteered to help out some were interested in working on the plot,
others the programming, others the art. People working on the project downloaded the latest version, worked on it for a
bit, and reposted it to the site, iteratively building up the project. On the day before Halloween, the group of contributors
(which at its peak involved more than 20 community members) announced a nal version. Community members gave the
creators positive feedback on this project a project that would have been challenging for any one of them to create on
their own.
Rachels development as an interactive media creator was supported by her participation in GFP. While program-
ming a GFP project, Rachel would occasionally encounter frustrating programming challenges. She found support from
her fellow company members, who helped her move past these difcult situations. And Rachel passes on the support she
received from working with others I like helping other Scratch members. If they need help with a drawing or a story
idea, I can help. She reached out to new users to help them nd answers to their pressing questions. Her contributions to
Green Flower Productions have made her a recognized member of the community, and many Scratch members consult
Rachel for her ideas and feedback, which Rachel happily spends time contributing.
The Teacher: Rebecca, 16, Southwestern United States
I get a lot of comments like thanks for teaching me this or that is so cool, I am going to show my science teacher. That
makes me happy.
Rebecca is midway through her junior year of high school. She is passionate about math. When she shared her love
of geometric proofs with her father, an electrical engineer, he mentioned that the logic of proofs is mirrored in computer
programming. This insight sent Rebecca off to nd a way to learn more about programming, and with a little help from a
search engine, she found Scratch.
Rebecca quickly learned that she could express her interest in math by creating projects with Scratch. The rst proj-
ect that she posted was designed to teach others about the quadratic equation, and she immediately connected with two
other Scratchers who also produced math-based projects. Since then, Rebecca has explored how Scratch can be used to
teach others about a wide variety of math concepts variables, parabolas, prime numbers, and hyperbolas. Her collection
of math-based projects connected her to a sub-community of Scratchers who are fans of math-focused Scratch projects.
Her relationships with these Scratchers have helped her to grow as a mathematician and as programmer.
After learning to program using the visual blocks in Scratch, Rebecca experimented with other, text-based program-
ming languages, such as Python. In discussing the similarities and differences between using Scratch and Python, she
identied the Scratch community as the most important difference. Rebecca understands how valuable connections with
others have been to her development as a creator of interactive media and wants to help other Scratchers make those
connections. She takes the time to view and comment on the projects of new users, who may not yet feel like they are
part of the community:
I really like the community and that everyone is willing to help out each other and give feedback. And so
I like helping people and I like putting my stuff up on the site so that other people can see it.
After successfully teaching others about math on Scratch, Rebecca took her newly acquired talents to a program-
ming camp where she was able to tutor others on the Scratch visual language. She has continually shown her afnity for
teaching, and sincerely enjoys helping others learn what she knows. Rebecca said it best they say that teaching is the
highest form of learning or understanding. So I think that making math projects has actually helped me understand the
concepts better than learning in school.
The Moderator: Jake, 15, Canada
What denes the Scratch forums is its a community that is welcoming to everyone and its really nice. Its appropriate
for all ages.
Jake was 12 years old when he rst encountered Scratch at a computer science camp hosted by his local university.
He was excited to create games and to share them with others on the Scratch website. He particularly enjoyed spending
time in the Scratch website discussion forums. In the forums, he could socialize with other Scratchers, advertise his proj-
ects to get more feedback, and even make suggestions for the next version of Scratch. As he became more experienced
with Scratch, he was able to help Scratchers with their questions or programming projects. He enjoyed assisting mem-
bers of the Scratch community, and he tried to think about other ways that he could help. After participating in the online
community for a couple of years, he decided to become a Scratch community moderator.
In his early experiences as a Scratcher, Jake looked up to the community moderators, who were very knowledgeable
about Scratch and helpful to the community. Community moderators are Scratch users, elected by the community, who
work with the Scratch website administrators in answering questions from users, explaining the values of the community,
and helping keep the Scratch website forums friendly. Jake saw this role as a way to help out at a larger scale saying,
Scratch has already given so very much to me and it would be an honor no, a privilege to give something back to it.
After being elected by the community, Jake was eager to get started. But he soon discovered complexities of the role
that he had not seen as a Scratch member. Before, he would answer questions to which he knew the answer or explain
Scratch ideas that he already understood. As a community moderator, he now became involved in handling questions to
which the answer was unclear and was faced with ideas that were totally new to him. As a new moderator, he learned to
rely on the more experienced moderators and the Scratch team to navigate his new role and responsibilities.
I learned quickly that if I didnt know what to do in a situation that I shouldnt just guess, I should leave
it up to the pros: the older moderators and Scratch Team[they] guided us those rst few months on
what to do in certain situations.
Over time, Jake has developed his condence, skills, and intuition. When trying to answer community members
questions or think about community issues, Jake has found it helpful to put himself in the mindset of those involved, con-
sidering how they might feel or react to his actions. He has also learned the importance of teamwork in his role, working
with the moderators to talk through ideas and working with the community to help maintain a friendly and welcoming
environment. With two new moderators recently elected, Jake now helps them to develop their condence, skills, and in-
tuition, as part of their orientation. Jake continues to be an active member of the Scratch community and the moderators
group, and feels positive about his contributions. Its just nice to be doing a good deed, volunteering for the community
and I really think Scratch is a great program and community.
The Contributor: Andrew, 12, Northeastern United States
[My goal] is similar to why the creator of Wikipedia created Wikipedia: to create a full source of human knowledge. In
my case, its that, except about Scratch.
Andrew was 9 years old when his father (a computer programmer) introduced him to Scratch. Wanting to be like his
father, Andrew started creating projects. But as he got started with Scratch, Andrew sometimes found it difcult to trans-
late his ideas into actual projects. He initially looked for support by studying the scripts from Scratch projects made by
other users, nding projects that he wanted to make and learning particular techniques and tricks. He also found support
in the Scratch discussion forums, where other Scratchers helped him with problems and answered his questions.
As much as he appreciated the help he received through the forums, he found its organization tedious it was too
difcult to nd answers to particular questions, as posts were sorted chronologically rather than thematically. While
searching for other resources that could help him use Scratch, Andrew discovered an unofcial wiki about Scratch.
Young members of the Scratch online community had developed this unofcial Scratch resource with the mission of
sharing helpful information about Scratch, such as programming procedures and hidden features. Andrew recognized
the wiki as a vast, powerful resource for members to share information, and knew that other members could benet from
having the wiki as a direct resource on Scratch.
Andrew began lobbying for an ofcial wiki on the Scratch website, arguing its signicance in his development as a
Scratcher: at rst, [Scratch] was pretty much a foreign world. If I had [had] the Scratch wiki [when I rst started], that
would have helped. After debating the decision for months, the Scratch website administrators nally decided to build
an ofcial Scratch wiki, and enlisted Andrew to perform several important tasks. He helped with the time-consuming
task of manually importing the old, unofcial articles into the new, ofcial Scratch wiki. He communicated with many
Scratchers in the community, taking polls to further understand how Scratchers nd help when they are in need of an-
swers. Furthermore, he spent several hours discussing the wiki with the Scratch administrators, participating in complex
and nuanced discussions about the nature of useful knowledge.
The Scratch wiki has been a major ongoing project for Andrew, requiring months of commitment and effort. Moti-
vated by the appreciation expressed by the community for the offcial Scratch wiki, he shows no signs of slowing down.
When asked about the future of the wiki project, he describes detailed plans to improve the Scratch wikis design for bet-
ter navigation, to make information access even easier for visitors. In addition, he tirelessly encourages other Scratchers
to contribute to the wiki, and strives to make the wikis contribution process easier for new users, so there [will be] a
low foor [to participating].
Imagine, program, share: Developing as a computational creator
Learning in a social setting involves the learner enacting a variety of roles that change over time (Brown, Collins, &
Duguid, 1989; Rogoff, 1994). This is evidenced by the case studies presented there is no single educator role and there
is no single student role. There is no simple apprenticeship structure, with a learner moving from newcomer to old-timer;
we see individuals who are deeply committed and invested in their activities as creators of interactive media, geeking
out on the possibilities of creation (Ito, 2009; Lave & Wenger, 1998).
Within this shared deep commitment to creation, there are gradations and we see a trajectory of developing rela-
tionships to others in the community across the six case studies. In the rst three case studies, participation is primarily
dened through action creating projects, nding colleagues and friends, collaborating on work. We began with Kylie,
the newcomer, who relies on the community as an outlet for expressing her creative possibilities. She shares her passions
with the community through her projects and comments, and is motivated by others responses to her work. John, the re-
mixer, relies on the community as a way to build up new ideas and new creations. Rachel, the collaborator, is motivated
by interactive collaborations, as she discovers that her creative work ourishes when partnering with others.
In the nal three case studies, we see a shift from an action-orientation to a reection-on-action-orientation think-
ing about what it means to be an interactive media creator and how to support that development (Schn, 1984). Rebecca,
the teacher, helps others learn particular content-area concepts, and is thoughtful about providing other members with
valuable feedback on their work as part of their development. Jake, the moderator, wants to ensure that the positive so-
cial interactions experienced by Kylie, John, and Rachel are made available to everyone in the community. Andrew, the
contributor, wants to establish support structures beyond those put into place by the designers of the online community.
This reection-on-action is something that occurs in part with more experience, but we are interested in further studying
what conditions in addition to more experience are necessary.
More broadly, we are interested in the implications of the experiences documented in the case studies for other
learning environments, particularly formal learning environments like classrooms. But there are limitations to the gen-
eralizations that can be made from the case studies. First, the case studies are intended to present the possibilities of
developing as a creator not every person who encounters Scratch and the Scratch website enjoys the successes of the
learners described in this paper. Second, unlike many formal learning environments, it is easy for someone who is not en-
gaged or connected to the online community to fade away; in a classroom, however, unsuccessful learners remain in the
learning environment. Finally, many of the case studies described emergent behavior interactions that were not planned
(or, even, in some cases, imagined) by the designers of the site and appeared with minimal scaffolding beyond the tech-
nological infrastructure of sharing and commenting on projects afforded by the website, as described earlier. Despite
these limitations, we see value in thinking about which elements of the experiences are common across the case studies
and how those elements can be used to inform design of meaningful learning experiences for a broader cross-section of
learners.
The motto of the Scratch project imagine, program, share which appears at the top of every page in the online
community, serves as our framework for discussing aspects of the experiences that are shared across the case studies
and how these aspects could be brought into other learning settings to support young peoples development as creators of
interactive media. First, in each of the case studies, the Scratcher had freedom to imagine what they might create. Wheth-
er it was connecting a personal passion for guinea pigs or a desire to create an elaborate game, each case study revealed
an imagining and exploring of personal interests. Whenever possible, creating opportunities for young people to express
themselves through an activity is an element for success because it so effectively motivates authentic participation (Buck-
ingham, 2007; Illich, 1971).
Next, the Scratcher had freedom to program or create in all of the case studies. Young people have insufcient op-
portunities to think and act like designers in learning environments, despite the powerful opportunities for learning
that are offered by design-based activities (Barron, 1998; Kolodner et al., 2003; Resnick, 2006). This is the core of the
Scratchers activities and of a constructionist approach to learning, which emphasizes the importance of building, mak-
ing, and designing:
Constructionism builds on the constructivist theories of Jean Piaget, asserting that knowledge is not
simply transmitted from teacher to student, but actively constructed by the mind of the learner. Children
dont get ideas; they make ideas. Moreover, constructionism suggests that learners are particularly likely
to make new ideas when they are actively engaged in making some type of external artifact be it a
robot, a poem, a sand castle, or a computer program which they can reect upon and share with others.
(Kafai & Resnick, 1996, p. 1)
Maintaining this emphasis on design and construction is sometimes undermined in formal learning environments, if
a teacher dictates the particular design activities or treats Scratch as a tool, with students completing quizzes involving
labeling parts of the programming environment. Creating opportunities for young people to engage in the design of inter-
active media will have a cultural resonance, empowering them to create the artifacts that they so frequently consume, as
well as better position them to negotiate the reality of living in the 21
century.
Finally, in each of the case studies, sharing was an essential aspect of each Scratchers development as an interactive
media creator. We know that creativity is a social process something that resides not solely in the individual, but occurs
through interaction with others (Csikszentmihalyi, 1997; Sawyer, 2006). From imaginative impetus to collaborative cre-
ations to appreciative audiences to reective reactions, the case studies demonstrate how creative work situated in social
settings has the potential to positively impact creative output, as feeling that one is contributing something to others
appears to be especially motivating (Bransford, 2000, p. 61). By designing learning environments so that young people
have opportunities to participate in various collaborative congurations, educators are enhancing student motivation and
expanding opportunities for their students learning at all stages of the interactive media design process.
Participating as a creator of interactive media is culturally relevant and is an important capacity for young people to
be developing but it is challenging. It can be made easier, however, by providing creators with access to a social context
in which others are also participating as interactive media creators. The case studies presented here show the possibilities
for development that can take place in a social context. As poignantly described by Jake, when asked about his rst few
days as a Scratcher compared with his participation now several years later, I would have probably left after a couple of
days if I hadnt [made a connection to others], [but now] I think theres nothing that would make me want to quit.
Barron, B., Schwartz, D., Vye, N., Moore, A., Petrosino, A., Zech, L. Bransford, J.D., & Vanderbilt, T. (1998). Doing with un-
derstanding: Lessons from research on problem- and project-based learning. The Journal of the Learning Sciences, 7(3&4),
Barron, B. (2007). A multiple case study on middle school learners technological uencies across different learning ecologies.
Proceedings of CSCL07, New Brunswick, NJ.
Brown, J.S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32-
Buckingham, D. (2007). Beyond technology: Childrens learning in the age of digital culture. Malden, MA: Polity.
Csikszentmihalyi, M. (1997). Creativity: Flow and the psychology of discovery and invention. New York: Harper Perennial.
Illich, I. (1971). Deschooling society. New York, NY: Harper & Row.
Ito, M., Baumer, S., Bittanti, M., boyd, d., Cody, R., Herr, B., et al. (2009). Hanging out, messing around, geeking out: Living
and learning with new media. Cambridge: MIT Press.
Kafai, Y.B., & Resnick, M. (Eds.) (1996). Constructionism in practice: Designing, thinking, and learning in a digital world.
New Jersey: Lawrence Erlbaum.; Papert, S. (1991). Situating constructionism. In S. Papert & I. Harel (Eds.), Construction-
ism. Cambridge, MA: MIT Press.
Kolodner, J.L., Camp, P.J., Crismond, D., Fasse, B., Gray, J., Holbrook, J., Puntambekar, S., & Ryan, M. (2003). Problem-based
learning meets case-based reasoning in the middle-school science classroom: Putting Learning by Design
into practice.
The Journal of the Learning Sciences, 12(4), 495-548.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge: Cambridge University Press.
Papert, S. (1993). Mindstorms: Children, computers, and powerful ideas (2nd ed.). New York: Basic Books.
Plaisant, C., Milash, B., Rose, A., Widoff, S., & Shneiderman, B. (1996). LifeLines: Visualizing personal histories. Proceedings
of ACM CHI 1996, Vancouver, BC, Canada.
Resnick, M. (2006). Computer as paintbrush: Technology, play, and the creative society. In D. Singer, R. Golikoff, & K. Hirsh-
Pasek (Eds.), Play = learning: How play motivates and enhances childrens cognitive and social-emotional growth (pp.
192-206). Oxford: Oxford University Press.
Resnick, M., Maloney, J., Monroy-Hernandez, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J.,
Silverman, B., & Kafai, Y. (2009). Scratch: Programming for all. Communications of the ACM, 52(11), 60-67.
Rogoff, B. (1994). Developing understanding of the idea of communities of learners. Mind, Culture and Activity, 1(4), 209229.
Sawyer, K. (2006). Explaining creativity: The science of human innovation. Oxford: Oxford University Press.
Schn, D.A. (1984). The reective practitioner: How professionals think in action. New York: Basic Books.
Supporting Collaborative Web-Based Education via Annotations 33
Supporting Collaborative Web-Based Education via Annotations
MARIN IMKO, MICHAL BARLA, VLADIMR MIHL, MARO UNK, AND MRIA BIELIKOV
Efcient education is a cornerstone of knowledge society. The key for further development of the knowledge society
is technology-enhanced learning, allowing for anytime, anywhere access to learning materials. However, even if e-learn-
ing has many successful applications, the traditional in-class learning is still much superior way of education. The reason
is in an implicit presence of two key elements: adaptation and collaboration, which are hard to implement in virtual
learning environments. A good teacher adapts the explanations according to a feedback provided by the students (either
to explicit one, when somebody raises his hand and asks a question, or implicit, recognizable from students faces and
reactions). A good teacher chooses the right (i.e., personalized) questions to determine how well a particular student mas-
ters a particular topic. However, the adaptation is almost always targeted to the whole class at once, usually considering
the performance of majority of present students. This one-size-ts-all approach must inevitably fail for advanced stu-
dents, who nd the teachers explanations slow and boring, as well as for weak students, who cannot keep pace with the
majority of the class. Collaboration between students of a class can be also achieved very easily, when it is adequately
supported by a teacher, e.g. by provoking a discussion between students themselves in order to make a common agree-
ment about a particular problem (Alavi 1994).
The challenge is to incorporate these key elements of traditional in-class learning into web-based learning, which
is currently the most popular way of technology-enhanced learning. Adaptive web-based learning occurs when an edu-
cational system adapts to user needs, goals and preferences (Beaumont & Brusilovsky 1995). Learning becomes more
efcient and a learner is able to learn better or learn faster (Weibelzahl & Weber 2002). Hypertext-like nature of web-
based education enables to overcome the mentioned one-size-ts-all problem of an in-class learning by tailoring the
presented content to a particular learner. Personalization effect is often delivered in a form of recommendations or course
sequencing that is based on techniques related to adaptive presentation and adaptive navigation support (Brusilovsky
1996). These techniques are based on a semantic descriptions of a subject domain (often forming a concept map) also
referred to as a course metadata, and a learner model storing students characteristics in relation to a domain model
(course metadata).
We can already see efforts to incorporate aspects of collaboration into virtual learning environments. The emergence
of web 2.0 principles such as user-oriented authoring (blogs, wikis), knowledge sharing and organization (annotating,
tagging, discussing), and collaboration (instant messaging, social networks) reect into learning as well. A web user
(learner) is no longer considered to be a content consumer, but is given an opportunity to participate in the content cre-
ation and enhancement. Web 2.0 principles together with advancing web-based technologies improve overall user experi-
ence during learning by offering interaction, active participation and more competences. Besides greater autonomy for
the learner, the traditional role of a teacher changes and distinction between teacher and student blurs (Downes 2005).
The need for collaboration led to development of collaboration-supporting components or services for LMS (Meccawy et
al. 2008) or attempts to extend existing learning standards (Ip & Canale 2003). Technology is leveraged in order to shift
traditional individual learning towards collaborative learning (Dillenbourgh 1999, Tvaroek 2011).
Sustainability of the learning content quality, as one of the major bottlenecks of state-of-the-art educational web-
based systems, is gaining popularity as we witness an incredible growth of learning materials available on the Web. The
content may (and often does) contain errors that prevent from smooth learning. It is becoming heterogeneous and this
reects into various levels of appropriateness for a learner. For example, some parts are less and some parts are more
difcult. In order to improve the quality of learning content, obvious question arises: How to develop and maintain the
content, which can serve for better learning? How to handle big amounts of learning material? The situation is even
worse when considering technology enhanced learning such as adaptive learning that relies on rich domain descriptions
(metadata). Conceptual description of a domain contains e.g. when considering concept map hundreds of concepts
34 imko, Barla, Mihl, Unk, Bielikov
(domain knowledge elements) and even thousands of relationships (imko & Bielikov 2009). Teachers are able to create
a content, but they do not have enough time and space to dene metadata. We believe these issues can be to some extent
addressed by collaborative learning systems, where learners themselves can utilize a concept of annotations during learn-
ing. In this context we see annotations as a feasible tool for a domain model enrichment not only by adding the content
(user-created exercises, questions, etc.), but also by providing additional content descriptions (tags, comments, etc.). Fur-
thermore, the potential of annotations exceeds the issue of content quality improvement; it also affects the motivation in
learning. Enriched and interactive content can help to get and keep attention. We believe that visual and competitive at-
tractiveness of annotations can increase learners motivation resulting into improvement of learning performance.
In this paper we focus on annotation as a concept and tool for collaborative educational support. We point to its im-
portance in the context of adaptive and collaborative web-based education. First we describe annotation as a whole. We
discuss various types of annotations, their particular goals. We analyze annotations from user perspective: we cover user
interfaces and usability. We emphasize the role of motivation within collaboration. We show how we integrated annota-
tions into ALEF, adaptive learning framework aimed to improve learning efciency and present results of several experi-
ments conducted with selected annotation tools.
COLLABORATING BY ANNOTATING
Collaboration Flow in Learning
Collaborative learning can be understood in various ways. In this paper we adopt the broader denition, where col-
laborative learning is viewed as a situation in which two or more people learn or attempt to learn something together
(Dillenbourg 1999). Learners work together to search for understanding, meaning or solutions to particular problems.
In our previous work we dened a concept of adaptive web-based learning 2.0 that merges adaptive learning with
emerging concepts of Web 2.0 (imko et al. 2010). Considering such an environment, we distinguish two different
groups of activities: learning ow and collaboration ow (Fig. 1). Although the ows are separated in the gure, in real
world situations they occur simultaneously a student can both learn and collaborate. The learning ow covers learn-
ing from study material delivered by a personalization engine a personalizer module adapts the content (conceptually
described by a domain model) to a particular user based on his actual knowledge, goals and needs (represented by a user
model). Semantic logging and user model inferencing are aimed to collect and process data related to the usage of a sys-
tem from visiting a page to providing explicit feedback and to update a user model.
Collaboration ow starts similarly adapted content is presented to student, who can collaborate on improvements
and enrichments of the content and metadata during learning sessions by using a number of collaborative adaptive con-
tent creator modules. Modules form the core of collaboration ow providing means for a collaboration support (creation,
sharing or update of annotations).
collaborative adaptive
semantic logger
user model
inferencer
personalizer
Figure 1: Activity fows during learning (imko et al. 2010).
We log student actions and derive semantics of those actions within each collaborative adaptive content creator mod-
ule. The important aspect is evaluation of student behavior by analyzing logs in order to understand student motivation
for actual creation of the annotation. Thus, the advantage of the course content annotations lies not only in the improve-
ments of domain model quality, but also in more accurate estimation of user interests and current knowledge reected in
her user model.
Annotation framework
We implemented the collaborative ow within our Adaptive Learning Framework (ALEF). We put an emphasis on
the design of annotations within the framework in order to allow for a straightforward creation of diverse types of anno-
tations to enable and facilitate rich and multi-purpose participation and interaction.
ALEF's annotation framework was designed with respect to reusability and extendibility. In order to achieve it, we
designed a common representation of content and annotations within the system. We see content and annotation as one
entity - Resource (Fig. 2). Resources are connected with Relationships of various types determining their semantics. We
support a general purpose relationship Annotates, that can be further specialized. Such a representation allows us to as-
sign annotations to content, but even annotations to annotations. For example, this way we represent comments replies
resulting into a discussion thread.
Resource Relationship
LearningObject Blog
Annotates
Concept Tag Comment
Subsumes
Figure 2: Extendibility of resource annotations. Resources can be assigned general purpose Annotates relationship that
can be further specialized according to the specifc needs.
Creating and Accessing Annotations User Perspective
We recognize two different types of annotations from a conceptual point of view: per-text-annotations and per-con-
tent-annotations. The former is related to a specic part of text - e.g., a word, a phrase, a paragraph, which was selected
during an annotation creation. The latter is related to a specic content as a whole - a question, an exercise, etc. Per-con-
tent annotations are typically created from the outside of the learning content using dedicated widgets.
Every annotation has two facets: content and context. Content is the actual payload of an annotation, inserted by
a student, for instance a comment or URL of an external source. Context holds information related to association (bind-
ing) of an annotation to the learning object and optionally also to the text, where the annotation has been originally in-
serted by a student, if it is a per-text-annotation. Having two distinct facets, we provide students with two different types
of navigation among annotations: access-by-context and access-by-content. The former is applicable only for per-text-an-
notations and is used to access an annotation within its context, i.e. in the text while reading it. The latter allows students
to browse and view annotations separately from the text.
To create and access both content and context information of annotations, we designed four distinctive user interface elements:
- in-text interaction and presentation,
- sidebar,
- annotation browsers,
- annotation lter.
In-text interaction and presentation provides visualization and access of per-text annotations in their exact positions.
To insert an annotation, a student selects part of a text and uses an in-text menu, which pops-up above the highlighted
text (Fig. 3, left), chooses a type of annotation and inserts the actual content. Once an annotation is inserted, the selected
text remains highlighted to indicate presence of an annotation, which can accessed by simply hovering the mouse over
that text. The content of the annotation is shown in a popup window (Fig. 3, right). In-text interaction and presentation
represents the fastest access to annotations with no signicant interruption during learning.
Figure 3: In-text menu for inserting new annotation and displaying an annotation (content in Slovak).
The drawback of in-text presentation of annotations is that the text become almost fully highlighted and consider-
ably less readable as the amount of inserted annotations grows over time. We handle this situation by the annotation side-
bar, which displays aggregations of regions with many annotations.
Sidebar provides an access-by-context navigation by visualizing annotated regions aside of positions of affected an-
notations within the text (Fig. 4). Regions on the sidebar are interactive; student can click on a region and view a list of
all annotations within it. In-text style of visualization is used only when student interacts with certain annotation from the
list, e.g. to view or edit its content.
Annotation browser provides the access-by-content navigation by listing all annotations related to the currently dis-
played learning object so students can easily read content of annotations regardless of their position within the text. Se-
lection or interaction with an annotation inside the browser invokes in-text visualization to indicate context of an annota-
tion, if any. Annotation browsers are implemented as widgets located on the right-side of the screen, not distracting from
the main text in the central part.
Annotation lter allows users (students as well as teachers) to select which types of annotations they would like to
have visible, e.g. to see only reported errors while xing them. The lter of displayed annotations is a part of adaptability
of the learning environment towards learners preferences and actual needs.
Figure 4: Annotation sidebar and annotation browsers (content in Slovak).
Collaborative Adaptive Content Creators
ALEF implements the annotation functionality within collaborative adaptive content creator components. We de-
signed and implemented several such components (further referred to as annotation widgets). Each annotation widget
introduces different goals for collaboration and may use any of aforementioned interface elements.
Tagger is simple annotation widget that enables to assign user dened tags to the content. Motivation behind the
tagging may differentiate among users. Some may use tags to categorize the content to their own categories, others to an-
notate content that they should pay more attention to in the future. Some may tag difcult learning objects, others learn-
ing objects that are important in order to achieve desired learning outcome. Tag annotations are realized as per-content
annotations, i.e. a user can tag the whole learning object, not its particular parts. Users can assign private tags as well as
public anonymous tags.
Besides providing additional style of navigation within a course, tags can be utilized for maintaining course meta-
data, as they represent a form of collaborative semantic descriptions. Moreover, user dened tags are very important for
managing course quality. Tag analysis can reveal implicit user ratings of learning materials, thus allowing (later) ltering
of popular, useful or well written learning objects.
Commentator serves as a general purpose annotation widget for creation of per-text annotations. Students can add
private, public or public but anonymous comments to any part of any learning object. Besides commenting the textual
content, students can also comment (reply to) other existing public or public anonymous comments, resulting into dis-
cussion threads on arbitrary topics, typically related to misconceptions or learning problems.
Comments as well as all of the following annotation types can be rated by other students. It helps us to distinguish
more and less relevant contributions.
Error reporter
Error reporter constitutes generalization of commentator widget. It is specialized for inserting error and bug reports
related to content, which are found by the students. Reported errors are evaluated by a teacher resulting into improved
content serving for a better learning. This process supports collaboration between student and a teacher.
External resources inserter
External source inserter provides functionality for inserting links to external sources into learning content in order to
enrich it with a quality source of knowledge. There are two ways to insert a link: either as a per-text-annotation, using in-
text menu, or through external source widget. Context of an external source inserted through the widget will be assigned
to whole learning text. After inserting external source, the widget will list external sources inserted into current learning
object. Displayed sources are sorted by their quality, high quality sources are on top of the widget. Because of possibly
large/considerable amount of inserted sources, we limited the number of visible sources to a default value. Student can
eventually expand the widget to see all sources.
Questions creator
Question creator widget provides students with an interface for adding questions and for answering the questions
added by their peers. This implicates that the system supports such types of question, which can be evaluated automati-
cally. Currently, students may add ve types of questions: (1) single choice question, (2) multiple choice question, (3)
simple free text answer question, (4) sorting question (the task is to re-order the lines into correct order) and (5) text
complement question (the task is to ll missing words into dedicated elds within the text, e.g. completing missing com-
mands in a programming code). As a user-generated question is a special kind of an annotation, the procedure of add-
ing a question is based on the principles of adding an annotation. The only difference is that after selecting the text and
choosing a question as a type of annotation, a student is presented with a form for specifying question type, title, descrip-
tion and options along with an indication of correct answer(s).
The part dedicated for answering question created by students has similar interface as ordinary questions present
within learning materials except that it is displayed within a widget instead of in a main content window when a student
chooses a question. After lling an answer, it is automatically evaluated by the system and the student receives an instant
feedback about her answer. Afterwards, she can rate the question in order to determine questions perceived quality. At
the time, rating can be also viewed as a substitution for expressing other issues related to a question, e.g. addressing mis-
conceptions.
Annotating, i.e. adding questions, is a complex process. It is necessary for given questions to exceed the minimal
level of quality. We designed a question lifecycle to allow teacher to select it among the other most useful student-created
questions and transform it into an expert (a teacher) questions to become a part of a curricula.
RATING MODELS FOR MOTIVATION AND CONTENT IMPROVEMENT
Motivation is an important factor affecting students learning performance. In our educational system we motivate
students by facing them with human basic instinct competitiveness. We allow students to engage in a simple game of
collecting activity points determined by type and intensity of actions that students perform within the system and to ob-
tain highest rating among all peers. The hidden purpose of the game is to lead students to fully utilize collaboration and
annotation options.
While students learn, they see their actual points and absolute position among other competing students. This is
a sufcient information for a student to determine his chances to win or at least - to keep his current ranking.
Besides motivation, we introduce resource rating model. It partially inuences students rating in the system, but its
major aim is to collect information about resource quality and usefulness that can be further utilized to improve the qual-
ity of learning content.
Student rating model
To increase motivation of students we designed a model of student rating. We introduce a score as an indicator of
students overall activity within the learning course. Our intent is to motivate students not to focus on single type of
learning task, but to perform wide variety of tasks allowed by annotation widgets. To address this requirement, we sepa-
rately evaluate score for each assessed learning activity. Score of an activity is computed as follows:
( , )
( , ) ln 1
N u a
u a k
S a
where S(u,a) is score for activity type a performed by a user u, S
(a) is a basic score for an activity type a, N(u,a) is num-
ber of realizations of activity type a by user u, and k is a factor of distortion. We assess the following learning activities,
categorized into two main ows (they may have different S
(a)):
Learning ow Collaboration ow
reading (visiting) explanations
answering questions
solving exercises
inserting annotations (comments, error
reports, questions, external sources etc.)
rating annotations
After evaluating scores for individual types of activities, we calculate a cumulative score reecting overall activity of
student. Besides particular learning and collaboration activities, student overall rating also depends on the quality of an-
notations he created:
( ) ( , ) ( )
a A r R
S u w S u a P r
where S(u) is overall score rating of a student u, w
denotes actual weight of an activity type a, S(u,a) is score for activity
type a performed by a student u, P
(r) is explicit rating of a resource r and R
is a set of all resources created by user u. By
using weights we set up different signicance/usefulness of activity types for students.
Resource rating model
In order to determine a quality of a resource within a system (either content or annotation), we propose a resource
quality rating model. Resource quality derives from the explicit resource rating given by students and implicit resource
ranking derived from actions of students associated with a given resource. It is computed as weighted sum of both rat-
ings:
( ) ( ) ( )
e e i i
P r w w P r P r = +
where P(r) is overall rating of a resource r, P
(r) is explicit rating and P
(r) is implicit rating of a resource r, and w
and w
are weights for determining the importance of an explicit and implicit rating, respectively. w
+ w
= 1. Explicit rating is
common for all resource types and is computed as weighted mean of all users ratings:
( ) ( ) ( , )
P r K u P u r
where K(i) is overall knowledge level of a student u, P
(u,r) is explicit rating of resource r by a student u. By considering
overall knowledge level of students, we prefer ratings given by more advanced students.
Computation of implicit rating depends on a type of resource. For each type of resource, different ratings can be de-
ned according to actions related to that type of resource. The following table introduces main factors that inuence the
computation of implicit rating P
(r) for selected types of resources (Tab. 1). Note that implicit ranking has typically lower
weight than explicit. Detailed description of each computation is out of the scope of this paper.
Table 1: Example factors inuencing the computation of implicit rating P
(r).
Resorce
(Content)
Example factors Resource
(Annotation)
Example factors
Explanation Number of repeated visits Tag Popularity of a tag
Question Ratio of correct to incorrect answers
Number of I dont understand
User-created
Ratio of correct to incorrect answers
Number of I dont understand answers
Number of errors (reported by error
reporter)
Exercise Number of hint requests Comment Number of replies
Ext. resource Number of accesses to external resource
Error report The relevance of error
We evaluated the proposed approach in a domain of programming learning. Due to the extensivity and complexity of
the framework, we have not evaluated all aspects related to collaboration and learning content quality improvement based
on annotations. However, we conducted three experiments in order to evaluate the following hypotheses:
1. Question creation supports/stimulates collaboration and course content enrichment.
2. Error reporter improves the quality of the learning content by revealing relevant errors in the content.
3. Motivating students based on a game increases learners activity in the system.
Question creation evaluation
We evaluated the rst hypotheses in real educational settings during Functional and logic programming course lec-
tured at Slovak University of Technology in Bratislava. Students had access to educational materials for learning pro-
gramming language Prolog and were instructed to contribute questions to this learning content, and/or to answer and rate
existing questions. We analyzed collected data (questions, answers, ratings) after seven days since the start of the experi-
ment and found out that 30 students were involved in this activity (nearly 60% of all students that could participate). Stu-
dents created together 88 questions and provided 660 answers.
We evaluated the hypothesis by assessing questions quality. We manually evaluated all obtained questions and divid-
ed them into three levels according to their quality (Tab. 2): First level contained high quality questions, which are about
to be used as new educational materials. The second level contained correct questions but their difculty wa rather low or
they were not spelled properly. The third level contained questions useless for educational purposes, defective questions
or ambiguous questions. The same sample of questions was evaluated by deriving ratings using question quality rating
model based on learning object quality rating model. The method marked 24 questions of 88 as quality questions (Tab.
3). We compared these questions with the questions evaluated manually. According to the table the rating model did not
estimate for quality questions high quality questions only, but also some questions with the second level of quality. Im-
portant result is that none of the faulty questions was marked as quality one.
Table 2: Manually determined quality level (88 questions).
Quality level No. of questions
1. High quality questions 33
2. Medium quality questions 48
3. Defective questions 7
Table 3: Manually determined quality level*.
2. Medium quality questions 7
* of 24 questons automatcally marked as quality questons
The results show that even in a small period of a time (a week) student can leverage annotation-based tool to contrib-
ute new quality content. This way an overall learning content quality increases. More details on this experiment can be
found in (Unk & Bielikov 2010).
Error reporting evaluation
We evaluated error reporting feature during Procedural programming course lectured at Slovak University of Tech-
nology in Bratislava. For six weeks, students had possibility to create annotations for the learning content of the course
(consisting of 213 explanations, 162 exercises and 420 questions). They created comments, bug reports and during the
last two weeks of the semester they also got the possibility of inserting links to external resources. Out of 272 students,
129 created at least one annotation (47.42% of all students) .They added 949 annotations in total (Tab. 4), from which
782 were relevant (those that did not contain error or were semantically correct). Interesting nding is that the most com-
mon type of annotation was an error report. Students mostly revealed grammatical errors and misspellings that occurred
in the learning content. Often duplication appeared among error reports, which increased the credibility of provided error
An important aspect of error reporting is the level of knowledge of students who participate in annotating. We identi-
ed that 8% of students with the best overall assessment created 50% of all error reports and 20% of the most advanced
students provided 82% error reports (similar statistics apply for all annotations). Such annotations also have higher qual-
ity and thereby help the weaker students. We also found that 5% of the most weak performing students did not create any
annotation. Nevertheless, the number of provided relevant errors (either more or less severe) and average occurrence of
errors (1 error per 1.46 learning objects) prove that error reporting feature reveals relevant errors and helps improving the
quality of the learning content.
Table 4. Annotation statistics over six weeks of the term.
Annotation type All Relevant Ratio
Comments 65 54 .83
Errors 697 546 .78
External sources* 187 182 .97
In total: 949 782 .82
* possible to add only during last two weeks of the term.
Motivation evaluation
In the last experiment we were observing how motivation inuences students behavior within a learning system. We
used the same setup as in the previous case during six weeks (from 8
till 13
week of semester) in the course of learn-
ing programming. In the beginning of the course we encouraged students to use annotations in order to help themselves
as well as others by improving the content. We also told the students that their activity and initiative will be in reward
changed for bonus points to their overall assessment in the course. In the period half-time, at the beginning of eleventh
week, we deployed the student rating model. Student activities are depicted in the following gure (Fig. 5).
Figure 5: Student activity counts per day (y-axis) from 8
week of a term (x-axis). Activities related to
annotations (left), activities related to content (right).
We observed increase in the count of specic learning activities after the deployment of student rating model. While
there was no signicant change in activities related to content (visiting an explanation, answering a question, etc.), the
number of activities related to annotations (adding an annotation, rating an annotation) substantially increased.
We noticed decreased degree of activity in the 13
week, in both content and annotations related activities. This
probably relates with the fact that it was the last week of the term and students were nishing their nal projects, having
not much time left for learning (in fact, at that time they were supposed to master majority of the topics). Further analysis
showed that students interest in more difcult and time-consuming content-related tasks (solving an exercise) increased
more signicantly and interest in simple and not demanding tasks was affected only slightly.
Considering the increase of activity, we believe that proposed score model increases motivation of students to solve
variety of tasks, resulting into a potential increase of learning performance and outcomes.
User-generated annotations of the Web content is an active area of research, which already proved their positive im-
pact in case of web-based learning. For instance, in (Farzan & Brusilovsky 2006), authors proposed a service for adding
annotations to any set of linked web-documents, including learning materials and built an efcient social navigation sup-
port on top of it. However, similarly to other known works, they do not consider the annotation itself as a part of the con-
tent, meaning that learners cannot collaborate on annotations themselves, only on content initially generated by a teacher.
In this paper, we described our approach to integration of content annotations into adaptive web-based learning,
where they play a crucial role of supporting the collaboration between students, leading to improvements of learning out-
comes. Even more, collaboratively created annotations increase quality of available content either by enriching it by links
to high quality external resources, pointing out key ideas in comments or just reporting a bug in the content.
Collaboration through annotations also positively inuences students motivation to actually learn through the sys-
tem, as they feel to be involved in the process, playing an active role. We can additionally setup a gaming and reward
mechanisms, which ensure continuous motivation to contribute quality annotations to the content.
We performed several long-term experiments in real courses of learning programming at bachelor degree of study.
The results show that annotations can be used as a tool for supporting collaboration and learning content quality im-
provement. In order to evaluate the whole proposed concept of annotations and its binding to all layers of educational
process, we need to (as a future work) conduct comprehensive multi-layer evaluation.
Alavi, Maryam. (1994). Computer-Mediate Collaborative Learning: An Empirical Evaluation. MIS Quarterly, 18(2), 159174.
Beaumont, I. & Brusilovsky, P (1995). Adaptive educational hypermedia: From ideas to real systems. In H. Maurer (Ed.), Proc.
of ED-MEDIA 95 World conf. on educational multimedia and hypermedia. Graz, Austria, 9398.
Brusilovsky, P. (1996) Methods and techniques of adaptive hypermedia. User modeling and user-adapted interaction. 6(2-3),
Dillenbourg, P. (1999). Collaborative Learning: Cognitive and Computational Approaches. Advances in Learning and Instruc-
tion Series. New York, NY: Elsevier Science, Inc.
Downes, S. (2005). E-learning 2.0. eLearn magazine. Issue 10. ACM, p. 1.
Farzan R., & Brusilovsky P. (2006). AnnotatEd: A Social Navigation and Annotation Service for Web-based Educational Re-
sources. Proc. of E-Learn06, 27942802
Ip, A., & Canale, R. (2003). Supporting Collaborative Learning Activities with SCORM. Educause in Australasia 03, 669678.
Meccawy, M., Blancheld, P., Ashman, H., Brailsford, T., & Moore, A. (2008). Whurle 2.0: Adaptive Learning Meets Web 2.0.
Times of Convergence. Technologies Across Learning Contexts, LNCS 5192, Springer. 274279.
imko, M. Bielikov, M. (2009). Automated Educational Course Metadata Generation Based on Semantics Discovery. LNCS
5794, Proc. of European Conf. on Technology Enhanced Learning, EC TEL 2009, Nice, France. Springer, 99105.
imko, M., Barla, M., & Bielikov, M. (2010). ALEF: A Framework for Adaptive Web-based Learning 2.0. N. Reynolds and M.
Turcsnyi-Szab (Eds.): KCKS 2010, IFIP Advances in Information and Communication Technology, Vol. 324. Springer,
Tvaroek, J. (2011). Bootstrapping a Socially Intelligent Tutoring. Information Sciences and Technologies Bulletin of the ACM
Slovakia. ISSN 1338-1237. 3(1), 3341.
Unk, M. & Bielikov, M. (2010). Annotating Educational Content by Questions Created by Learners. Proc. of 5th Int. Work-
shop on Semantic Media Adaptation and Personalization, SMAP 2010, IEEE, 1318.
Weibelzahl, S., & Weber, G. (2002). Adapting to prior knowledge of learners. P. de Bra, P. Brusilovsky, R. Con-ejo (Eds.). Proc.
of the second int. conf. on Adaptive Hypermedia and Adaptive Web Based Systems, AH2002, Springer, 448451.
This work was partially supported by the grants VG1/0675/11, KEGA 028-025STU-4/2010 and it is the partial result of the Re-
search & Development Operational Programme for the project Research of methods for acquisition, analysis and personalized convey-
ing of information and knowledge, ITMS 26240220039, co-funded by the ERDF.d
Lecture Capture: Good Student Learning or Good Bedtime Story? 43
Lecture Capture: Good Student Learning or Good Bedtime Story?
An Interdisciplinary Assessment of the Use of Podcasts in Higher Education
LENA PAULO KUSHNIR, KENNETH BERRY, JESSICA WYMAN, FLORIN SALAJAN
The use of podcasts to deliver undergraduate lectures is increasingly popular and often viewed as an effective way
to supplement and enhance undergraduate education (Aguiar et al., 2009; Brotherton & Abowd, 2004; Heilesen, 2010;
McKinney et al., 2009; Traphagan et al., 2010; Yunus et al., 2006). Since about 2005, there has been a tremendous in-
crease in the use of podcasts, and this is likely correlated to an increase in Internet accessibility and the use of mobile
devices for accessing portable media (Abt & Barry, 2007; Bongey, et al., 2006; Karppinen, 2005). It was not too long
ago that lecture podcasts were reported to be among the top two technologies that were implemented in higher educa-
tion (Horizon Report, 2006). Generally the literature is positive about the use of podcasts, and current research shows
that students report overwhelmingly positive attitudes towards the use of podcasts in their courses (Aguiar et al., 2009;
Brotherton & Abowd, 2004; Evans, 2008; Goldberg et al., 2006; McKinney et al., 2009; Nicholson & Nicholson, 2010;
Parson et al., 2009; Traphagan et al., 2010; Yudko et al., 2008). Notwithstanding these ndings, a review of the empiri-
cal literature investigating the effectiveness of podcasts in undergraduate education reveals that there are some conicting
and inconsistent ndings, particularly with respect to the impact podcasts have on student grades and class attendance.
Instructors who favor the use of podcasts argue that it does support student learning (Abt & Barry, 2007; Aguiar et al.,
2009; Brotherton & Abowd, 2004; Yunus et al., 2006), while putting no restrictions on the time and place of lecture de-
livery that a face-to-face class imposes (Aguiar et al., 2009; McKinney et al., 2009; Sutton-Brady, 2009; Taylor, 2009;
Walls et al., 2010; Yunus et al., 2006). Instructors who tend to reject the use of podcasts generally argue that the use of
this technology leads to a decrease in class attendance (Holbrook & Dupont, 2009; Lonn & Teasley, 2009; Walls, et al.,
2010; Wang, et al., 2010), while others are concerned with the implications of having their course materials online and
available to anyone (Bongey et al., 2006; Taylor, 2009).
Still today, the available research does not present much empirical evidence for learning being enhanced with the use
of podcasts; more research needs to be conducted to investigate the effectiveness of this technology (Abt & Barry, 2007;
Aguiar et al., 2009; McKinney et al., 2009; Scutter et al., 2010; Shantikumar, 2009; Traphagan et al. 2010; Yunus et al.,
2006). We surveyed students in various undergraduate courses that used podcasts to supplement face-to-face lectures
in Art, Dentistry, Design and Psychology. We did this to better understand student opinions and perceptions about pod-
casts, and to determine how they used the podcasts in these particular courses. This multi-course and interdisciplinary
approach allowed us to explore many aspects regarding podcast use in undergraduate education that could contribute to
our knowledge of principles that inform teaching and best practices.
1.1 What are podcasts?
The noun podcast is one of the top 5 new words added to the Oxford Advanced Learners Dictionary (Oxford Uni-
versity Press, 2010). It has many meanings and synonyms associated with it. Normally, it refers to audio les that are
available to be downloaded and saved to a local computer or other device, but we often consider podcasts to include
video, pictures, and in some cases, synchronized images (e.g., PowerPoint slides or whiteboard notes with synchronized
audio). The term podcast is often used synonymously with many other terms, for example, lecture capture, screencast,
web-lecture, vodcast, etc., (Aguiar et al., 2009; Brotherton & Abowd, 2004; Evans, 2008; McGar, 2009; Taylor, 2009;
Traphagan et al., 2010; Rosell-Aguilar, 2007). Although podcasts are often used for reasons other than just presenting
lecture material (e.g., tutorial material, pre-class material that is related to lectures but frees up class time for additional
material to be presented in class, review of difcult concepts, etc.). Despite all of the different denitions, synonyms and
uses of podcasts, for the purposes of this paper, the instructional contexts in which these terms are used are similar, that
44 Kushnir, Berry, Wyman, and Salajan
is, facilitating students access to lecture content using online technologies (i.e., for the purpose of this paper, podcasts
refer to the presentation of online lectures to augment face-to-face lectures).
1.2 Impact of Podcasts on Students Learning
Some authors argue for the use of podcasts as an effective and constructive tool for learning in online environments
as there are measurable increases in learning (Abt & Barry, 2007; Aguiar et al., 2009; Brotherton & Abowd, 2004; Yu-
nus et al., 2006), and there is evidence that podcasts support students with different learning styles and specic needs
(Barsky, 2008; Karppinen, 2005; Larkin, 2010; Lonn, & Teasley, 2009; Piecka et al., 2008; Taylor, 2009). Some authors
have reported increases in student motivation to learn and complete assigned work (Aguiar et al., 2009; Oliver, 2005),
while others suggest that podcasts allow students to concentrate on course material better than they normally would in
face-to-face lectures (Yunus et al., 2006). Young (2008) suggests that podcasts facilitate retention of course material, and
that courses that use podcasts have fewer student drop-outs compared to courses that do not use podcasts. Parson et al.,
(2009) found that students who used podcasts reported that they found them helpful for course review and exam prepa-
ration. Brotherton and Abowd (2004) suggested that podcast use helps students study more efciently (i.e., with less
work, p. 147), and Nicholson & Nicholson (2010) had a similar nding where access to recorded lectures decreased stu-
dents efforts required to complete assignments. A few authors have reported increases in student grades both in courses
that used podcasts to supplement face-to-face lectures, and in courses that only used podcasts (McKinney et al., 2009;
Young, 2008; Yunus et al., 2006). While many authors have found no signicant increases in student grades, the general
consensus is that the use of podcasts does not hinder student performance and that if anything, podcasts help with other
learning related variables discussed here.
Theoretically, the use of podcasts in higher education should help students to learn better than compared to face-to-
face situations. Cognitive psychology and more specically, Mayers (1997) cognitive theory of multimedia learning
(which combines knowledge from various cognitive processing theories and draws on research-based principles of mul-
timedia learning) helps us to understand why this should be so. Our cognitive processing systems are made up of multi-
modality channels that represent each of our sensory systems, and that feed into corresponding perceptual and memory
systems (e.g., visual sensory input leads to our iconic memory system, auditory sensory input leads to our echoic mem-
ory system, haptic sensory input leads to our tactile memory system, etc.). According to Mayer, students learn best by
using multimodality processing systems simultaneously, specically, the visual information processing system and the
auditory information processing system. In doing so, students have opportunities to process course material at a deep
level by making meaningful connections and associations between material presented to each of these systems. (For a
more complete description of this theory, see Mayer, 1997.)
That the current literature (for the most part) is not consistent with Mayers theory might be a signal to instructors,
instructional designers and others in this eld of study, that in order for us to understand the educational benets of new
technologies, our methodological approaches must aim to link and apply existing theoretical frameworks to data that
adds to our understanding of the design and use of instructional technologies. This type of research linking theories to
online learning is too often absent in the education literature, and Chalmers (2000) and Kraus et al (2001), Mayer (1997)
suggest that more of this type of linking is necessary so that educators can make informed, data-based decisions about
the most effective implementation of educational technologies.
1.3 Impact of Podcasts on Class Attendance
The most common concern about using podcasts to supplement face-to-face lectures that is reported in the literature,
and the authors of this paper have each discussed anecdotally with colleagues, is the concern that the availability of lec-
ture podcasts will contribute to a decrease in class attendance. While there are not many published examples evidencing
such decreases (e.g., Holbrook & Dupont, 2009 and Wang et al., 2010;), the bulk of empirical evidence suggests that
there is no negative impact of lecture podcasts on class attendance, and students report that they prefer to participate in
face-to-face classes even when podcasts are available (Brotherton & Abowd, 2004; Copley, 2007; Parson et al., 2009;
Traphagan et al., 2010; Young, 2008; Yudko et al., 2006). In fact, Brotherton & Abowd (2004), Traphagan et al., (2010)
and Yudko et al., (2006) found that, although students believed that the availability of podcasts would tempt students to
skip class, the participants themselves did not report missing classes just because they could access the lecture material
online. As Brotherton and Abowd (2004) and others have suggested, there is no proof that lecture podcasts have a nega-
tive impact on class attendance.
The fact that both instructors and students hold the misconception that podcasts negatively impact class attendance
is interesting. It shows a consistency in reporting causal attributions of student behaviours (i.e., that one possible cause
of students skipping class is students knowledge that lecture podcasts are available and can be used instead of attending
face-to-face lectures). Social-cognitive explanations of behaviour help us better understand this consistent misconcep-
tion. Attribution theorists explain how we attribute causes of our own behaviours, the behaviours of others, or events
around us, by considering the sorts of biases and attribution errors we commonly make when trying to explain behaviour
(for our discussions here, student behaviour). Briey, when we explain the behaviours of others, we have a tendency to
use personality-based explanations (or dispositional variables) and motivation-based explanations as the causes of the be-
haviours, even though we are not personally familiar with the individuals whose behaviour we try to explain. Attribution
theorists call this the fundamental attribution error. When explaining our own behavior, attribution theorists suggest that
we often fall prey to a self-serving bias. In explaining positive aspects of our own behaviours (or successes), we tend to
use dispositional (or personality-based) explanations as the causes of behaviours (i.e., taking personal credit for our suc-
cess). In contrast, when explaining negative aspects of our own behaviours (or failures), we tend to use situational or ex-
ternal, event related explanations as the causes of behaviours (i.e., not taking personal responsibility for negative aspects
of our behaviours). (For a more complete description of this theory, see Heider, 1958.)
The data regarding class attendance (e.g., as reported in Brotherton & Abowd, 2004, Traphagan et al., 2010, and
Yudko et al., 2006) are consistent with the predictions of attribution research. When predicting the behaviour of oth-
ers, both instructors and students predict that, by and large, students will be tempted to skip class, likely for various
personal reasons, when podcasts are available (i.e., a fundamental attribution error and surely a negative view of student
behaviour). When students report on their own (class attendance) behaviour, they report that they do not skip classes just
because podcasts are available, rather they prefer attending the face-to-face classes and use the podcasts as a supplement
because they enjoy the interactivity of face-to-face classes, they need a structured learning environment, they experience
better student engagement and they are more focused on class discussions (Brotherton & Abowd, 2004; Copley, 2007).
This sort of explanation can represent a self-serving bias and certainly a positive representation of successful student be-
haviour.
2. RESEARCH QUESTIONS AND RATIONALE
This study investigated the following three questions: (1) Does the use of podcasts to supplement face-to-face lec-
tures in undergraduate courses increase student learning and class absenteeism?; (2) How do students use podcasts to
supplement their learning?; (3) What are students experiences with podcasts?
While the second and third research questions were exploratory, we wondered if there were any discipline specic
variables that might have an impact on students experiences with podcasts. As outlined in the literature review, there are
mixed ndings regarding student learning outcomes and class absenteeism. In exploring students experiences and how
students used podcasts, we hoped to better understand what we describe as misconceptions about the impact of podcasts
on student grades and class attendance. This study is part of a larger project and more data is being collected. The re-
sults of this introductory study will guide us through subsequent research.
3. DESIGN OF STUDY
3.1 Description of study
We conducted three surveys and surveyed students in three groups: (i) Art & Design, (ii) Dentistry, and (iii) Psychol-
ogy. One of three different podcast surveys (an Art & Design survey, a Dentistry survey or a Psychology survey) was
deployed to students in each of the specic groups. All students were asked if they used the lecture podcasts, if they
thought it helped them learn course material, if they wanted to see more courses in the university use podcasts, if the
knowledge that podcasts were available tempted them to skip class, and what their experiences were with using podcasts.
While the surveys for each of the three groups were not identical, many of the questions were very similar. For example,
to facilitate course specic factors, there were some minor modications in the wording of survey questions adminis-
tered to Dentistry and Psychology students. For Art & Design students, instructors asked extra questions and many more
open-ended questions about students use and experience with using podcasts in those courses. The psychology course
instructor had included an extra measure and correlated students nal course grades with their survey responses. We
developed different surveys for each of the groups in order to accommodate individual instructors research goals and in-
structional needs. Students participation in completing the surveys was voluntary and they were encouraged to complete
the surveys to help with the continuing development of the courses and of the use of podcasts within these disciplines.
Participants were three hundred and eighty-six undergraduate students enrolled in one of two universities. One was
a large urban centre university, the other a small urban center university. Data were collected from six courses using lec-
ture podcasts to supplement face-to-face lectures (i.e., one Art course, two Dentistry courses, one Design course, and two
Psychology courses). Participants were enrolled in one of two second year Introductory Psychology courses (n= 108), a
second year Art History course or third year Design course (n=156), or a third or fourth year Dentistry course (n= 122).
3.3 Procedure
All participants were given several opportunities to complete the survey online using the institutional learning man-
agement systems (LMS), with the exception of participants enrolled in the Psychology courses. Those students complet-
ed a paper-pencil version of the podcast survey in class, at the end of the term in which the online data from Dentistry,
Art and Design were collected. Use of the lecture podcasts was voluntary and each of the instructors (of the courses in-
cluded in this study) used the podcasts as a supplement to the face-to-face lectures. Podcasts were made available shortly
after the face-to-face classes met, and were accessible through the LMS and integrated media storage facilities associated
with each institution. Students accessed their course podcasts through the separate course shells in the course manage-
ment tool of the LMS.
3.4 Analyses
Analyses of the data included response frequencies of the quantitative survey questions across the three groups, and
independent samples t-Tests to measure any differences between nal course grades of students in the Psychology cours-
es who used the lecture podcasts compared to those who did not use the podcasts. Qualitative analyses of the open-end-
ed survey questions included weighted word lists that were calculated and puzzled out into word clouds that were gen-
erated from the students text answers. The word clouds represented a summary of the text that students wrote in their
open-ended answers. A user generated word cloud visualizes information that is related to a specic survey question
and, in essence, it depicts visually, the frequency of specic topics that students write about in their survey answers. The
importance (or frequency) of specic words is often displayed through font size (as in our examples below), font colour,
or some other attribute (see Bateman et al., 2008 for an overview of word/tag clouds).
In this study, we investigated whether podcasts had any impact on student learning and class attendance; we also at-
tempted to get a better understanding of how students used podcasts and what their experiences were when they relied on
podcasts to supplement their learning. Response frequencies for students in Dentistry and Psychology are presented in
Table 1; response frequencies for students in Art & Design are presented in Figures 1 through 10.
While explicit measures of learning where computed only for students enrolled in the psychology courses, all stu-
dents perceptions of whether they believe podcasts help them learn were measured across the three groups (i.e., Art &
Design, Dentistry, and Psychology). Our measures of the impact of podcasts on student learning yielded no differences
between groups. We found no statistically signicant differences between the scores of students in Psychology who
used the lecture podcasts and those who did not, F (38, 12) = 1.03, p = 0.3730 for one course, and F (28, 24) = 1.13, p
= .7440 for another. Interestingly, across the disciplines, students responded in similar ways when asked if they thought
the podcasts helped them learn. For example, as depicted in Figure 1, 80% of Art & Design students reported that they
strongly agreed or somewhat agreed that the podcasts increased their understanding of the material covered in
class, and as shown in Table 1, question 2, 81% of Dentistry and 87% of Psychology students reported that yes they
believed that the use of podcasts in this lecture course helped their learning. These data support what others have re-
ported in the literature, that is, that there is not much empirical evidence showing that podcasts have a positive impact on
learning outcomes, but that students believe that podcasts help them earn higher grades.
Table 1. Response frequencies (%) for students in Dentistry
and Psychology
Dentistry Psychology
Yes No Yes No
Did you use/listen to/review the podcasts made available on our course Web
Site?
95% 5% 65% 35%
Do you believe that the use of podcasts in this lecture course helped your
learning?
81% 18% 87% 5%
Would you like to see more courses in this Faculty making use of podcasts? 74% 26% 83% 6%
If you used the lecture podcasts, did you nd that having the podcasts
available tempted you to miss classes if you knew ahead of time they would
be available for you to review?
If you DID NOT use the Podcasts, do you think youd be tempted to miss
class if knew ahead of time that podcasts were available?
Figure 1: Art & Design students perception of the impact of podcasts on learning
Across all groups, instructors reported that there was not a detectable decrease in class attendance, and with the ex-
ception of Dentistry, students reports of whether they were tempted to miss class (because of the availability of the pod-
casts) were consistent with Instructors reports of no decline. As shown in Table 1, questions 4 and 5, 42% and 32% of
students in Dentistry reported that they were tempted to skip class, whereas only 22% and 23% of students in Psychol-
ogy. As depicted in Figure 2, only 4% of students in Art & Design reported this. Interestingly, four of the six courses
included in this study (i.e., the Psychology, Art and Design courses) were taught at one particular institution that has a
student-class-attendance-policy, plus, the two Psychology courses had surprise quizzes on 7 out of 8 possible quiz-fair
days (some classes were deemed unfair quiz days for various reasons, e.g., rst day of class in which none of the students
would be prepared for a quiz, classes in which a term test was administered, and the last class when students generally
had a large assignment due). These institution and course specic factors introduce confounds that made it difcult for
us to decipher whether podcasts had no impact on attendance, or if institutional and course factors made it undesirable to
skip class. None of these factors where present for the Dentistry courses and yet almost 40% of students predicted that
they would be tempted to skip class. So while we are tempted to suggest that our data support the majority of the lit-
erature which suggests that podcasts have no negative impact on attendance (Brotherton & Abowd, 2004; Copley, 2007;
Parson et al., 2009; Traphagan et al., 2010; Young, 2008; Yudko et al., 2006) , that would be premature given our data.
We do not know how much of the no impact on attendance is related to podcasts (if at all), or if it is more related to
institutional and course specic factors.
Figure 2: Art & Design students report of class attendance
A large proportion of students reported that they used the podcasts made available to them (i.e., as shown in Table 1,
question 1, 97% of Dentistry students and 65% of Psychology, and as depicted in Figure 3, below, 97% of Art and De-
sign students). Figure 4 shows how Art & Design students used podcasts in their learning. It makes sense that students
believed that podcasts helped them learn, Figure 4 indicates that they used them for the right reasons, or at least for rea-
sons that should have affected behaviours that support learning (i.e., specically reviewing and studying).
Figure 3: Art & Design students report of podcast use
Figure 4: Art & Design students report of how they use podcasts to learn
Figures 5 to 8 show how podcasts affected students experiences. Again, data in Figure 5 supports what one would
think helps students to learn (e.g., supplement note-taking) , and the word clouds depicted in Figures 7 and 8 corrobo-
rate the ndings displayed in Figure 5 (e.g., that podcasts helped with missed classes, notes, points, etc., and
that podcasts changed students class experience with note-taking, review, missed lectures, etc.,). The larger the
words in the word clouds, the more often students used the terms in their responses about how podcasts affected them.
Interestingly, data in Figure 6 is consistent with the literature that suggests that podcasts support students with different
learning styles and specic needs (Barsky, 2008; Karppinen, 2005; Larkin, 2010; Lonn, & Teasley, 2009; Piecka et al.,
2008; Taylor, 2009). Our data here show that students reported that podcasts helped them with comprehension of course
material, helped students for whom English is a second language, and helped students with learning disabilities.
Figure 5: Art & Design students report of how podcasts affect note taking
Figure 6: Art & Design students report of how podcasts help them
Figure 7: Word cloud of Art & Design students report of how podcasts help them
Figure 8: Word cloud of Art & Design students report of how podcasts affect classroom experience
Table 1 (question 3) and Figures 9 and 10, below, show (overwhelmingly) other ways that podcasts impact students
and ways that students would like to see podcasts used in the future. For example, students reported that they want more
of their courses to use podcasts (i.e., Table 1, question 3: 74% of Dentistry and 83% of Psychology students), for reasons
other than just supplementing lectures (i.e., Figure 9: e.g., for guest speakers, interviews, other faculty and other types of
lectures), and, as discussed earlier (Aguiar et al., 2009; Brotherton & Abowd, 2004; Evans, 2008; Goldberg et al., 2006;
McKinney et al., 2009; Nicholson & Nicholson, 2010; Parson et al., 2009; Traphagan et al., 2010; Yudko et al., 2008),
students reported very positive attitudes toward using podcasts (i.e., Figure 10).
Figure 9: Word cloud of Art & Design students report of what other types of podcasts they would like to have
Figure 10: Word cloud of Art & Design students comments about podcasts
5. CONCLUSIONS AND RECOMMENDATIONS
The ndings in this study suggest that although students believed that the podcasts helped them to learn, that stu-
dents seemed to use them in pedagogically sound ways, that they liked having them as part of their learning experience
and that they wanted more of them, we did not nd any signicant impact on learning. When Mayers (1997) cognitive
theory of multimedia learning is considered, these ndings do not make much sense; the technology should help student
learning. Given this, it might be that there is something in the way instructors implement this technology. Future re-
search will need to drill down to the component parts of podcast use and investigate how closely instructors follow best
practices and research-based principles of multimedia learning, and if instructional materials are grounded in theories of
meaningful learning. It seems fair to say that given the inconsistencies found in the literature that merely making pod-
casts available is not sufcient to impact student learning.
Also, future research should systematically measure what students are doing with these podcasts. Perhaps they sim-
ply apply the same good or poor study habits when using the technology that they normally apply in face-to-face con-
texts. It might be that high performing students use the same effective strategies when using podcasts that they use when
studying, attending lectures, etc., and that low performing students might use the same ineffective strategies when using
podcasts. If poor performing students are not good listeners in class, are not skilled at synthesizing information and
processing it deeply, then they might not be good listeners in podcasts, and might be ineffective at synthesizing and
processing information deeply from podcasts. This type of transferring of current skill sets and capacities might result in
a null effect of new technologies that are intended to facilitate learning.
Abt, G., & Barry, T. (2007). The quantitative effect of students using podcasts in a rst year undergraduate exercise physiology
module. Bioscience Education e-Journal, 10, April 14th 2010. Retrieved from http://dx.doi.org/10.3108/beej.10.8
Aguiar, C., Carvalho, A. A., & Carvalho, C. J. (2009). Use of short podcasts to reinforce learning outcomes in biology. Bio-
chemistry and Molecular Biology Education, 37(5), 287-289.
Barsky, E., & Lindstrom, K. (2008). Podcasting the sciences: A practical overview. Issues in Science and Technology Librarian-
ship, (55)
Bateman, S., Gutwin, C., Nacenta, M. (2008). Seeing Things in the Clouds: The Effect of Visual Features on Tag Cloud Selec-
tions. In: Proc. of the 19th ACM conference on Hypertext and Hypermedia, pp. 193202. ACM Press, New York.
Bongey, S. B., Cizadlo, G., & Kalnbach, L. (2006). Explorations in course-casting: Podcasts in higher education. Campus-Wide
Information Systems, 23(5), 350-367.
Brotherton, J. A., & Abowd, G. D. (2004). Lessons learned from eClass: Assessing automated capture and access in the class-
room. ACM Trans.Comput.-Hum.Interact., 11(2), 121-155.
Chalmers, P.A. (2000). User interface improvements in computer-assisted instruction, the challenge. Computers in Human
Behavior, 16, 507-517.
Copley, J. (2007). Audio and video podcasts of lectures for campus-based students: Production and evaluation of student use. In-
novations in Education and Teaching International, 44(4), 387-399.
Evans, C. (2008). The effectiveness of M-learning in the form of podcast revision lectures in higher education. Computers &
Education, 50(2), 491-498.
Goldberg, H. R., Haase, E., Shoukas, A., & Schramm, L. (2006). Redening classroom instruction. Advances in Physiology
Heilesen, S. B. (2010). What is the academic efcacy of podcasting? Computers & Education, 55 (3) , 1063-1068. (The)
Horizon Report (2006). The NEW MEDIA CONSORTIUM and EDUCAUSE Learning Inititative, The Horizon Report ---
2006, available at: www.nmc.org/horizon/ (accessed April 14, 2010).
Heider, F. (1958). The Psychology of Interpersonal Relations. New York: John Wiley & Sons.
Holbrook, J., & Dupont, C. (2009). Profcasts and class attendance--does year in program matter? Bioscience Education, 13
Karppinen, P. (2005). Meaningful learning with digital and online videos: Theoretical perspectives. AACE Journal, 13(3), 233-
Kraus, L.A., Reed, W.M., & Fitzgerald, G.E. (2001). The effects of learning style and hypermedia prior experience on behav-
ioral disorders knowledge and time on task: a case-based hypermedia environment. Computers in Human Behavior, 17,
Larkin, H. E. (2010). But they wont come to lectures... the impact of audio recorded lectures on student experience and atten-
dance. Australasian Journal of Educational Technology, 26(2), 238-249.
Lonn, S., & Teasley, S. D. (2009). Podcasting in higher education: What are the implications for teaching and learning? The In-
ternet and Higher Education, 12(2), 88-92.
Mayer, R. E. (1997). Multimedia learning: Are we asking the right questions. Educational Psychologist, 32, 1-19.
McGarr, O. (2009). A review of podcasting in higher education: Its inuence on the traditional lecture. Australasian Journal of
Educational Technology, 25(3), 309-321.
McKinney, D., Dyck, J. L., & Luber, E. S. (2009). iTunes university and the classroom: Can podcasts replace professors? Com-
puters & Education, 52(3), 617-623.
Nicholson, J., & Nicholson, D. B. (2010). A stream runs through IT: Using streaming video to teach information technology.
Campus-Wide Information Systems, 27(1), 17-24.
Oliver, B. (2005). Mobile blogging, Skyping and podcasting: Targeting undergraduates communication skills in transnational
learning contexts. Microlearning, 157-162.
Oxford Advanced Learners Dictionary (2010). Oxford University Press, available at:
http://www.oxfordadvancedlearnersdictionary.com/?cc=global (accessed, November 23, 2010).
Parson, V., Reddy, P., Wood, J., & Senior, C. (2009). Educating an iPod generation: Undergraduate attitudes, experiences and
understanding of vodcast and podcast use. Learning, Media and Technology, 34(3), 215-228.
Piecka, D., Studnicki, E., & Zuckerman-Parker, M. (2008). A proposal for ozone science podcasting in a middle science class-
room. AACE Journal, 16(2), 203-233.
Rosell-Aguilar, F. (2007). Top of the pods--in search of a podcasting podagogy for language learning. Computer Assisted Lan-
guage Learning, 20(5), 471-492.
Scutter, S., Stupans, I., Sawyer, T., & King, S. (2010). How do students use podcasts to support learning? Australasian Journal
of Educational Technology, 26(2), 180-191.
Shantikumar, S. (2009). From lecture theatre to portable media: Students perceptions of an enhanced podcast for revision. Med
Teach, 31(6), 535-538.
Sutton-Brady, C., Scott, K. M., Taylor, L., Carabetta, G., & Clark, S. (2009). The value of using short-format podcasts to
enhance learning and teaching. ALT-J: Research in Learning Technology, 17(3), 219-232.
Taylor, M. Z. (2009). Podcast lectures as a primary teaching technology: Results of a one-year trial. Journal of Political Science
Education, 5(2), 119-137.
Traphagan, T. (2010). Impact of class lecture webcasting on attendance and learning. Educational Technology Research and De-
velopment, 58(1), 19-37.
Walls, S. M., Kucsera, J. V., Walker, J. D., Acee, T. W., McVaugh, N. K., & Robinson, D. H. (2010). Podcasting in education:
Are students as ready and eager as we think they are? Computers & Education, 54(2), 371-378.
Wang, R., Mattick, K., & Dunne, E. (2010). Medical students perceptions of video-linked lectures and video-streaming ALT-J:
Research in Learning Technology, 18(1), 19-27.
Young, J. R. (2008). The lectures are recorded, so why go to class? Chronicle of Higher Education, 54(36), March 26 2010. Re-
trieved from http://chronicle.com/
Yudko, E., Hirokawa, R., & Chi, R. (2008). Attitudes, beliefs, and attendance in a hybrid course. Computers & Education,
50(4), 1217-1227. S
Yunus, A. S. M., Kasa, Z., Asmuni, A., Samah, B. A., Napis, S., Yusoff, M. Z. M., et al. (2006). Use of webcasting technology in
teaching higher education. International Education Journal, 7(7), 916-923.
Findings from Net Geners Multi-Modal and Multi-Task Learning 55
Findings from Net Geners Multi-Modal and Multi-Task Learning
ROBERT ZHENG
Controversy exists concerning the practice of multi-modal and multi-task learning (Gonzalez, Jover, & Cobo, 2010;
Martin, 2008; Skylar, 2009). Some asserted that multi-modal and multi-task learning improves information processing,
thus enhances efciency in learning (Kennedy, Judd, Dalgarno, & Waycott, 2010). Others contended that the access of
information through multiple sensory stimuli and working with multiple tasks simultaneously can short change the quali-
ty of deep learning. Critics of this camp (e.g., Bennett, Maton, & Kervin, 2008) complained about the reduced abilities of
Net Geners in writing, spelling, grammar, etc. showing that the writing norm of the Net Geners is represented by email
and texting styles. They thus concluded that Net Geners may gain breadth in learning, but such knowledge is acquired at
the sacrice of depth in thinking. The goals of this presentation are to explore (1) the cognitive and affective aspects re-
lated to Net Geners online learning and (2) the inuence of individual differences on learners performance where multi-
modal and multi-task learning is implemented.
FACTORS INFLUENCING MULTI-MODAL/MULTI-TASK LEARNING
Researchers have found that factors like the design of multimedia and cognitive load can signicantly inuence
learners performance in learning. For example, Mayer and his colleagues (Mayer & Anderson, 1991; Mayer & Moreno,
2003; Mayer & Sims, 1994) conducted a series of studies to examine the impact of different sensory inputs on compre-
hension and transfer in learning. They found a signicant difference between multimedia and non-multimedia learning.
Mayer and his colleagues concluded that learners performance can be signicantly affected by multimedia.
Sweller and his colleagues (Sweller & Chandler, 1991, 1994; Low, Jin, & Sweller, 2009) propose a cognitive load
theory based on the assumption that human working memory is limited in information processing. While studying the
difculty of learning complex materials, Sweller and his colleagues found that high mental load occurs if (1) the content
is complex enough to make it difcult to learn and (2) the delivery of the content (i.e., content presentation) becomes
so complex that requires extra cognitive resources to process the information. Sweller and Chandler (1991) called the
mental load induced by the rst type of learning as intrinsic load and the second extraneous load. Both intrinsic and
extraneous cognitive loads can be detrimental to learning. This is because our ability to process information is limited
by our working memory (WM) capacity (Baddeley, 1999; Baddeley & Logie, 1992). When intrinsic or extraneous load
increases, the cognitive resources in our WM will decrease. Limited cognitive resources mean a reduced WM capacity in
information processing which can further affects our cognitive information processes such as comprehension, analysis,
and synthesis of new information.
The above multimedia theory and cognitive load theory are important in interpreting the multimodal and multi-
tasking learning phenomenon. First, multimodal learning such as visual, verbal and tactile modes can affect the way the
information is processed. An optimal design with multimodal can facilitate learning. In contrast, poorly designed multi-
media can hinder learning. Next, multi-tasking may induce high extraneous load. This is because our working memory
can get quickly ll up when multi-tasking.
Other factors like visual spatial ability can also inuence learners multimodal and multi-task learning. High spatial
learners are usually more comfortable with visual learning like multimedia (George-Palilonis & Filak, 2010; Hofer,
Prechtl, & Nerdel, 2010). Finally, learning including multi-modal and multi-task learning, can be inuenced by motiva-
tional factor like self-efcacy. Research has shown that self-efcacy is the most central or pervasive variable that inu-
ences learners learning behavior and performance (Bandura, 1993; Lodewyk & Winne, 2005). It can be reasonably as-
sumed that high self-efcacy learners can perform better in cognitive high demanding task like multi-tasking than those
with low self-efcacy. Based on the above literature, the following research questions were proposed as the basis for the
56 Zheng
1. Is there a difference between multi-modal/multi-task and non multi-modal/multi-task learning in terms of
cognitive load, self-efcacy, and achievement performance?
2. Do participants cognitive styles inuence their performance in learning?
THE STUDY
The study examined the differences between multi-modal/multi-task and non multi-modal/multi-task learning in
terms of cognitive load, self-efcacy, cognitive styles, and achievement performance. A total of 43 college students were
recruited. Of 43 participants, 28 (65%) began accessing computers in elementary school, 6 (15%) started as early as kin-
dergarten, and the rest (n=9, 20%) started either in middle or high schools. The participants average age was 20.14 (SD
= 2.44).
Participants rst signed the consent form and were asked to ll out demographic information. Then they took a
spatial visualization test and a pretest in self-efcacy. The study consisted of two experimental conditions: multi-modal/
multi-task learning (experiment group) and non multi-modal/multi-task learning (control group). Participants were ran-
domly assigned to the experiment or the control group based on the results of randomization. Participants in each group
were then given a URL to go to the study webpage where they performed problem solving tasks. After the problem-
solving tasks, the participants were asked to complete a cognitive load questionnaire and a post test in self-efcacy. The
entire study took about 90 minutes.
Problem Solving Tasks (PST). The PSTs consisted of two multiple rule-based problems. They were Tower of Hanoi
and Seating Arrangement. The tasks were counter-balanced to off-set the item carry-over effect in the experiment. Figure
1 shows an example of multiple rule-based problems. For the experiment group, the learners will work on the problems
which were multimodal, that is, the problems were presented with graphics (visual) and manipulation (haptic) forms
of learning (multimodal learning). Learners read the text with graphics and then manipulated the disks by moving the
images around. Learner in the experiment group were also required to monitor the number in a circle by counting and
memorizing the number as it changed.
Figure 1. An Example of Multiple Rule-Based, Multi-Task Problems
Visual Spatial Test. The spatial visual test was adopted from VZ(2) of Kit of Factor-Referenced Cognitive Tests
(Ekstrom, French, Harman, with Derman, 1976). The VZ(2) has 20 problems. The test took about 7 minutes. The total
possible score one could obtain on the test was 20 points. The test reported a reliability of .75 for males and .77 for fe-
males, and an overall reliability of .84 for college students.
Cognitive Load Questionnaire. The measure of cognitive load was adopted with authors permission from an instru-
ment called Cognitive Load Questionnaire developed by Paas (1992). The instrument reported a reliability of .72 to .92.
Table 1 shows the means and standard deviations for problem-solving, cognitive load, changed self-efcacy and
spatial visual scores between experiment and control groups. The changed self-efcacy score was calculated by sub-
tracting pre self-efcacy scores from the post self-efcacy scores. A t-test was performed for the visual-spatial between
experiment and control groups. Results showed no signicant difference between the groups, t(2, 41) = .798, p = .429
(2-tailed). To answer the rst research question, we compared the two groups using t-test based on their cognitive load,
problem solving performance, and self-efcacy scores. The results showed signicant differences for problem solving
performance, t(2, 41) = 4.714, p < .001 (2-tailed); and self-efcacy, t(2, 41) = 2.906, p < .01 (2-tailed). No signicance
was found for cognitive load, t(2, 41) = -.149, p = .882 (2-tailed).
To answer the second research, a Multivariate Analysis of Covariance (MANCOVA) was performed with visual-spa-
tial scores as covariate. Results showed that spatial-visual ability was not a signicant covariate for cognitive load, F(1,
40) = .069, p = .794; problem solving performance, F(1, 40) = .082, p = .776; and self-efcacy, F(1, 40) = .808, p = .374.
Finally, we ran a correlation analysis to determine if the variables under study had signicant correlations among
them. Our ndings indicated that self-efcacy was signicantly correlated with problem solving performance (r = .333),
indicating the higher the performance scores on problem solving, the better the self-efcacy of the participant. Self-ef-
cacy was also negatively correlated with cognitive load (r = -.641), showing the higher the cognitive load, the lower the
self-efcacy of the participant. Problem-solving performance was negatively correlated with cognitive load (r = -.363),
meaning the better the problem-solving performance of the participant, the lower the cognitive load (Table 2).
Table 1. The Descriptive Statistics for problem-solving test, cognitive load, visual-spatial between experiment and con-
trol groups (n=43)
Problem Solv-
Cognitive Load Spatial-Visual Self-Efcacy
Experiment group 3.00(0.54) 6.81(1.24) 10.76(0.53) 26.71(1.87)
Control group 2.18(0.58) 6.86(1.12) 10.63(0.49) 24.50(2.97)
Table 2. The correlation analysis for cognitive load, problem solving performance, self-efcacy and spatial-visual ability
Problem Solving -
Cognitive Load -.363* -
Spatial-Visual .037 -.044 -
Self-Efcacy .333* -.641** -.076 -
* . Correlation is signicant at the 0.05 level (2-tailed)
**. Correlation is signicant at the 0.01 level (2-tailed)
Firstly, the ndings of the study conrmed our prediction that there were differences between multi-modal/multi-
task group and non multi-modal/multi-task group in terms of self-efcacy and problem-solving performance. Secondly,
the ndings indicated signicant correlations among the variables under study which suggests that researchers and prac-
titioners need to put in perspective the relationship among those variable when researching or designing multi-modal and
multi-task learning for Net Geners. Thirdly, we found that cognitive styles (e.g., spatial-visual ability) had no signicant
inuence on learners performance, self-efcacy and cognitive load. This nding is surprising because the literature has
shown that spatial-visual ability is signicantly correlated with multimedia learning (Mayer & Sims, 1994). One possible
explanation is that the current nding was probably confounded in that the dual task which may impose a high cognitive
burden on the leaner could wash away some of the benets the spatial-visual ability brought to learning. For instance,
learners with high spatial-visual ability usually perform well with multimedia because the visuals in multimedia enhance
their cognitive information processes. However, if the learner experienced a high cognitive load, in this case high extra-
neous cognitive load as induced by dual tasks (i.e., multi-tasking), his learning could be seriously affected regardless of
the visual benets the multimedia has brought to him. Following the same line of argument, we examined the results of
cognitive load in this study. Contrary to our previous ndings, the interactive multimedia (i.e., manipulating the images
to solve problems) showed no benets in terms of load reduction. One explanation is that since this study added a dual
task component in the experiment design, learners probably experienced a higher cognitive load than those in our previ-
ous studies. In other words, multi-task learning regardless of the benets of multiple modalities induced the same amount
of load as did the non-multimodal learning.
Research on Net Geners multi-modal and multi-task learning is far and few between. The current study reects the
imminent needs of the eld to understand Net Geners cognitive and affective aspects related to multi-modal and multi-
task learning. This study focused on three important aspects: (a) the differences between multi-modal/multi-task and non
multi-modal/multi-task learning, (b) the inuence of cognitive styles on both types of learning, and (c) the relationships
among the variables under study. To achieve our goals, we examined specically the following variables that related to
Net Geners multi-modal/multi-task learning. They included cognitive load, cognitive styles (e.g., spatial-visual style),
self-efcacy, and learning performance. The ndings of the study revealed that multi-tasks can induce high cognitive
load thus lowers performance in learning. This can happen regardless of the learning benets multiple modalities bring
to learning. Although previous research suggests that multimedia learning is closely related to spatial-visual cognitive
style (Mayer & Sims, 1994), the current study did not nd signicant results for such relationship. This nding points to
an important fact that visual benets for spatial learners can be compromised when learning, in the case of multi-tasking,
incur high cognitive load for learners. Finally, our correlation analyses not only lend support to the above nding but also
reveal the important relationship between variables such as the negative and positive relationships among the factors in-
volved in multimodal and multi-task learning. .
In conclusion, the current study provides preliminary ndings about Net Geners multi-modal and multi-task learn-
ing. It revealed the benets and at the same time the constraints associated with such learning. The study is exploratory.
Further research is needed to verify and conrm the ndings of this study.
Baddeley, A.D. (1999). Essentials of human memory. Hove, England: Psychology Press.
Baddeley, A. D., & Logie, R.H. (1992). Auditory imagery and working memory. In D. Reisberg (Ed.), Auditory imagery. (pp.
179-197). Hillsdale, NJ: Lawrence Erlbaum Associates.
Bandura, A. (1993). Perceived self-efcacy in cognitive development and functioning. Educational Psychologist, 28, 117-148.
Barnes, K., Marateo, R.C., & Ferris, S.P. (2007). Teaching and elarning with the net generation. Innovative 4, Retrieved on
, 2011 from http://www.innovateonline.info/index.php?view=article&id=382
Bennett, S., Maton, K., & Kervin, L. (2008). The digital natives debate: a critical review of the evidence. British Journal of
Educational Technology, 39, 775786.
Ekstrom, R.B., French, J.W., Harman, H.H. with Derman, D. (1976). Kit of factor-referenced cognitive tests. Princeton, NJ:
Educational Testing Service.
George-Palilonis, J., & Filak, V. (2010). Visuals, path control, and knowledge gain: variables that affect students approval and enjoy-
ment of a multimedia text as a learning tool. International Journal on E-Learning, 9(4), 463-480.
Gonzalez, J.A., Jover, L., & Cobo, E. (2010). A web-based learning tool improves student performance in statistics: a randomized
masked trial. Computers & Education, 55(2), 704-713.
Hofer, T.N., Prechtl, H., & Nerdel, C. (2010). The inuence of visual cognitive style when learning from instructional animations and
static pictures. Learning and Individual Differences, 20(5), 479-483.
Kennedy, G., Judd, T., Dalgarno, B., & Waycott, J. (2010). Beyong natives and immigrants: Exploring types of net generation students.
Journal of Computer Assisted Learning, 26, 332-343.
Lodewyk, K. R. &Winne, P. H. (2005). Relations among the structure of learning tasks, achievement, and changes in self-efca-
cy in secondary students. Journal of Educational Psychology, 97, 1, 312.
Mayer, R.E. (2001). Multimedia learning. Cambridge, UK: Cambridge University Press.
Mayer, R. E., & Anderson, R. (1991). Animations and narrations: An experimental test of a dual-coding hypothesis. Journal
of Educational Psychology, 83, 484-490.
Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist,
38(1), 43-52.
Mayer, R. E., & Sims, V. K. (1994). For whom is a picture worth a thousand words? Extensions of a dual-coding theory of mul-
timedia learning. Journal of Educational Psychology, 86(3), 389-401.
Paas, F. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: a cognitive load approach. Journal of
Educational Psychology, 84, 429-434.
Paivio, A. (1986). Mental representations: A dual coding approach. Oxford, England: Oxford University Press.
Sweller, J., & Chandler, P. (1991). Evidence for cognitive load theory. Cognition and Instruction, 8(4), 351-362.
Sweller, J., & Chandler, P. (1994). Why some material is difcult to learn. Cognition and Instruction, 12(3), 185-233.
Device Independent Mobile Applications for Teaching and Learning: Challenges, Barriers and Limitations 61
Device Independent Mobile Applications for Teaching and Learning: Challenges, Barriers
and Limitations
FERIAL KHADDAGE AND GERALD KNEZEK
This unprecedented spread of mobile devices amongst university students has played an important role in the rapid
evolution and development of mobile applications (Apps). This has presented a valuable opportunity for interaction and a
direct distribution channel for developers to produce Apps for educational purposes. These Apps are a form of education-
al inter action delivered via mobile technology and accessed by stu dents from anywhere without them being restricted to
a certain location. The benets and potentials of educational Apps have been discussed by researchers in the educational
community. However, the development and implementation approaches to date have been faced with various challenges
and some technological limitations. This is because the Application Programming Interfaces (APIs) for accessing, pro-
cessing, editing, streaming, and managing multimedia content, are still limited in some mobile devices. Also other issues,
such as small screen size, low battery power, limited processing and memory capacity have emerged. Therefore perform-
ing a simple task such as retrieving an image or attaching a le may take a long time and would consume signicant
amounts of battery power, eventually reducing the mobile device general use time.
Therefore there is a need to mesh the most ideal mobile Apps from a scientic perspective, while at the same time
keeping them simple, cost-effective, efcient, accurate, easily accessible and device independent. The application of mo-
bile technology for learning purposes is boundless, especially because mobile devices are now common amongst stu-
dents, and the mobile phone in particular is the most commonly used device by the majority of students (AMTA, 2007).
Therefore building Apps on the underlying platform of various mobile devices represents a signicant challenge for
institutions to consider. So far there are only a few mobile Apps in the area of higher education, especially at universities,
and there is very little mobile support in the current online course delivery systems at most institutions (Seibu and Biju,
2008). Also since mobile learning forms a crucial part of the e-learning environment and will soon be the sole distributer
of learning content to students at universities (Tatar et all,2003), it is a crucial fact that there is an urgent need to develop
unique approaches that can form the base of new mobile Apps for teaching and learning in higher education. This chal-
lenge could be made possible with the deployment of the emerging new and unique cloud-based computing for teaching
and learning.
In the following sections technological issues in relation to mobile Apps developments are discussed and the fast
emerging Apps and their deployments are reviewed. Also device independent mobile Apps for teaching and learning are
described as well as how these advancements and limitations of current Apps has led to the emerging of a new technol-
ogy called cloud-based computing for teaching and learning in higher education.
Technological and Development Considerations for Mobile Apps
A recent study by the mobile research specialist group indicated that the mobile Apps market has reached 2.2 billion
USD worldwide (Jahns, 2010). This rapid penetration is due to an open, easy and direct access to App stores globally.
Also it is due to the availability of tools and resources for developers to develop manage and maintain Apps for any par-
ticular purpose. Currently there are many stores worldwide and they are constantly expanding at a rapid pace. In fact the
App stores were established only in 2009, and within less than a year the numbers of available Apps have been increas-
ing at an enormous rate. They are still growing rapidly. In the near future users will be able to nd Apps for business,
Apps for health, Apps for education, and Apps for almost anything (Jahns,2010). This will even simplify the process of
Apps integration for teaching and learning for non-technical users from all different educational backgrounds, as the only
task they would be required to do is to get the App and integrate it into their curriculum to support an efcient, effective
and exible delivery mechanism.
62 Khaddage and Knezek
It is worth noting that Apps development and design for teaching and learning require the application of good design
principles. According to some usability experts, such as Jakob Nielsen (2000) and Rolf Molich (2007), usability is an im-
portant aspect to consider during Apps and systems development processes, and especially for the ones that are built for
teaching and learning. Usability is the degree to which the design of a particular interface takes into account the human
aspects and physiology of the students, and makes the process of using Apps effective, challenging and satisfying.
Nielsen and Molich wanted to nd the most effective interface solutions that users prefer, nd easy to use, and are
able to use to maintain their engagements with the Application (Nokia, 2002). The results did vary, depending on how
experienced users are with the technology itself. Most of generation Y users found the Apps very easy and engaging,
while the older generation were slow in grasping the content and preferred the audio instruction provided by some Ap-
plication (Nokia, 2002).
Since designing Apps for teaching and learning to suit everyone is not an easy task, designers should have opted to
support the lowest common denominator, and this should be applied not just in regards to who will be using the Apps,
but also what type of mobile devices are being used to access the Apps.
Designing and implementing mobile Apps for teaching and learning requires also a deep understanding of the major
components of each application, and how the components can be classied into software, hardware, and the network:
The software is the Apps and programs which are developed to a serve a specic purpose, such as, readers, Gmail
mobile, and QQ. These have all attracted millions of active users. The success of these applications is due to their
ease of use, accessibility, and their rich and multiple functionalities (Nokia,2002).
The hardware is all the devices that are able to run the mobile Apps, such as mobile phones, PDAs, Laptops, iPods
and iPads, as well as any device that is capable of running mobile applications.
The network is the wireless application protocol which is responsible for the data connection and transmission.
Operating systems are another major issue. There are eight major mobile operating systems platforms that are being
used today: iOS (iPhone), Android, Symbian, BlackBerry, Java ME, Windows Phone, Flash/Flash Lite and mobile web
which includes WAP (Wireless Application Protocol), XHTML (Extensible Hypertext Mark-up Language), CSS (Cas-
cading Style Sheet) and Java-Script (Parton, 2010). Therefore device independent Apps should be developed to support
all these mobile operating systems in order to enable the Apps to run on the largest possible range of devices. This com-
bination of mobile devices gives the user access to Apps and information anytime, anywhere and via any device. The
minimum requirements should only be that the device supports Java and has access to a data connection, and this is true
for almost all mobile devices that are being used today. Khaddage and Chonka (2009) also identied a list of techno-
logical aspects and design guidelines that should be considered when designing educational content for the small screen
mobile device. They emphasised the technological capabilities of mobile devices, with one of the many important issues
they discussed being about the character encoding, which determines how the pages render in a browser. This is espe-
cially important when developing Apps for a mobile phone device. Correct character encoding should be specied for all
pages, because if it is not specied then the page may display with strange characters. Correct character encoding is es-
sential to ensure that pages render correctly on any mobile device. Different character encodings are required for differ-
ent document types, for example if using XML (Extensible Mark-up Language) documents should always have a UTF-8
character set. UTF-8 is a multi-byte encoding, where each character can be encoded in as little as one byte and as many
as four bytes (Davis, 1999), (Rabin,2008), (Khaddage and Chonka, 2009).
Considerations for Teaching and Learning via Device Independent Mobile Apps
The most important aspects to consider when developing mobile Apps for teaching and learning are: 1) to under-
stand the technological aspects and barriers discussed above, and also 2) to understand how to provide easy access to
content thus making learning content meaningful, benecial, engaging and reachable by all students. It is also impor-
tant that mobile Apps should remain useful long enough for students, and that they should have the ability to maintain
student engagement at all times.
Singh (2003) discussed ten important factors to consider when designing educational content for any mobile device.
Despite the advancements and the rapid technological changes, these factors are still considered valuable, and can be ap-
plied as a framework for designing mobile Apps for teaching and learning. These factors are summarized in [table 1].
Table 1: Factors to consider when designing educational mobile content
Lack of comfort with mobile communication by some
users, login setting and secure details.
Optimizing the devices power management features,
Strain on battery life imposed by mobile Apps
User Interface, how adjustable is the screen resolution,
format and size of displayed font
Touch screen, key in code, voice audio.
keys are difcult for some to use or even see
Form factor Check box option or ll in text
Storage capacity The storage space provided within each device
Processing Power
Certain interactive Apps require high levels of power
consumption when in use
Communications options Via Bluetooth, Wi-Fi, or any other wireless connection
Tools Type of Apps running on the device
High price and the cost of data services Price of voice
services, & Multimedia content Affordable plan types
(at-rate, pay-go etc.)
On the other hand Khaddage and Chonka (2009) discussed the possible use of a le called WURFL (Wireless Uni-
versal Resource) File which is a simple XML le, and can be integrated to identify the mobile device being used, and
can deal with different screen sizes, orientations and device capabilities and retrieve the content according to that specic
But WURFL comes with a few limitations as well, such as: it detects only if the device in use is a mobile phone or
desktop, it requires resources (thus making it harder to implement), a prole must reload each time (this will slow the
process of retrieving the content down), and often the le requires frequent updates to catch up with new technologies,
tools and devices. Therefore it is unlikely that any of these initiatives can be a permanent solution for the current mobile
application, but at the same time, they do offer students more choice, and a wide range of applications will become avail-
able to students via different mobile devices. However, in this new open world of information, students are diverse and
they use different mobile devices and are willing to use them to download and access application, so Apps alone should
not be the only option available to them. Hence comes cloud-based computing as a unique and alternative solution to
overcome most of the barriers and limitations universities have faced while trying to integrate mobile technologies and
applications into their curriculum. Also students with basic and low-tech mobile devices are likely to nd that using
cloud-based computing is both easier and far better suited to the limitations of their device.
Cloud-based Computing Options for teaching and Learning
Cloud-based computing is considered an innovative solution for the above discussed issues in relation to mobile
Apps development. Recently the development of cloud-computing and its application and tools are becoming very
popular for teaching and learning, and are considered a step forward for educational institutions. Therefore develop-
ing a shared and mobile learning environment may now be possible through this advanced and emerging technology
(Hamms,2009).
Cloud-based computing consists of three layers and they form the three building blocks of cloud computing as il-
lustrated in Figure 1, and they are as follow: The Infrastructure as a service (IaaS), the Platform as a service (PaaS) and
the Software as a Service (SaaS). This gives the user the exibility in selecting the appropriate service of their choice
(Creeger, 2009). The combination of the three layers allows users to run and access Apps from anywhere and at anytime
and store information and content online. The three layers can be identied as follow:
The Infrastructure layer (Iaas): is the major foundation layer, where everything is built, it is responsible of the
cloud hosting applications, and user can run any App of their choice, or any existing Apps can be migrated to the
cloud using the cloud hardware.
The Platform layer (PaaS): allows user to develop and implement their own Apps using certain tools resources.
A good example of this is Google Engine App; it allows users to develop their own App tailored to their needs.
The Software layer (SaaS): allows users to access and run existing online Apps such as Google Apps. The most
advantages of this is they are excellent for global collaborative work and they are free of charge.
[Figure 1] is an illustration of the three layers of cloud-based computing.
Figure 1: The three layers of cloud-based computing
Cloud-computing can also be considered the process of develop ing cloud-based Apps that work on multiple devices,
such as mobile, desktops, browsers and Apps within the cloud-based computing environment. This is illustrated in [Fig-
ure 2].
Figure 2: Device independent cloud-based computing
PDAs, iPods, the web, phones, iPads, laptops and other wireless and online technologies can be merged together on
the network. This has the potential to improve the use of different kinds of mobile devices for educational purposes with-
out the full understanding of the programming behind the Apps. Also it has the potential to provide an inex pensive layer
of functionality and performance to the university, while protecting the infrastructure that is increasingly supported by
cloud services and by the technologies that all students bring to campus. The Apps are stored in the one cloud and users
can have fast and secure remote or local access from just about any client device or OS. The plug-and-play feature within
the cloud-computing technology is responsible for publishing the Apps onto the network or the Web. Tools such as cloud-
based email, calendar, chat and forum to improve messaging, as well as interaction capabilities and shared resources,
data, and information can all be integrated. The students can make multiple copies of the learning content, viewable on
a variety of sync-able devices, and hence all users within the same eld can have access to share and use the information
in the account. Cloud computing applications should be designed for scalability to support large numbers of students and
surges in demand. Universities can develop and remotely host custom built educational applications within the cloud,
and this can reduce costs and time spent, thus providing benets to students as well as to the university. In order to create
such Apps it is necessary to develop these applications on the underlying platform and using design principles mentioned
earlier, in order to cover most mobile devices such as iPads, Blackberry, iPhone, Windows Mobile, etc. There are many
good examples of cloud-based mobile applications such as Gmails mobile provided by Google, Google documents, etc.
and they are all accessible via the small mobile device.
Since the current cloud-computing environment enables different platforms to communicate, access and share con-
tent, information and resources, this technology is considered device independent, location independent, exible, global
and can provide an effective collaborative learning environment. Furthermore cloud-based computing is not limited to a
specic physical location and it is not only about having access to mobile Apps and content on demand, but also about
students from diverse backgrounds and different areas of study who can collaborate and share resources. In the near
future that would take universities towards a more open and global educational environment, and ensure greater future
prospects for todays students as they strive to reach out to an open access global learning environment.
CLOUD-BASED COMPUTING BARRIERS AND LIMITATIONS
Security concerns remain an issue for cloud-based computing. Despite what this technology has to offer, it has raised
questions about the security and privacy of data and information sharing processes in a wireless network environment.
All educational institutions are governed by legislation and regulations and the deployment of any new educational tech-
nology is comparatively more difcult than in most other industries. Certain security protocol should be deployed to
protect the security and privacy of students as well as the security of the learning material, especially during the transfor-
mation process in and out of the cloud. The student will connect to the cloud via a mobile device and the requested con-
tent will pass over the Internet and can be a subject to threats. Therefore universities should carefully consider security
aspects when moving data and information into the cloud, as in the near future more appropriate tools for data security
will be developed.
Universities should also ensure that cloud-based data storage and back-up are integrated and deployed. But if the de-
ployment of an Internal Cloud (private access) could be applied, where will the cloud with all the Apps be kept within the
university rewall and security settings? And even this mechanism cannot 100% guarantees that the transmitted content
and learning material does not reach unauthorized users. But again by applying cloud-based computing it is possible that
hackers may nd it difcult to determine the exact physical location of the server that has been used to store content data
Therefore low condence in cloud-based computing security, privacy, authentication and authorization still a major
concern especially with regards to education (Hamms,2009).
Another concern is the bandwidth and the high cost of data transfer, since bandwidth costs do vary with most cloud
computing services, and applications which require frequent data transfer between the cloud and the university may in-
cur high bandwidth charges. In this case a locally hosted service may be more cost efcient. It is also believed that this
limitation will be overcome in the near future, as mobile network operators continue to increase data speeds, and perfor-
mance can only get better.
Mobile technologies and applications for teaching and learning have a promising future and could potentially change
university settings and environments. Cloud computing has the potential to empower mobile learning with its unique
advantages. Since the advent of the Apps and cloud-based computing as an innovative educational model, the uptake of
information on mobile devices has been phenomenal. It seems that applications will soon be the primary modes of pre-
senting information processing on mobile devices. Universities and administrators should begin now to make informed
decisions about mobile technology integration via Apps, tools and devices and provide a set of standards for proper de-
ployment in the near future. It appears that there is no going back. Therefore, in this paper the characteristics and meth-
ods of a device independent mobile learning technologies and applications are analyzed, some technological barriers of
various existing methods are explained; the advantages of a mobile learning environment under a cloud-based comput-
ing, device independent Apps are described.
Australian mobile telecommunication association,(2007) The Impact of the Mobile Phone on Work/Life Balance
Creeger M.,(2009) CTO Roundtable: Cloud Computing Communications of the ACM, vol. 52, no. 8, august 2009, pp. 50-56
Davis, M. (1999) Forms of Unicode September 1999, available online from [Last accessed 12 September 2009] http://www.
ibm.com/developerworks/unicode/library/utfencodingforms/index.html
Hamms. (2009) How cloud computing will change business, Business Week, June 4, 2009. Available online from http://www.
businessweek.com/print/magazine/content/09_24/b4135042942270.htm[Last accessed 16 September 2010]
Jahns, R. Mikalajunaite E. (2010) the smart-phone Application Market Research2guidance the mobile research specialists.
August 2010. Available online from http://www.research2guidance.com/ [Last accessed October 2010]
Khaddage, F. Chonka, A. (2009) E-Learning Over Mobile Phone Technology: Best Practices and Guidelines International
Journal of Interactive Mobile Learning. Available online from http://bb.xieke.com/les/elearning_over_mlearning.pdf [Last ac-
cessed October 2010]
Khaddage, F. Chonka, A. and W. Zhou (2009), Indexing Images for Mobile Phone Learning Content Retrieval Global Con-
ference on Learning Technology, Association for the Advancement of Computing in Education, AACE , May 2010, ISBN
Molich, R. (2007), Usable Web Design, 1st edition, Nyt Teknisk Forlag
Nielsen, J (2000), Why You Only Need to Test with 5 Users, Available online from http://www.useit.com/alertbox/20000319.html
[Last accessed November 2009]
Nokia Corporation (2002) Mobile Application Development Guidelines. Available online from http://www.forum.nokia.com/
[Last accessed Septmeber 2009]
Parton, J. (2010) Mobile Developer Economics 2010 and Beyond VisionMobile 2010. Available online from http://www.vi-
sionmobile.com/research.php#devecon [Last accessed October 2010]
Rabin, J. (2008) Mobile Web Best Practices 1.0, Basic Guidelines W3C Recommendation 2008 Jo Rabin, mTLD Mobile Top
Level Domain,Charles McCathieNevile, Opera Software. Available online from http://www.w3.org/TR/mobilebp/#d0e116
Seibu,M.J. and Biju I.(2008) Mobile Technologies and its Impact An Analysis in Higher Education Context. iJIM Volume 2,
Issue 1, January 2008
Singh, H., (2003) Leveraging Mobile and Wireless Internet. Available online from http://www.learningcircuits.com/2003/sep2003/
singh.htm [Last accessed July 2010]
Tatar D., Roshelle J., et all, Handhelds Go to School: Lesson Learned, Communications of the ACM, September 2003.
Towards Personalised and Adaptive Multimedia in M-learning Systems 67
Towards Personalised and Adaptive Multimedia in M-learning Systems
ARGHIR-NICOLAE MOLDOVAN AND CRISTINA HAVA MUNTEAN
Mobile devices such as smartphones, tablets and netbooks are fast becoming the primary means of accessing the
Internet. Driven by the fast increase in mobile device sales, mobile Internet not only that grows signicantly faster, but
this is expected to become ten times bigger than desktop Internet in terms of device numbers (Meeker 2009). The fact is
further indicated by various forecasts, that estimate yearly global shipments of smartphones and tablets alone to be close
to 850 million (In Stat 2011), and more than 100 million (Abi Research 2011) respectively, by 2015.
As mobile devices have improved signicantly in terms of features and capabilities, they are increasingly used for
accessing m-learning applications. Latest embedded devices come with larger and crisper screens, high-speed wireless
connectivity and faster processors, enabling learners access to rich educational multimedia content anytime and any-
where. At the same time, educators can create more engaging, interactive and multimedia rich m-learning applications
in order to enhance learners experience and their satisfaction, key factors contributing to the acceptance of m-learning
systems (Liaw et al. 2010).
In particular, video content production and consumption for educational purposes has seen a fast growth and is ex-
pected to accelerate over the coming years (Kaufmann & Mohan 2009). Educational multimedia clips offer a rich display
of information and can help learners understand difcult concepts, while real-time multimedia streaming enables face-
to-face-like experiences in online learning environments. However, high-quality multimedia requires signicant network
bandwidth and device resources in order to be properly retrieved and displayed by the learner device. Learners may have
mobile devices with different characteristics (e.g. screen size, CPU power), and multimedia playback capabilities, which
have to be considered for encoding the educational multimedia content at an optimum quality level. Moreover, during its
delivery the quality of the multimedia clip can be signicantly affected by a low or variable network bandwidth. There-
fore, each mobile learner needs to be provided with multimedia educational content encoded with appropriate parameters
and quality level, so that it can be displayed properly by his/her mobile device, and can be delivered over the available
wireless network. To achieve this, mechanisms for personalising the multimedia content based on learners device char-
acteristics and for adapting the content to the available network conditions need to be integrated with m-learning sys-
tems.
This paper extends the signicant body of research that addressed educational content personalisation and adapta-
tion in e-learning generally and m-learning particularly, by proposing a solution for enabling educational multimedia
content personalisation and adaptation in m-learning systems. Personalisation based on learner device screen resolution
and adaptation based on the available network bandwidth are particularly addressed. To overcome the multitude of device
screen resolutions, learners devices are grouped in different classes, and video proles consisting of recommended en-
coding settings are associated to each class. While intervals of recommended values for streaming educational multime-
dia clips are provided, selecting specic values for encoding particular clips is especially difcult. This is because both
content related factors such as the clip dynamicity, and network transmission conditions, have to be considered in order
to provide learners with a good quality level. Content specic bitrate-selection is enabled with the help of an automatic
mechanism that uses objective video quality assessment metrics to estimate learner perceived quality of the multimedia
clips. A subjective study was conducted using educational multimedia clips with different characteristics. The results
presented in this paper have shown that the bitrate values selected using the automatic mechanism offer a good enough
quality level for enabling knowledge acquisition from the multimedia clips.
The rest of the paper is organised as follows. Next section presents research background related to this paper. Fol-
lowing, the proposed solution for enabling multimedia content personalisation and adaptation in m-learning systems is
described. The subjective study methodology and results are presented after that, while in the end conclusions are drawn.
68 Moldovan and Muntean
To support learners access to educational content anytime and anywhere, m-learning has to overcome a number
of issues and challenges that are mainly related with the learners (e.g. their needs, preferences, location, etc.), as well
as with the variety of mobile device characteristics. To overcome these challenges, personalisation and adaptation have
gradually been brought to the forefront of the research in the area of m-learning in particular and e-learning in general.
Educational content personalisation and adaptation may be learner oriented or device oriented.
Approaches for learner oriented personalisation have mainly concentrated on delivering personalised educational
content tailored to individual learners or groups based on their characteristics. Various learner characteristics were con-
sidered as important input in the personalisation process such as their needs, preferences, goals, knowledge level, skills
or learning styles (De Bra et al. 2003; Brusilovsky & Milln 2007; Graf et al. 2009; Chang et al. 2009). Learners loca-
tion was also considered for personalising the educational content to the learning context (Yin et al. 2010).
Delivering learning content to mobile devices requires the m-learning applications to be able to run efciently on
a multitude of mobile devices with different characteristics (e.g. device screen and its resolution, CPU speed, memory
capacity, I/O interfaces (e.g. touch screen, keyboard), wireless connectivity (e.g. WiFi, 3G)), different operating systems,
as well as different features and capabilities (e.g. web-browser performance, audio/video formats supported, Flash/Java
support, etc.).
The majority of the device oriented personalisation and adaptation solutions have mainly focused on the mobile
device screen resolution. Various frameworks for creating device independent user interfaces were proposed (Ally et al.
2005; Zhao & Okamoto 2008), which separate the learner interface from the educational content and use a device inde-
pendent language such as XML to describe the interface. The user interface is then automatically generated, based on
the mobile device characteristics. Device connectivity related factors such as the available network bandwidth, were also
addressed in the educational content adaptation (Muntean 2008). Most of the research addressing learner device oriented
personalisation and adaptation of educational content, has focused on adapting the content presentation, where the con-
tent is represented by various pieces of information such as text, images, audio and/or video clips.
Educational multimedia content personalisation and adaptation in particular, may also be driven by learner charac-
teristics and the learning context, as well as by the device characteristics and network related factors (Muntean C.H. &
Muntean 2009). Device characteristics such as processing power, screen resolution (Kim & Yoon 2009) and the battery
life (Moldovan & Muntean 2009), as well as the available wireless network bandwidth (Muntean V.H. & Muntean 2009),
have been addressed.
Multimedia content personalisation usually is done a single time before the content delivery. This consists in provid-
ing a learner with educational material that better ts his/her needs, or with a version of the same multimedia content
encoded in a way that meets the device and network constraints. Encoding parameters that can be changed include the
compression format, resolution, bitrate and frame-rate, while the versions can be created off-line or on-line using real
time transcoding.
Additionally, multimedia content adaptation involves changing the content characteristics during its delivery, to
overcome factors such as, e.g. the variable network bandwidth. This can be done by switching between several versions
of the same clip created off-line (Zambelli 2009), by re-encoding the clip in real-time or by using a scalable video codec
(Schwarz et al. 2007).
Enabling Educational Multimedia Content Personalization and Adaptation
This section presents the proposed solution for enabling multimedia content personalisation based on learner device
screen resolution, and multimedia content adaptation based on the available network bandwidth. Various types of mobile
devices that are most suitable for m-learning are presented rst.
M-learning Devices
Currently there are a multitude of mobile devices that can be used for m-learning. These so called m-learning de-
vices (Quinn 2008), differ broadly in terms of characteristics such as device size and its form-factor, screen size and its
resolution, wireless/cellular connectivity type and speed, CPU speed, memory capacity, I/O components, etc.
Although Feature Phones with Web connectivity and Laptops can be used for m-learning, they either have poor
performance and limited functionality (feature phones), or they are too bulky and have poor battery life for enabling
true mobility (laptops). As opposed, Smartphones and Ultra Mobile Devices (UMDs) (see Figure 1), not only have
more advanced web-browsing and multimedia capabilities than feature phones, but they also are more compact and
lightweight, and often have better battery life than laptops. UMDs cover a multitude of mobile devices such as Mobile
Internet Devices (MIDs), Tablets, Ultra Mobile PCs (UMPC), Netbooks, Smartbooks, etc.
Apart of smartphones and UMDs other devices with wireless connectivity but more dedicated functionalities such
as Portable Media Players (PMPs), Handheld Game Consoles (HGCs) or e-readers can also be used for m-learning.
However such devices may be expected to decrease in popularity as mobile users increasingly rely on their capable
smartphones or tablets to perform more and more activities, from browsing the Internet, to watching videos and listening
music, playing games, reading e-books, etc.
Figure 1: Common form-factors, screen sizes and screen resolutions of the latest m-learning devices
Looking at the screen size and form-factor (see Figure 1), one can note the similarities and the differences between
the various types of m-learning devices. Smartphones usually are more compact, may present a physical keyboard and/
or a touch-screen, and as of 2011 may have a screen size as large as 4.5
in the touch-screen variant. UMDs as opposed
come in a signicant higher variation of form factors and may have a screen size as small as 4 or as large as 12. Figure
1 also shows that different m-learning devices with different screen sizes may present the same screen resolution. For ex-
ample, a resolution of 1024x600 pixels can be seen in a 10.2 Netbook
, a 7 tablet
, as well as in a 4.8 UMPC
Video Proles for Classes of M-learning Devices
The multitude of screen resolutions with different aspect ratios used by m-learning devices, make difcult educa-
tional multimedia clips personalisation based on learners device screen resolution. Ideally, for optimum quality level
each learner should receive a version of the multimedia clip adapted to t its particular screen resolution. However, in
practice this is not feasible due to the very high number of versions that would need to be created and stored for each
multimedia clip.
1 [] Samsung i997 Infuse 4G Specications, http://www.gsmarena.com/samsung_i997_infuse_4g-3705.php
2 [] Samsung NP-NC10 Specications, http://www.umpcportal.com/products/Samsung/NC10/NP-NC10
3 [] Samsung P1000 Galaxy Tab Specications, http://www.gsmarena.com/samsung_p1000_galaxy_tab-3370.php
4 [] Viliv N5 Specications, http://www.umpcportal.com/products/Viliv/N5
A list of the most common m-learning devices screen resolutions, compiled using information from Cartoonized
and UMPC Portal
, is presented in Table 1. To reduce the number of versions to be created, this paper proposes to group
the m-learning devices in four different classes based on their screen resolution: Large Screen Resolution Devices (LRD),
Medium-Large Screen Resolution Devices (MLRD), Medium-Small Screen Resolution Devices and Small Screen Resolu-
tion Devices (SRD).
Table 1: Common display resolutions for m-learning devices
M-Learning
LRD 720p 1600x768 25:12 UWXGA Netbooks
1366x768 16:9 WXGA Netbooks, Tablets
1280x800 16:10 WXGA Netbooks, MIDs & UMPCs
1280x768 5:3 WXGA Netbooks
1024x768 4:3 XGA Netbooks, Tablets
MLRD 480p 1024x600 16:10 WSVGA Netbooks, Tablets, MIDs
1024x576 16:9 WSVGA Netbooks
1024x480 UWVGA Smartphones, MIDs
960x640 3:2 DVGA Smartphones
960x540 16:9 qHD Smartphones
960x480 2:1 UWVGA Smartphones, MIDs
854x480 16:9 FWVGA Smartphones, MIDs
800x600 4:3 SVGA MIDs & UMPCs, Tablets
800x480 5:3 WVGA Smartphones, MIDs, Netbooks, Tablets
640x480 4:3 VGA Smartphones, PDAs,
MSRD 360p 640x360 16:9 nHD Smartphones
480x360 4:3 HVGA+ Smartphones
SRD 240p 480x320 3:2 HVGA Smartphones, PMPs, MIDs
480x272 16:9 HD1080/16 Smartphones, PMPs, HGCs
432x240 9:5 WQVGA Smartphones
427x240 16:9 WQVGA Smartphones
320x240 4:3 QVGA Smartphones, PMPs, PDAs, MIDs
A video prole consisting of a set of video encoding parameters (video codec, reference resolution, bitrate and
frame-rate) is associated to each device class. As m-learning devices may have screen resolutions as low as 320x240 and
as high as 1600x768, the 240p, 480p, 360p and 720p, video proles are used. The video prole names indicate the verti-
cal resolutions at which the educational multimedia clips are encoded.
Since the m-learning devices screen resolution, as well as the educational multimedia clips video resolutions may
have different aspect ratios, both resolutions should be considered when selecting the prole for a particular learner de-
vice. For example a mobile device with a 640x480 resolution can easily accommodate the 480p prole (640x480 clip
resolution) for an educational clip with a 4:3 aspect ratio, but only the 360p prole (640x360 clip resolution) for an edu-
cational clip with a 16:9 aspect ratio.
The 16:9 aspect ratio has become very popular in recent years, being widely adopted in HD cameras (720p and
1080p), television and displays. Moreover a large number of m-learning devices use wide screen resolutions such as
nHD, WVGA or WXGA, that are more suitable to display educational clips with a 16:9 aspect ratio. Figure 2 illustrates
the reference resolutions with an aspect ratio of 16:9, corresponding to the four video proles. As indicated in the gure,
the resolution basically doubles in size with every prole.
5 [] Cell Phone Screen Resolution by Brand and Model, http://cartoonized.net/cellphone-screen-resolution.php
6 [] Netbook, Tablet, UMPC and MID Comparison, http://www.umpcportal.com/products/
+ 44%
Figure 2: Reference resolutions for the four video proles
For a particular video resolution and frame rate, the video bitrate of a compressed educational multimedia clip de-
pends on the video codec being used, with some codecs offering better compression for similar quality level (e.g. bad,
good or excellent). Furthermore, depending on the content characteristics (e.g. details, motion content, colours used,
etc.), different educational clips may require different bitrates for the same quality level, with more complex and dynam-
ic clips usually requiring a higher bitrate.
Various aspects need to be addressed when selecting the video codec and the bitrate for encoding educational mul-
timedia clips. Examples include but are not limited to what video codec is supported by learners device and/or applica-
tion? The videos are intended for downloaded or for streaming? If they are streamed what is the available wireless net-
work bandwidth?
Examples of popular video codecs are H.264/MPEG-4 AVC, Theora, VP8 and VC-1. H.264 is the latest version of
the MPEG-4 video codec standard, providing approximately 50% bit rate saving as compared to the previous generation
MPEG-4 Part 2, for the same quality level (Wiegand et al. 2003). Over the past years the format was increasingly adopt-
ed by Internet multimedia services such as IPTV, Video on Demand, video conferencing or video sharing (e.g. YouTube).
Furthermore, an increasing number of mobile devices come with H.264 decoding capabilities.
Table 2 presents intervals of bitrate values corresponding to the four video proles, which can be used for encoding
H.264 educational multimedia clips for both local playback and streaming to m-learning devices over WiFi. The bitrate
intervals are based on guidelines and recommendations for encoding H.264 multimedia content for local playback
, and correspond to standard video frame-rates such as 24fps, 25fps and 30fps.
7 [] Creating Amazing Video Content with H.264, http://www.apple.com/mx/quicktime/tutorials/h264.html
8 [] Dynamic streaming with FMS 3.5, http://www.adobe.com/devnet/ashmediaserver/articles/dynstream_on_de-
mand.html
9 [] Welcome Streaming Media Roundtable Discussion with Four Thought Leaders in the Industry, http://event.on24.
com/event/28/89/58/rt/1/documents/slidepdf/stm_slides_1095.pdf
Table 2: Video proles for classes of m-learning devices
[pixels]
Local Playback
[Kbps]
[fps]
LRD 720p H.264 1280x720 5000-6000 1500-2200 24-30
MLRD 480p 854x480 1000-2000 600-1000
MSRD 360p 640x360 500-1000 350-550
SRD 240p 427x240 300-500 150-300
Selecting Specic Bitrate Values
When creating personalised educational multimedia clips for m-learning, specic bitrate values need to be selected
in the recommended intervals for each video prole. While the high bitrates recommended for download clips enable ex-
cellent visual quality, it is not clearly mentioned and backed up by evidence what quality offer the recommended stream-
ing bitrates. Since video quality is one of the main factors contributing to learners experience, it is important to select
bitrate values that offer at least a good level of learner perceived quality. Moreover, when an educational multimedia clip
is streamed to the learner device over the wireless network, its perceived quality may be signicantly affected by trans-
mission errors such as loss and delay, especially during real-time streaming (e.g. student watching a live lecture record-
ing). In such situations, adapting the bitrate of the streamed clip for enabling a smooth viewing experience may result in
a higher overall quality, than streaming the clip at a higher but xed bitrate.
To enable educational multimedia content adaptation based on the network conditions, a bitrate value as small as
possible that still offers a good learner perceived quality has to be detected for each video prole. An automatic mecha-
nism, BitDetect, for detecting thresholds up to which the bitrate of a multimedia clip can be decreased while maintaining
a good quality level, was proposed for the particular case of saving the mobile device battery life (Moldovan, Moraru, &
Muntean 2011). BitDetect takes as input an excellent quality multimedia clip with a specic resolution and a high bitrate
suitable for local playback, and returns a good quality version of the clip having the same resolution but a signicantly
lower bitrate that is suitable for streaming. This is done by gradually decreasing the video bitrate and estimating the qual-
ity of the resulting versions using objective video quality assessment metrics. Two objective metrics, the Peak Signal-to-
Noise Ratio (PSNR) (Wang & Bovik 2009) and Structural Similarity Index (SSIM) (Wang et al. 2004) are used.
Although the mechanism was proposed for the particular context of battery power saving, this can be used as well
for detecting good quality bitrate values for enabling the educational multimedia clips adaptation based on network con-
ditions. Since different educational clips may have different characteristics (e.g. dynamicity, colours), they may require
different bitrates for a good quality level. Therefore, the good quality bitrate value has to be detected individually for
each multimedia educational clip and for each video prole. This process can be facilitated by integrating an automatic
mechanism such as BitDetect with an adaptive m-learning system (Moldovan, Molnar & Muntean 2011).
Case Study and Experimental Results
A subjective study was conducted in order to assess if the specic bitrate values recommended by the BitDetect
mechanism are suitable for multimedia-based m-learning. The goal of the study was twofold: (i) to assess how learners
perceive the quality of different educational multimedia clips encoded at the recommended bitrate values, and (ii) to as-
sess if the quality is good-enough so that learners can achieve knowledge from the clips. To assess the results consistency
across different video proles and educational multimedia clips, the study was conducted for two video proles (360p
and 480p) using eight educational multimedia clips with different content characteristics.
Educational Clips
Eight clips were selected from a list of 904 educational multimedia clips that were downloaded from iTunes U
Miro Guide
. Eight categories of educational multimedia clips were identied during the selection process: slideshows,
interviews, lab demos, screencasts, graphics (3d game and virtual world recordings), presentations, documentaries, and
(computer generated) animations. Each multimedia clip that was selected corresponds to one of these categories. Repre-
sentative frames for the eight clips are presented in Figure 3. More detailed descriptions of the multimedia clips can be
found in (Moldovan, Molnar & Muntean 2011). The high quality original clips were H.264 compressed and had a resolu-
tion of 960x540 or 1280x720 pixels, a frame-rate of 24 fps or 30 fps, and a bitrate between 2 Mbps and 5 Mbps.
Two up to 30 seconds long test sequences were extracted from each educational clip. Eight of the sixteen sequences
(Arts A, Dubus A, etc.), were re-encoded at the reference resolution corresponding to the rst prole tested, 360p, while
the other eight sequences (Arts B, Dubus B, etc.) were re-encoded at the reference resolution corresponding to the sec-
ond prole tested, 480p. A low, good quality bitrate value was detected individually for each of the sixteen sequences us-
ing the BitDetect mechanism. For the sequences corresponding to the 360p prole, the bitrate was detected as 512 Kbps
for the Sleep A sequence and 256 Kbps for the other seven sequences. For the sequences corresponding to the 480p pro-
le, the bitrate was detected as 768 Kbps for the Sleep B sequence and 384 Kbps for the other seven sequences. This sug-
gests that documentaries usually require higher bitrate due to the higher content dynamicity. The encoding characteristics
of the sixteen test sequences are presented in Table 3. The original frame-rate of the clips was maintained, while the rest
of encoding settings (e.g. audio bitrate) were maintained constant across the sequences.
10 [] Apple iTunes U, http://www.apple.com/itunes/
11 [] Miro Guide, http://www.miroguide.com/
Arts (Slideshow) Dubus (Interview) Hotness (Lab Demo)
Hulu (Screencast) Languagelab (Graphics) Obesity (Presentation)
Sleep (Documentary) Sol (Animation)
Figure 3: Representative frames for the eight educational multimedia clips used for testing
Table 3: Encoding characteristics of the sixteen test sequences
[sec]
Audio Encoding
Arts A / B 20 / 27 H.264 640x360 /
256 / 512 24 Audio Codec: AAC
Audio Bitrate: 128
2 channels (Stereo)
Dubus A / B 30 / 25 256 / 512 30
Hotness A / B 30 / 30 256 / 512 24
Hulu A / B 27 / 21 256 / 512 30
Languagelab
A / B
30 / 30 256 / 512 30
Obesity A / B 30 / 30 256 / 512 30
Sleep A / B 26 / 20 384 / 768 30
Sol A / B 30 / 30 256 / 512 30
Each participant to the study was asked to view the sixteen test sequences encoded as presented in Table 3. The test
sequences corresponding to the 360p prole were viewed rst on a HP iPAQ 214 PDA with a 640x480 pixels resolution.
The sequences corresponding to the 480p prole were viewed next on a Dell Inspiron Mini 10 netbook with a 1024x600
pixels resolution. Similar conditions were maintained for all the participants and standard recommendations (ITU-T
2009) for video quality assessment were followed.
After viewing each sequence, the participants were asked to rate their overall quality on a ve-point scale (1-Bad,
2-Poor, 3-Fair, 4-Good, 5-Excellent) and to answer an educational question related strictly to the visual information pre-
sented in that clip. Twelve of the sixteen questions were multiple choice type with only one correct answer, two questions
were yes/no type, and two questions were short answer type. An assumption was made that if learners are able to answer
correctly the questions after viewing each clip only once, they are also able to acquire knowledge from the clip. The
sound was maintained in order to replicate as much as possible a normal viewing experience. However, additional care
was made that the answers to the questions are found solely in the video.
The decision to use different sequences from each clip for the two proles was made in order to avoid inuencing
the answers provided by the participants. If the same sequence is displayed on both devices, a participant would have
answered the rst question after seeing only once the sequence on the rst device, and the second question after viewing
the sequence twice, on both devices. 21 (M = 13, F = 8) subject with ages between 21 and 37 years old (AVG = 27.14),
have participated in the study.
Video Quality Assessment Results
For each of the sixteen test sequences the video quality ratings on the 1-5 scale were averaged across all 21 partici-
pants, obtaining the Mean Opinion Scores (MOS). The standard deviation (STDEV) of the statistical distribution of the
assessment ratings across the participants was also computed for each sequence.
The quality assessment results are presented in Figure 4. The results show that the majority of the sixteen sequences
have scored close to 4 (Good). One sequence corresponding to the 360p prole (Languagelab A), and two sequences cor-
responding to the 480p prole (Arts B and Obesity B) have scored slightly lower than 4, but higher than 3.5. The aver-
age MOS scores for the two proles are both equal with 4.1, while a t-test on the mean values indicate that there is no
statistical difference in the nal scores for the two proles being tested ( = 0.01, t = 0.16, t-critical = 2.99, p(t) = 0.43, r
= 0.81). For both proles the sequence with the lowest MOS score has the highest STDEV (Languagelab A, MOS = 3.6,
STDEV = 1.12; Arts B, MOS = 3.7, STDEV = 1.10). The Pearson correlation further indicates that there is decreasing
relationship between the MOS and STDEV values (r = 0.678 for 360p, r = 0.732 for 480p, r = 0.704 overall), thus
the ratings across participants tend to have a higher variation, for the clips with lower perceived quality.
The quality results show that the bitrate values recommended by the BitDetect mechanism offer a good level of quality
as perceived subjectively by learners, across different video profles and different educational multimedia clips.
Test Sequences
Video Quality Results360p
MOS STDEV
Video Quality Results 480p
Figure 4: Subjective video quality assessment results for the two proles
Knowledge Transfer Assessment Results
The second goal of the study was to assess if the quality is also good-enough for enabling learners to acquire knowl-
edge from the clips. For each educational question corresponding to one of the sixteen test sequences, the total number of
correct answers provided by the participants, and the average correct response rate was determined. The results presented
in Figure 5, show that for the majority of the clips, the majority of the subjects have answered the questions correctly.
Only for a single question, less than 50% of the participants have provided the correct answer, although the correspond-
ing test sequence (Sleep B) has achieved a MOS equal to 4. Statistical analysis on this data set also shows that there is
no relationship between the MOS scores and the number of correct answers (r = 0.357 for 360p, r = 0.165for 480p, r =
0.015 overall). Although the average correct response rates of 91% (360p) and 85% (480p), indicate that the bitrate val-
ues provide a good-enough quality for enabling knowledge transfer, further testing is needed for detecting how the video
quality impacts the learning outcome.
Question Answering Results 360p
No. Correct Answ. % Correct Answ.
100%100%
Figure 5: Numbers and percentages of correct answers to the educational questions for the two proles
This paper presented a solution for enabling educational multimedia content personalization and adaptation in m-
learning systems. Device screen resolution driven personalisation and available wireless network bandwidth driven ad-
aptation are targeted. To enable personalisation based on learner device screen resolution mobile devices were grouped
in four classes depending on their screen resolution: Large Screen Resolution Devices, Medium-Large Screen Resolution
Devices, Medium-Small Screen Resolution Devices and Small Screen Resolution Devices. Video proles consisting of
recommended encoding settings are proposed for each device class. To enable educational multimedia content adaptation
based on the available bandwidth, low video bitrate values still offering good quality levels are detected using the BitDe-
tect mechanism.
A subjective study was conducted to assess if the bitrate values recommended by BitDetect for the eight educational
multimedia clips encoded at two different resolutions are suitable for m-learning. The results have shown that the bitrate
values offer a good quality for the majority of the test sequences, a 4.1 score (Good on a 1-5 scale) being achieved on
average. Furthermore, the results have conrmed that the bitrate thresholds offer a good-enough quality for enabling
knowledge transfer, as overall the subjects have answered correctly to questions related to the information presented in
the clips in 88% of the cases.
The support of IRCSET (Irish Research Council for Science, Engineering & Technology) Embark Initiative is greatly
acknowledged.
ABI Research. (2011). Apple iPad Held 85% Media Tablet Market Share in 2010. Retrieved from http://www.abiresearch.com/
press/3665-Apple+iPad+Held+85%25+Media+Tablet+Market+Share+in+2010
Ally, M., Lin, F., McGreal, R., & Woo, B. (2005). An Intelligent Agent for Adapting and Delivering Electronic Course Materials
to Mobile Learners. Presented at the 4th World Conference on Mobile Learning (mLearn 2005), Cape Town, South Africa.
Retrieved from http://www.mlearn.org.za/CD/papers/Ally.pdf
Brusilovsky, P., & Milln, E. (2007). User Models for Adaptive Hypermedia and Adaptive Educational Systems. In P. Brusi-
lovsky, A. Kobsa, & W. Nejdl (Eds.), The Adaptive Web, Lecture Notes in Computer Science (Vol. 4321, pp. 3-53). Berlin,
Heidelberg: Springer-Verlag.
Chang, Y.-C., Kao, W.-Y., Chu, C.-P., & Chiu, C.-H. (2009). A Learning Style Classication Mechanism for E-Learning. Com-
De Bra, P., Aerts, A., Berden, B., de Lange, B., Rousseau, B., Santic, T., Smits, D., et al. (2003). AHA! The Adaptive Hyperme-
dia Architecture. Proceedings of the 14th ACM Conference on Hypertext and Hypermedia. Nottingham, UK. 81-84.
Graf, S., Yang, G., Liu, T. C., & Kinshuk, D. (2009). Automatic, Global and Dynamic Student Modeling in a Ubiquitous Learn-
ing Environment. Knowledge Management & E-Learning: An International Journal (KM&EL), 1(1), 18-35.
In-Stat. (2011). More Than Half of US Handset Shipments Will be Smartphones by 2012, Worldwide Smartphone Ship-
ments Move Toward 1 Billion by 2015. Retrieved from http://www.instat.com/newmk.asp?ID=3012&Source
ID=00000501000000000000
ITU-T. (2008). P.910: Subjective video quality assessment methods for multimedia applications. Geneva, Switzerland: Interna-
tional Telecommunication Union. Retrieved from http://www.itu.int/rec/T-REC-P.910-200804-I/en
Kaufman, P., & Mohan, J. (2009). Video Use and Higher Education: Options for the Future. New York, USA: Intelligent Televi-
sion, New York University. Retrieved from http://library.nyu.edu/about/Video_Use_in_Higher_Education.pdf
Kim, S., & Yoon, Y.-I. (2009). Adaptation E-learning Contents in Mobile Environment. Proceedings of the 2nd International
Conference on Interaction Sciences: Information Technology, Culture and Human (ICIS 09) (pp. 474479). New York,
NY, USA: ACM
Liaw, S.-S., Hatala, M., & Huang, H.-M. (2010). Investigating Acceptance toward Mobile Learning to Assist Individual Knowl-
edge Management: Based on Activity Theory Approach. Computers & Education, 54(2), 446-454.
Meeker, M. (2009). Economy + Internet Trends. Presented at the 2009 Web 2.0 Summit, San Francisco, CA. Retrieved from
http://www.morganstanley.com/institutional/techresearch/pdfs/MS_Economy_Internet_Trends_102009_FINAL.pdf
Moldovan, A.-N., & Muntean, C. H. (2009). Personalisation of the Multimedia Content Delivered to Mobile Device Users.
Presented at 4
IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB 09), Bilbao,
Spain. doi:10.1109/ISBMSB.2009.5133750
Moldovan, A.-N., Molnar, A., & Muntean, C. H. (2011). EcoLearn: Battery Power Friendly e-Learning Environment for Mobile
Device Users. In A. Lazakidou & I. El Emary (Eds.), Learning-Oriented Technologies, Devices and Networks - Innovative
Case Studies (pp. 273-296). Saarbrcken, Germany: LAP LAMBERT Academic Publishing.
Moldovan, A.-N., Moraru, C. A., & Muntean, C. H. (2011). MediaMTool: Multimedia Content Management Tool. Presented at
IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB 11), Metropolitan Area
Nuremberg, Germany. doi:10.1109/BMSB.2011.5954907
Muntean, C. H. (2008). Improving learner quality of experience by content adaptation based on network conditions. Computers
in Human Behavior, 24(4), 1452-1472.
Muntean, C. H., & Muntean, G.-M. (2009). Open corpus architecture for personalised ubiquitous e-learning. Personal and
Ubiquitous Computing, 13(3), 197-205.
Muntean, V. H., & Muntean, G.-M. (2009). A Novel Adaptive Multimedia Delivery Algorithm for Increasing User Quality of
Experience during Wireless and Mobile E-learning. Presented at 4
IEEE International Symposium on Broadband Multi-
media Systems and Broadcasting (BMSB 09), Bilbao, Spain. doi:10.1109/ISBMSB.2009.5133739
Quinn, C. (2008). mLearning Devices. Quinnovation. Retrieved from http://quinnovation.com:8000/MobileDevices.pdf
Schwarz, H., Marpe, D., & Wiegand, T. (2007). Overview of the Scalable Video Coding Extension of the H.264/AVC Standard.
IEEE Transactions on Circuits and Systems for Video Technology, 17(9), 11031120.
Wang, Z., & Bovik, A.C. (2009). Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures. IEEE Signal
Processing Magazine, 26(1), 98-117.
Wang, Z., Lu, L., & Bovik, Alan C. (2004). Video Quality Assessment Based on Structural Distortion Measurement. Signal Pro-
cessing: Image Communication, Special Issue on Objective Video Quality Metrics, 19(2), 121132.
Wiegand, T., Sullivan, G. J., Bjontegaard, G., & Luthra, A. (2003). Overview of the H. 264/AVC video coding standard. IEEE
Transactions on Circuits and Systems for Video Technology, 13(7), 560576.
Yin, C., Ogata, H., Tabata, Y., & Yano, Y. (2010). Supporting the Acquisition of Japanese Polite Expressions in Context-Aware
Ubiquitous Learning. International Journal of Mobile Learning and Organisation (IJMLO), 4(2), 214234.
Zambelli, A. (2009). IIS Smooth Streaming Technical Overview. Microsoft Corporation. Retrieved from http://dvb-crew.atw.hu/
applications/documents/IIS_Smooth_Streaming_Technical_Overview.pdf
Zhao, X., & Okamoto, T. (2008). A Device-Independent System Architecture for Adaptive Mobile Learning. Proceedings of the
8th IEEE International Conference on Advanced Learning Technologies 2008 (ICALT 08), Santander, Spain. 23-25.
Knowledge Creation in MMOG: An Empirical Study 79
Knowledge Creation in MMOG: An Empirical Study
JOSEPH SIU-LUNG KONG AND RON CHI-WAI KWOK
This study aims to empirically examine the four modes of collaborative knowledge creation processes in Massively
Multiplayer Online Game (MMOG) game-play, and their dynamic relationship with game players engagement level,
duration and perceived addictiveness.
From the gaming perspective, MMOGs are dened as games which are highly graphical 2- or 3-D videogames
played online, allowing individuals, through their self-created digital characters or avatars, to interact not only with the
gaming software (the designed environment of the game and the computer-controlled characters within it) but with other
players avatars as well. These virtual worlds are persistent social and material worlds, loosely structured by open-ended
(fantasy) narratives, where players are largely free to do as they please slay ogres, siege castles, barter goods in town,
or shake the fruit out of trees (Steinkuehler 2004).
From the technology-mediated collaborative learning prospective, MMOG consists of several unique characteristics
(e.g. avatar/virtual identity, co-presence, group identity and transparency) that may trigger collaborative learning behav-
ior in the game context. Recently, an increasing number of researchers (e.g. Gee 2004, Childress and Braswell 2006,
Mikropoulos 2006) explore the use of MMOG as a new generation of educational platform, allowing players to interact
with each other, and to learn and create knowledge together through collaborative game-play. However, the study for ex-
amining the empirical evidence of the occurrence of collaborative learning behavior in MMOG is still under-researched.
To explore this under-researched area, this study intends to extend our understanding of learning in MMOG game-
play, by looking at the dynamics on collaborative knowledge creation process throughout the progression of game-play
in a typical massively collaborative online game-play environment.
In particular, the rst objective of this study is to empirically examine the occurrence of knowledge creation in
MMOG game-play based on Nonakas (1994) four modes of knowledge conversion in his dynamic theory of organiza-
tion knowledge creation. Our second objective is to examine the dynamic relationship of the four modes of collaborative
knowledge creation processes with the game players engagement level and duration.
In the next section, we will present our theoretical review of the four modes of collaborative knowledge creation pro-
cesses in the MMOG game-based collaborative learning environment. Then we will discuss our empirical survey study
on the occurrence of collaborative knowledge creation in MMOG game-play. Lastly, the research potential of MMOG
game-based collaborative learning will be presented in the nal section.
COLLABORATIVE KNOWLEDGE CREATION PROCESSES IN MMOG GAME-PLAY
Explicit and Tacit Knowledge in MMOG
According to Polanyi (1966), human knowledge can be classied into explicit and tacit knowledge. Kong and Kwok
(2009b) proposed a denition of explicit and tacit knowledge in MMOG, in which the explicit knowledge in MMOG can
be dened as knowledge written in game specic language that is transmittable and sharable among game players, while
tacit knowledge in MMOG as knowledge embedded in game specic skills, action strategies and experiences that is dif-
cult to communicate and share amount game players using formal languages.
With the postulation that knowledge is created through conversion between explicit knowledge and tacit knowledge,
Nonaka (1994) suggests that there are four modes of knowledge conversion taken place in organizational knowledge
creation: (1) Internalization from explicit knowledge to tacit knowledge; (2) Externalization from tacit knowledge
to explicit knowledge; (3) Combination from explicit knowledge to explicit knowledge; and (4) Socialization from
tacit knowledge to tacit knowledge.
According to Nonakas theory, we suggest that MMOG game players learn collaboratively through social interac-
tion by helping their team members to learn, and creating and sharing their explicit and tacit knowledge with each other.
80 Kong and Kwok
Moreover, we postulate that the occurrence of knowledge creation in the MMOG game-based collaborative learning en-
vironment can be taken place in four modes of knowledge creation processes in the MMOG game-play based on the fol-
lowing qualitative ndings mentioned in Kong and Kwok (2009b).
FOUR MODES OF COLLABORATIVE KNOWLEDGE CREATION PROCESS IN MMOG GAME-PLAY
Externalization
Externalization refers to the knowledge creation process through converting tacit knowledge to explicit knowledge in
MMOG game-play. According to Kong and Kwok (2009b), experienced MMOG game players tend to externalize their
knowledge through writing their experience and skill of playing the game in community forums, and share the knowl-
edge to new players who seek for opportunities to cooperate and team up in the MMOG.
The sharing of the explicit knowledge can also be occurred through verbal conversation or in-game chat, and written
experience in blogs, which are accessible and communicable to other players.
Combination refers to the knowledge conversion from explicit knowledge to explicit knowledge in MMOG game-
play. According to Kong and Kwok (2009b), knowledge combination in MMOG game-play can be viewed from two dif-
ferent perspectives: private and public.
The private perspective of knowledge combination is related to the extrinsic motive of collaborative victory which
motivates players to share their knowledge to in-group members using a private mode of knowledge combination in form
of private blogs and forums. In the knowledge combination process, each player externalizes his/her explicit ideas and
suggestions related to the game task, and their suggestions will be combined and consolidated. Finally, a commonly ac-
cepted strategy will eventually come out.
The public perspective of knowledge combination is related to the intrinsic rewards and the fun part of the game-
play process, which drive players to share their experiences and ghting tips with others using their own blogs and reply
to discussions in public forums.
Internalization
Internalization refers to the knowledge conversion from explicit knowledge to tacit knowledge in MMOG game-
play. According to Kong and Kwok (2009b), learning usually starts with a tutorial session in the MMOG. In the tuto-
rial session, a player internalizes the explicit knowledge through reading the instructions and guides (explicit knowledge
of MMOG) developed by the MMOG developer, and practices on his/her own, with the purpose of consolidating and
mastering the game-play skills. More specically, the internalization process emphasizes that a player can learn through
reading explicit knowledge (e.g. written walkthroughs and guides in blogs and forums, listening to advices and sugges-
tions in in-game chats, etc.) using game specic languages. Through internalization, players transform the shared explicit
knowledge into their own game specic skills, action strategies and experiences (tacit knowledge of MMOG).
Socialization refers to the knowledge conversion from tacit knowledge to tacit knowledge in MMOG game-play.
According to Kong and Kwok (2009b), MMOG game players can learn through observing and imitating others, without
direct interaction or communication using formal languages. The socialization process in the MMOG environment can
be further classied into actively performing tacit knowledge to an observer (i.e. socialization by performing), or actively
observing tacit knowledge from a performer (i.e. socialization by observing).
In MMOG, players can see each other when travelling around the virtual space or ghting with other players during
the game-play. A player can observe and imitate how others are playing the game without verbal and written communica-
tion with each others. Also, when a MMOG player cannot gure out the strategy to ght against a challenging monster
or handle a difcult task even after an intensive search of explicit knowledge (e.g. written instructions, guides or experi-
enced players experience sharing in blogs and forums) online, the player may choose to construct their own knowledge
through observing and imitating other experienced players through socialization.
To summarize, learning in the MMOG game-play can be taken place in the four modes of knowledge creation pro-
cesses in the MMOG game-play. We have established a questionnaire to examine the occurrence of these four modes of
knowledge creation processes and their dynamic relationship with game players engagement level and duration, and
conducted a self-reported survey to collect and analyze quantitative data from MMOG players.
A survey design was used to investigate the aforementioned postulations. Survey questions were constructed with
reference to prior empirical work and modied to t the context of MMOG game-play activities. All questions were
phrased from the perspective of current World of Warcraft players. This study was set in the context of World of Warcraft
(WoW), an MMOG game-based virtual environment. The reason for choosing WoW is that it is currently the best sell-
ing MMOG in the world, with over 12 million subscribers worldwide. (Blizzard 2011) All questions were anchored on
7-point Likert scale from Strongly Disagree (1) to Strongly Agree (7).
Questions on the engagement in modes of collaborative knowledge creation processes were drafted based on Becer-
ra-Fernandez and Sabherwal (2001) and Chou and He (2004). These questions assessed socialization by performing
(SOP), socialization by observing (SOO), externalization (EXT), combination (COM) and internalization (INT) in the
context of WoW.
Questions on perceived addictiveness were drafted based on Young (1998). These questions assessed perceived ad-
dictiveness (ADD) in the context of WoW.
A total number of 26 questions were modied from the perspective of current WoW players based on Kong and
Kwok (2009b). Conceptual construct validation of these 26 items was carried out following Moore and Benbasats
(1991) card sorting procedure. Table 1 summarizes descriptive statistics of these items and their corresponding Cron-
bachs Alpha.
In addition, Engagement Level was measured by a self-reported question on the skill level of the most experienced
avatar that the player possesses currently. Engagement Duration was measured on the number of hours spent on playing
the World of Warcraft in the previous week.
This study utilized an online survey website to collect data from current players of WoW. Players of WoW were in-
vited to ll in the survey through online WoW communities, like game forums and corresponding network on Facebook.
To guarantee the quality of respondents, raw data were ltered based on the method used by Kong and Kwok (2009a):
World of Warcraft Experience
One yes-or-no question at the beginning of the questionnaire required the respondent to answer Have you ever
played World of Warcraft? All respondents who indicated without experience in WoW were ltered.
According to the design of WoW, players create their avatar by selecting race and class. Limited by the design, sev-
eral classes are exclusive to certain races. Based on these rules of combinations, two questions were included in the ques-
tionnaire, which asked the respondent to select the race and the class of their strongest avatar. The combination of race
and class were then checked with the above rules. Those who failed this check, or indicated others in the selection,
were dropped for analysis.
Table 1: Measurement of Items in the Self-Reported Survey
Construct Item Mean SD Alpha
When I played World of Warcraft last week,
Socialization by
(SOP)
... I demonstrated game skills (e.g. controlling a robot) to other players that
I was not able to explain using any language.
4.64 1.918 0.959
... I performed game ghting techniques (e.g. pulling a monster) to other
players that I found difcult to explain in words.
4.79 1.911
... I showed game movements (e,g. solving a quest) to other players that I
found challenging to express verbally.
... I displayed game actions (e.g. collecting quest items) to other players
that were not easy to communicate in text.
(Observing)
(SOO)
... I imitated game skills (e.g. controlling a robot) of other players without
talking to him/her.
... I observed game ghting techniques (e.g. pulling a monster) of other
players without chatting with him/her in words.
... I followed game movement (e.g. solving a quest) of other players without
communicating with him/her in text.
... I replicated game actions (e.g. collecting quest items) of other players
without speaking to him/her using language.
(EXT)
... I talked to other players about my knowledge of playing the game. 5.28 1.391 0.859
... I contributed my gameplay knowledge in the game channel with other
... I shared my gameplay experiences with other players in our guild forum. 4.82 1.713
... I share my gameplay experiences with other players through personal
communication/instant messaging (e.g. Windows Live Messenger).
... I shared my gameplay experiences with other players on game wiki (e.g.
WoWHead).
(COM)
... I merged several blog entries of other players gameplay experiences, to
share them with players in our guild/public forum.
... I combined several forum discussion threads of other players gameplay
knowledge, to share them with players in our guild/ public forum.
... I integrated several in-game discussions of other players gameplay
... I put together several other players gameplay experiences that I came
across, to share them with players in our guild/ public forum.
... I captured several parts of the user game manual, to share them with
players in our guild/ public forum.
(INT)
... I studied the gameplay knowledge of other players in the game channel. 5.20 1.438 0.760
... I read the gameplay experiences of other players written in our guild
... I learnt the gameplay experiences of other players from personal com-
munication/instant messaging (e.g. Windows Live Messenger).
... I examined the gameplay experiences of other players documented on
game wiki (e.g. WoWHead)
Perceived
... I played longer than I intended 5.15 1.593 0.819
... my job performance / academic performance suffered because of playing
... I acted annoyed if someone bothered me 4.60 1.793
... my friend / family complained about the amount of time I spent on the
Denition of Current Participants
One question was added in the questionnaire and the subject was required to disclose When was the last time you
played the game World of Warcraft? Only those responses which indicated within one day and longer than one day
and within one week were used for analysis.
Table 2: Demographic Information of Respondents
Total n 149
Gender Male 131
Age 10-15 14
> 40 5
Play Other MMOG Yes 75
The survey received a total number of 149 valid responses from current active players of the WoW after correspond-
ing data ltering procedure. Table 2 summarizes the demographic information of respondents:
The descriptive statistics, cronbachs alpha and t-test results were calculated in SPSS 17. PLS Graph 3.00 with 200
iterations of bootstrapping technique was used to estimate reliability and validity of items, as well as Partial Least Square
(PLS) path analyses.
Reliability and Validity
In PLS analysis, the Internal Composite Reliability (ICR) attempts to assess inter-item reliability, so as to ensure
internal consistency of indicators. Average Variance Extracted (AVE) attempts to measure the amount of variance that a
latent variable component captures from its indicators relative to the amount due to measurement error, so as to assess
the convergent validity of the constructs. The acceptable value of ICR for perceptual measure is above 0.70, where the
AVE should be higher than 0.50. (Chin, 1998) The following table summarizes the measurement model results with all
ICR and AVE values satised the acceptable value:
Table 3: Result of Conrmatory Factor Analysis
Construct No. of Item ICR
SOP 4 0.970 0.889
4 0.937 0.789
Satised criterion > 0.7
> 0.5
In PLS analysis, discriminant validity is determined by comparing the square root of AVE for each construct with
other involved constructs. The square root of AVE should be higher than the levels of correlations with another construct
(Chin 1998). The following table demonstrates square root of AVE (i.e. in grey color) for each construct which satised
the criterion.
Table 4: Correlation between Constructs
SOP SOO EXT COM INT ADD
SOP 0.943
SOO 0.746 0.888
EXT 0.332 0.319 0.805
COM 0.439 0.286 0.542 0.922
INT 0.383 0.236 0.433 0.414 0.763
ADD 0.175 0.190 0.430 0.406 0.242 0.806
OCCURRENCE OF COLLABORATIVE KNOWLEDGE CREATION PROCESSES
The rst objective of this study is to empirically examine the occurrence of collaborative knowledge creation pro-
cesses in MMOG game-play based on Nonakas dynamic theory on knowledge creation. Our analysis showed that, on
average, players gave scores of higher than 4 (i.e. mid-point of 7-point Likert scale) to all our survey items which mea-
sured four modes of collaborative knowledge creation processes, namely socialization (by performing), socialization (by
observing), externalization, combination and internalization (see table 1). In other words, our ndings conrmed the oc-
currence of the four modes of collaborative knowledge creation processes in MMOG game-play.
DYNAMIC RELATIONSHIP OF COLLABORATIVE KNOWLEDGE CREATION PROCESSES
Our second objective is to examine the dynamic relationship of four modes of collaborative knowledge creation
processes with game players engagement level, engagement duration and perceived addictiveness in terms of changes of
knowledge creation modes during the game-play process.
A series of t-test were used to determine the dynamic relationship of the same collaborative knowledge creation be-
havior at different engagement level. The mean value of a specic collaborative knowledge creation behavior was calcu-
lated by taking average of corresponding predictors. The probability values of two-tailed t-test, mean and sample size are
Socialization (by Performing)
Table 5: t-test results on socialization (by performing) at different engagement level
Engagement Level
1-19 20-29 30-39 40-49 50-59 60-69 70-79 80
(n=26) (n=15) (n=15) (n=10) (n=14) (n=15) (n=17) (n=37)
0.511 5.01 4.75
(not sig) (n=26) (n=123)
(sig) (n=56) (n=93)
(sig) (n=112) (n=37)
The two groups had signicant difference in terms of socialization (by performing) from level 40 onwards. By com-
paring the mean value, junior players (i.e. below 40) engaged in signicantly more socialization (by performing) as com-
pared with senior players (i.e. 40 or above). In addition, the mean value of socialization (by performing) drops below 4
(i.e. the mid-point of 7-point Likert scale) at level 80.
Socialization (by Observing)
Table 6: t-test results on socialization (by observing) at different engagement level
(not sig.) (n=26) (n=123)
(marginal) (n=56) (n=93)
(sig.) (n=78) (n=71)
(sig.) (n=112) (n=37)
The two groups had signicant difference in terms of socialization (by observing) from level 40 onwards. By com-
paring the mean value, junior players (i.e. below 40) engaged in signicantly more socialization (by observing) as com-
pared with senior players (i.e. 40 or above).
Table 7: t-test results on externalization at different engagement level
(not sig.) (n=56) (n=93)
(not sig.) (n=112) (n=37)
Our t-test analysis was not able to nd signicant difference between senior and junior players in terms of their en-
gagement in externalization.
Table 8: t-test results on combination at different engagement level
(sig.) (n=26) (n=123)
The two groups had signicant difference in terms of combination at all t-tests. By comparing the mean value, junior
players engaged in signicantly more combination as compared with senior players. In addition, the mean value of com-
bination drops below 4 (i.e. the mid-point of 7-point Likert scale) from players engagement level 40 and onwards.
Table 9: t-test results on internalization at different engagement level
gagement in internalization.
Engagement Duration
Table 10: t-test results on engagement duration at different engagement level
(Duration)
0.002 13.27 21.59
0.002 12.7 26.06
The two groups had signicant difference in terms of engagement duration at all engagement level. By comparing
the mean value, junior players showed signicantly lower duration (spending less time on playing MMOG) as compared
with senior players.
Perceived Addictiveness
Table 11: t-test results on perceived addictiveness at different engagement level
0.038 4.8 4.51
The two groups had signicant difference in terms of perceived addictiveness at all engagement level. By comparing
the mean value, junior players showed signicantly higher perceived addictiveness as compared with senior players.
To achieve a better understanding of the dynamic relationship between four modes of collaborative knowledge cre-
ation processes with players engagement level and perceived addictiveness, we performed and compared the following
two sets of Partial Least Square (PLS) path analysis for junior and senior players as follows:
Junior Players (Engagement Level Below 40)
(by Performing)
= 0.036
(by Observing)
-0.260**
*** p < 0.01, ** p < 0.05, * p < 0.1, n = 56
Figure 1: PLS Path analysis on junior players (engagement level below 40)
The PLS path analysis on junior players showed that there were signicant and positive correlations between en-
gagement level and socialization (by observing); socialization (by performing) and perceived addictiveness; and Exter-
nalization and perceived addictiveness. However, the PLS path analysis showed that there was a signicant but negative
correlation between socialization (by performing) and perceived addictiveness.
In other words, junior players engaged in more socialization (by observing) progressively from level 1 to 39, while
they were constantly engaging in other modes of knowledge creation at all engagement levels. In addition, junior play-
ers engaged with more socialization (by performing) perceived lower addictiveness. However, junior players engaged in
more externalization perceived higher addictiveness
Also, the R
value for Perceived Addictiveness is 40.9%, contributed by two signicant paths from socialization (by
performing) and externalization.
Senior Players (Engagement Level 40 or Above)
-0.306***
0.219*
Figure 2: PLS Path analysis on senior players (engagement level 40 or above)
The PLS path analysis on senior players showed that there was a signicant and positive correlation between combi-
nation and perceived addictiveness. However, the PLS path analysis showed that there were signicant but negative cor-
relations between engagement level and socialization (by performing); engagement level and combination.
In other words, the senior players engaged in less combination progressively from level 40 onwards leading to lower
perceived addictiveness. Also, the senior players engaged in less socialization (by performing) progressively from level
40 onwards while they were constantly engaging in other modes of knowledge creation at all engagement levels.
value for Perceived Addictiveness is 17.9%, contributed by the signicant path from combination.
To summarize, the second objective of this study is to examine the dynamic relationship of the four modes of collab-
orative knowledge creation processes. Our ndings showed a sharp turning point at game players engagement level 40
with a signicant change on some modes of knowledge creation process. For data analysis purposes, we encode players
with engagement level less than 40 as junior players, while with more than 40 as senior players.
More specically, our ndings indicated that junior players engaged in signicantly more socialization (by perform-
ing), socialization (by observing), and combination as compared with senior players. However, there was no signicant
difference between senior and junior players in terms of their engagement in externalization and internalization. Inter-
estingly, we found that the mean value of combination drops below 4 (i.e. the mid-point of 7-point Likert scale) from
players engagement level 40 and onwards. Also, our ndings indicated that junior players showed signicantly higher
perceived addictiveness, but lower duration (spending less time on playing MMOG) as compared with senior players. To
explain the above interesting ndings, our PLS analyses showed that junior players engaged in more externalization per-
ceived higher addictiveness, while the senior players engaged in less combination progressively from level 40 onwards
leading to lower perceived addictiveness.
LIMITATION AND PROPOSED FUTURE RESEARCH OF MMOG GAME-BASED COLLABORATIVE KNOWLEDGE CREATION
There are two major limitations of our study. The rst concern is related to the sample of the study. Our data was
collected from a number of MMOG players of WoW only. We suggest further research may conduct a boarder data col-
lection and compare players of different MMOGs for a more generalizable conclusion. The second concern arises from
the data collection method. Our data were collected from MMOG players using an online survey method only. The small
sample size and survey method may lead to sample bias. For future research, we suggest using a longitudinal multi-meth-
od approach, including interviews and focus group discussion, to collect qualitative and quantitative data at different time
schedule of data collection to further explain the change of knowledge creation modes during the MMOG game-play.
Also, we propose capturing data using the MMOGs chatlog command for further content analysis of the process and
outcome of the four modes of knowledge creation in the future research.
In addition, MMOG game addiction is an important aspect that we should not overlook. We suggest searching for
appropriate game-play strategies to maintain a balance between engagement of knowledge creation and addiction in
MMOG game-play.
IMPLICATION AND CONCLUSION
We are one of the pioneering studies to illustrate and empirically examine the dynamic model of collaborative
knowledge creation process in MMOG game-play. Our ndings support our postulation of the occurrence of four modes
of knowledge creation process (i.e. externalization, internalization, combination and socialization) in the MMOG game-
play. Also, our ndings show a sharp turning point at game players engagement level 40 with a signicant change
on some modes of knowledge creation processes, which have potentially high impacts on perceived addictiveness of
To conclude, the ndings presented in this paper provide theoretical and empirical contributions to the research of
MMOG collaborative knowledge creation in that they reveal the four modes of collaborative knowledge creation process-
es in MMOG game-play, and their dynamic relationship with game players engagement level, duration and perceived
addictiveness. We wish our ndings will provide directions for educators, researchers and game developers who seriously
explore using MMOG as a new generation of educational platform.
Becerra-Fernandez, I. and Sabherwal, R. (2001) Organizational knowledge management: A contingency perspective. Journal of
Management Information Systems, 18(1), 23 55
Blizzard (2011) Press Release: http://us.blizzard.com/en-us/company/press/pressreleases.html?110110, 11th January
Childress, M. D. and Braswell, R. (2006) Using massively multiplayer online role-playing games for online learning. Distance
Education, 27(2), 187 196
Chin, W. W. (1998) Issues and Opinions on structural equation modeling. MIS Quarterly, 22(1)
Chou, S. and He, M. (2004) Knowledge management: The distinctive roles of knowledge assets in facilitating knowledge cre-
ation. Journal of Information Science, 30(2), 146 164
Gee, J. P. (2004) Learning by design: Games as learning machines. Interactive Educational Multimedia, 8(Apr), 15 23
Kong, J. S. L. and Kwok, R. C. W. (2009a) Intention to Learn in MMOG: Examining the Roles of Peer Intrinsic and Extrinsic
Motivations. In proceedings of The 17th European Conference on Information Systems, pp 2533 - 2544, 8 10 June, Ve-
rona, Italy
Kong, J. S. L. and Kwok, R. C. W. (2009b) MMOG Game-based Collaborative Learning: An Exploratory Study and Its Re-
search Potential. In proceedings of Pacic Asia Conference on Information Systems 2009, pp 172, 10 12 July, Hyderabad,
Mikropoulos, T. A. (2006) Presence: a unique characteristic in educational virtual environments. Virtual Reality, 10, 197 206
Moore, G. and Benbasat, I. (1991) Development of an instrument to measure the perceptions of adoption and information tech-
nology innovation. Information Systems Research, 2(3), 192 222
Nonaka, I. (1994) A Dynamic Theory of Organizational Knowledge Creation. Organization Science, 5(1), 14 37
Polanyi, M. (1966) Personal Knowledge, Chicago, IL: The University of Chicago Press
Steinkuehler, C. A. (2004) Learning in massively multiplayer online games. In proceedings of the 6th international conference
on learning sciences, 521 528
Young, K. S. (1998) Internet Addiction: The Emergence of a New Clinical Disorder. CyberPsychology & Behavior, 1(3), 237
The Lost City: Development of a National-Level On-Line Mystery Game Using Freeware and Low Budget Technology 93
The Lost City: Development of a National-Level On-Line Mystery Game Using Freeware
and Low Budget Technology
TOM BENJAMIN
The Lost City was a scaled-down attempt to create an engaging multimedia experience using text, HTML forms,
freeware, audio/video hosting, and blogs in place of professionally-programmed interactivities. The departments Cen-
tre for Learning Innovation & Curriculum (CLIC) had since 1994 been running an online eco-murder mystery, Murder
under the Microscope (MuM) to increase long-term public awareness of Australias ecological concerns, aimed at stu-
dents aged 9-15 with teams from across Australia and some international participants. Nielsen (2008, 2011) reviewed
its unique national scale made possible in pre-internet days through use of existing resources such as studio facilities, a
national satellite education network with satellite dishes in rural school, as well as briengs aired on a national public TV
station. Interaction was initially via telephone and fax. By 2009 the developers were able to draw on web 2.0 technolo-
gies to develop online learning communities, extend the focus beyond Australia to the Pacic-Asian area, increase inter-
active features of the website, and incorporate user-generated content into the website.
Drawing on the success of Mum, CLIC prepared a collection of Flash-powered mystery games in collaboration with
a Singapore-based commercial game company for launch in 2011 as Mystery Matters (MMs) which covers a range of
KLA for middle years students. As with MuM, the MMs used live actors for voice-overs and professional artwork. Vid-
eos and other resources are linked to the game sequence. Teachers notes are provided to help teachers integrate MMs in
each subject area.
The 1980s had seen the rise of murder mystery games, drawing on an earlier tradition of board and parlor games.
MuM and MMs incorporated many of these elements, with a storyline introducing a murder context and students re-
quired to identify a correct permutation of crime scene, victim, and villain. MuM preceded but could be well described
as an alternate reality game with its lengthy ofine collaborative problem-solving activities, hidden clues and links to dif-
ferent locations in the physical world. The MuM game is played over a six week period. Four months beforehand teach-
ers are emailed alerts and classroom resources to download which allow them to plan how to integrate MuM with their
particular class and begin the game with some familiarity. A help line is active throughout the game with ready advice
and information for teachers. Emails and wikis now supplement phone support. What were originally weekly broadcasts
now via the internet became a series of over 100 sequenced multimedia clues. Over the 6 weeks of the game actors-as-
scientists reveal the clues embedded in a professionally-scripted irtatiously irreverent lab banter familiar from the foren-
sic crime TV genre. Real scientists are brought online to answer questions. As many as 40,000 students play at a time,
generating sometimes millions of hits per day. Teachers comment especially on the capacity of the format to engage the
less forthcoming students. The winners of each category and part of the game receive awards as Eco Planners.
The Lost City was conceived as a similar mystery challenge. Lost cities are a staple of TV documentaries and the
archaeology involves a full range of sciences from school curricula. Most important, however, was its potential to incor-
porate some of the mathematics concepts such as number sequences that can be hard to associate with the real world.
These can be made into the codes and secret messages common to lost city movies. The successful migration of MuM
to Web 2.0 technology raised the possibility that a scalable online challenge might be mounted with little or no budget
by an individual teacher and the Lost City topic seemed ideal as it was open to interpretation by teachers of any level or
subject. The prologue for the quest was developed as follows: Quite commonly we read that archaeologists discover a
lost city. Sometimes these are famous from historical records, such as Troy, but the exact location and ruins had been
lost. Others were abandoned so long ago that the newer inhabitants of the land never even knew of their existence. The
explorers usually had to convince the people back home of the importance of the nd and the need for further trips. Often
they brought back specimens, treasures, and even people to show off the city. This challenge will be presented in a series
of clues as though from the log book of an explorer/treasure hunter who was pursuing a legend. You then present the city
to todays world.
94 Benjamin
It was decided to approach the Tournament of Minds (T.O.M.) organization as they hosted an existing Australian on-
line problem-solving competition. Nearly 20 per cent of NSW public schools compete in the tournament and it is being
expanded internationally. It already had an extensive judging protocol. They agreed to try the Lost City as their 2010 Ap-
plied Technology Challenge. T.O.M. teams pursue quests online then create movies, plays and prototypes to portray their
quest in live performances to the judges in a series of regional and national nals. The Tournament, like other national-
scale live events such as MuM, Connected Learning Awards, and the Schools Spectacular, incur administrative expenses
by hosting the nals around the nation and awarding prizes. The Tournament provides air fares and accommodation for
judges. Participants travel interstate to attend. Many volunteers and commercial donations are required to sustain such
The Lost City was intended as a prototype for custom classroom or school-level quests that could eliminate many of
the big expenses, as the judging and prizes would just be part of the normal academic roles of the teachers. Once classes
became familiar with the toolkit, mysteries could be created, customized and shared by teachers and/or students. Thus it
was a major goal of this project to use a follow-up questionnaire to gauge the extent to which the multimedia, prizes and
competition elements contributed to the engagement of the Lost City compared to the lower-budget elements including
the intrinsic interest of the subject matter.
The hope was to sustain suspense and difculty levels sufcient to engage and seem fair to both primary and second-
ary students by doling clues out in a sequence. The teams able to capitalize on early clues gained a weeks head start
in the four week quest. The earliest clues were the hardest, using carbon-date data, cryptic references to star positions
to establish location, and obscure historical references not easy to nd with simple internet quote searches. The second
round of clues included comical and entertaining clues and, importantly, a crossword puzzle. The crossword for many
became a give-away. Care was taken to pilot-test such clues to get an idea of difculty level. Final week clues were cer-
tainties. Even if slower teams had bypassed the hardest clues at rst those dead ends could still be used productively as
part of their movie presentations, which were judged on the criteria of portraying their quest and bringing the people of
the lost city back to life. For example, a team might have not understood the carbon-dating or star clues. This would not
stop them from going back over these and incorporating them into their presentations. Dead-end investigations are a nor-
mal part of science and can make interesting documentary. Since there were no penalties except for rule violations teams
were free to pretend in drama that it was these difcult clues that had led them to the lost city. These are students in the
role of journalists/adventurers rather than professional archaeologists. The goal is to gain their interest in the difcult
clues even if they need benet of hindsight to understand them.
The On-Line Challenge
In the vein set by Christopher Columbus and others in the Age of Exploration, teams were called on to locate a par-
ticular lost city on the basis of clues given out then recreate it for an audience of today, bringing it to life again through
their presentation: Each clue will narrow it down and by the 3rd to 4th clue should be obvious. Those who narrow the
eld quickest through a process of elimination are rewarded by having more time to prepare their presentation. Your pre-
sentation will be assessed on the logic you followed in pursuing the clues, how quickly you found it, and how well you
can bring it to life. The clues will cover Astronomy, Mathematics, History, Environmental Science, Culture and Geogra-
phy.
Teams were given an online form including a map on which to nominate each week what they at that stage believed
to be the lost city, specifying its longitude and latitude. Based on that assumption they were to prepare a combination
stage-play, talk, and/or documentary lm of at least 5 minutes duration, 1-2 minutes showing their discovery logic and
the rest being a dramatization of how the lost city may have looked and how its people lived, bringing the lost people to
life through re-enactments of their dress and customs, appearance of the land and its animals. Applied technology was
largely in the techniques used to create their documentary, such as sound effects, music, dialogue, character and setting.
The Tournament rules are strict as to the specic technology list for solutions, being mostly items deemed available from
their school or library, such as Computer/notebook/laptop, Internet, planisphere, star atlas, spreadsheet or calculator, data
projector, props: toys, costumes, blue screens, software, Microphone, camera etc. They also had to submit log forms,
presentation script, cost forms, outside assistance form and presentation skills forms, and a CD/DVD of the documentary
lm, play or talk.
A manual detailed all of this and a used the Tournaments existing formal scoring system based on accuracy of iden-
tication of the Lost City, originality of the explanation of how and at what stage team came to identify the Lost City,
creativity and procient use of movie making techniques in bringing to life the people and environs of the Lost City, ef-
fective logical use of the use of visual and sound effects, music, dialogue, character and setting in the documentary, and
use of at least three different items of present day technology. They nominated two presentation skills, explaining their
choice of dramatic technique and how this technique enhances the communication of the solution to the audience. Judges
also allocated points for the overall dramatic performance of the team, dened as the extent to which the performance
had excitement, air, creativity, basic stagecraft skills, and impact related to the challenge solution.
Weekly progress reports were optional. They were scored after the main decisions had been made and were largely a
check to ensure that there were not anomalies suggestive of cheating or other inuences -ie- a team miraculously coming
up with a correct answer having either been clearly wrong on early reports or having failed to lodge any progress reports.
In creativity contests, such considerations are less important as there is no clear answer that can be shared. The Lost City
did, however, have a denite location and early identication gave distinct advantages.
This was not an immersive, interactive experience. As Mori (1970) observed, there are limits to the impact of
immersive media on the player avatars can be creepy rather than engaging. The Lost City offered a few movie vignet-
tes but most of the introduction was text with links to a few pictures. It was static HTML with still images. The action
took place ofine as a quest, looking up clues and preparing a presentation. One of the key clues was a crossword puzzle
generated with Eclipse software. Traditional clues of this nature, such as hangman, codes, and puzzles, are easy to gene-
rate, require no programming skill, and have extremely low bandwidth requirement. These are traditional methods and it
was predicted they would still have appeal.
New South Wales participants were given a questionnaire at the competition after each team completed its presenta-
tion. These were anonymous.
Table 1: A total of 114 individual students and 21 teachers responded. Student responses were as follows with the aver-
age rating expressed as distance along the scale (for example on a three-point scale, 1.5/3 = .5):
-Would you like to study more projects like Lost City for other subjects? 87% Yes
-Did the game help you with any of your school subjects? 61% No
-Did you enjoy playing the Lost City? (scale 1-3 very much.. a little.. not much) 2.69/3 = .90
-How important are the rewards and prizes? (scale 1-3 very much.. a little.. not much) 1.85/3 = .62
-How much did you learn from playing Lost City? (a lot..quite a bit..a little..nothing new) 3.11/4 = .78
-What did you enjoy most or least about Lost City: (enjoyed the most..enjoyed.. not try..did not enjoy)
Solving the mystery 3.63/4 = .91
Competing with others 3.00/4 = .75
Getting prizes 2.85/4 = .71
Primary and high school students were judged together but awarded separately. There was the expected developmen-
tal improvement with age suggesting cognitive and emotional maturity to be important determinants but considerable
overlap between the primary and secondary teams. These results combined with the qualitative responses suggest that
the intrinsic interest of the subject matter was a key motivator. Students felt that they learned quite a bit even though it
did not help them particularly with their school subjects. Nor were they merely in it for the prizes and glory. It is often
seen with MuM that teams who dont end up submitting entries to the competition still rate the experience highly and re-
enroll from those schools. Students described a full range of skills and benets from the Lost City. The camaraderie from
team participation was mentioned as was a whole range of cognitive skills.
These results are comparable to survey results of previous online games. This was quite predictable from Clark &
Sugrue (1991) whose meta-analysis showed that most innovative approaches show an initial benet. It could have been
assumed from this before even bothering with a survey that there would be a positive rating. The focus here was the cost
denominator of benet/cost as proposed by Clark (1994). The cost element had already demonstrated that the current
crop of free and budget tools can create comparable materials to those in more programming-intensive larger-budget im-
mersive tools. If there was any immersion in this it had to have been the traditional one of an interesting subject matter,
playing with the crossword and less-serious clue toys, and enjoying the camaraderie of stage-plays and movie-making.
Results were encouraging. Winners and losers all rated their enjoyment highly and comparable to other national chal-
lenges hosted in our state. The competition and prizes were said to be less important than factors such as collaboration
within teams and the intrinsic interest of lost cities as a topic involving sciences, mathematics, geography and history.
It would be easy for a game evangelist to overlook the important nding that the majority reported that participa-
tion in Lost City did not help them with their school subjects. Indeed, some even penned detracted in their responses.
There is an emphasis in the Tournament on creativity and performing arts. It was worth noting that perhaps the majority
of teams took this as a license to depart from the strongly-suggested documentary format to instead opt for pop culture
presentations. At all levels, primary and secondary, high scoring and low scoring teams we saw performances that includ-
ed portrayals of Lady Gaga, Elvis, Master Chef, the Great Race, and Supermodel to name but a few. The materials had
all been put in place for them to do a typical National Geographic or Attenborough style of documentary yet few took
this up. For example, some had crafted nice traditional school-project table-top models of the Lost City. Yet after all that
work only one such team thought to lm this as part of their presentation. This one had been done so casually that the
teachers and their staff room teacups in the background had not been cropped from the movie, yet such students routinely
use other sophisticated special effects such as blue screen.
These ndings suggest caution before making any plans to implement such a game on an assumption that it will
improve any particular cognitive or social skill, much less make any contribution to specic curricula. They also open
some debate about the nature of open-ended creativity. Some might question whether the aping of a current generation
of pop culture is a demonstrably valid alternative to earlier forms of expression like the school play and orchestra. Cer-
tainly, earlier generations would have likely incorporated the Disney offerings of their own day (such as Davy Crockett)
into their work where allowed. While some may see any open-ended expression of childrens creativity as benecial,
one answer is that movie-making in the classroom has been studied since at least the 30s ie- The claims made by Katz
(1936) for the pedagogical benets of silent lms read much as they do for digital movies today. He mainly contrasted
movie-making with the passive involvement of the class trip. A similar case could be made for drawing or still-lming
a class trip. Silent lm-making did not revolutionize the classroom in the intervening years. No such claim should be
made today without control-group evidence.
The issue is whether game formats can be harnessed to teaching anything specic or are merely a non-specic for-
mat like the old class trip. This was documented with judges feedback forms given to schools clearly showing the areas
the judges thought to be outstanding or in need of improvement. The lowest feedback scores were often for not focusing
on the Lost City topic itself ie- not showing the audience how their team came to identify it, portraying nothing about
its people, nor showing any impact of the discovery of the Lost City to 21st century life. Often it was just a miscellany
of pop culture. There is an objective test to this ie- would a non-involved audience member learn anything about the
Lost City from watching this student presentation? If the answer was no even to the point of being seriously misled,
the concept needs to be questioned. Take for example, a portrayal that this Lost City had been discovered by Livingston
followed by a battle. This particular site had in fact been settled very peacefully by the French. To excuse such mis-
education on the basis of the supposed creativity benets is very bad science as the latter might have been achieved at
even lower cost by merely having a paint what you like or pirate day at school and not bothering to craft an immersive
quest.
This may have been an artistic choice as teams chose to emphasize verse and dance but teams that made sure to
include all content areas were rewarded for doing so. Teams that went on to the nals excelled in all categories, used the
technique to tell the story, and left no glaring omissions, providing both presentation and content, rather than trading one
at the expense of the other.
The few negative comments overall may have reected a perceived change from the more open-ended creative arts
approach ie- make a movie to a more science-documentary approach ie- make a movie about X . Rather than dilute
the scientic content of such a quest in favor of more artistic open-slather there is a case for leaving free-form lm-mak-
ing to the many other existing competitions for young lm-makers, not to mention YouTube.
The Lost City task could be best described in the terms of Mayer (2001) as guided discovery rather than pure dis-
covery. If teachers chose to offer such a challenge there is much discretion as to how the curriculum could be tied in and
how much time would be allowed for different forms and media for student inquiry. Background scientic knowledge
could be gained from the Web, books, lectures, or through question sessions. Solutions need teamwork due to the sheer
number of permutations, with dozens of potential leads to follow up. There is an inbuilt chance element with the Web
allowing countless links between clues, allowing some uke discoveries. The general difculty level of Lost City was
pitched high enough that only a few teams benetted from the earliest and most complex clues but none missed the target
altogether.
This is in line with the ndings of Christenfeld (1996) that the most popular sport games have just over half of their
variance accounted for by ability with the rest left unpredictable. It is in line with Skinners ndings that intermittent
reinforcement leads to more persistence than continuous, linear feedback. For example, despite only a 1 in 800 chance
of a team winning and that only half even complete the full exercise, enrolments for MuM have tended to draw repeat
school teams and to increase each year. This would indicate that game play becomes or is perceived as in itself engaging,
independent of the external rewards. The elaborate reward ceremonials may contribute more to the overall celebrity cult
atmosphere as a TV event than as motivators in the traditional sense of academic or sporting competitive prizes.
Tasks can be converted to game versions by adding elements such as chance and competition (Benjamin, 1980a,
1980b). For example, Scrabble transformed a vocabulary task into a popular game. Some imprecision in assessment
is added by the chance elements but this can be offset by sufcient repetitions of the task (Benjamin, 1981). The Lost
city was not was tied to the state syllabus in its design so the issue of assessment did not arise. However, if the format
imposed any additional unwelcome burden on teachers or students this would likely soon show up as complaints. Edu-
cational innovations can be expected to work unless there are some fatal aws (Russell, 1999, Nichol & Watson, 2003).
Flaws are most likely to be tangential relationship to curriculum and excessive extra time requirements (Kirriemuir &
MacFarlane, 2004). Most reported anxiety in Lost City was related to the change away from open-ended creativity. But
such anxiety need not be tied to curricula. Any teachers designing such a game themselves would have control over intru-
siveness as they would be the game-masters.
Older children did not report engagement as due to this online game as trendy, novel, or in line with their supposed
digital orientation. This might be explained in a number of ways. Cognitive Dissonance theories would see this as an
adaptation to explain their actions. Simpler explanations would be that the game was fashionably retro, inherently a bit
of fun and light relief, and/or that such mysteries were in themselves absorbing. The latter seems highly likely because of
the enduring interest in lost cities and its regular television appearance.
The younger end of the spectrum also tells a lot. After the MuM game, the younger students invariably asked for
gossip about the characters where theyre working, whether the male and female leads hooked up romantically the
usual sort of gossip associated with TV characters. This shows them to be operating at a normal level of social develop-
ment for their age - not being able to or at least not caring to distinguish between actors and their ctional characters.
This contrasts with and highlights the adult level of cognition these same young children are able to muster for the Lost
City and Mum games. This bodes well for this game format as a way of engaging high levels of participation in complex
socio-scientic debate, a particularly urgent consideration worldwide. Primary school children of mixed background,
not merely the gifted, have proven themselves capable of understanding and even investigating some of the subtle issues
relevant to the climate change and related ecological debates in MuM and the social issues of indigenous peoples in the
Lost City.
Learning design is seen as the key to the robustness of the murder mystery format across different delivery modali-
ties. Teachers must also be kept enthused and on side. Outside the classroom the murder mystery party and game for-
mats have continued in popularity into the current century. However, there was always potential for the novelty of the
MuM broadcast format to wear off. This has not happened. There are of course many different players each season, but
each new MuM group has responded with similar enthusiasm.
Internet delivery has improved administration. Other organizations wishing to implement a similar game might be
tempted to harness the resources of the internet, substituting avatars and digital sets. One way to maintain the MuM-level
quality of attachment to celebrity characters would be to use famous public domain characters such as Captain Nemo and
Sherlock Holmes. These can now be brought to life using software such as CrazyTalk without the need for advanced
computer programming skills. It is important to consider such effects in trying to tease out the active ingredients that
could make MuM or Lost City a role model.
Corporate and adult party versions of murder mysteries emphasize the stepping out of customary roles. Participants
dress up in costumes. School students need little encouragement to do this. The stepping out of the student role is nearly
the reverse they step out by being allowed to genuinely participate in important real world activities. They take on a
role familiar to them from the TV explorer and forensic dramas. Although there is not much remaining novelty in com-
puter delivery, the TV celebrity aura has so far transferred well to the small internet screen. The professionally-written
plot and dialogue help to bond them with the characters in MuM but a similar effect may have been achieved within the
Lost City with mere public domain text, such as the autobiography of a native. Ancient writings have by denition stood
the test of time and can be used for both their literary quality and their mystery.
Teacher support is a critical factor in such innovation. Their enthusiasm would be much more certain if the material
itself did the double duty of engaging students through a novel medium and directly addressing their curriculum. That
is why this project hoped to set a standard that would be achievable for a custom game by an individual teacher without
special funding, programming skills, or outside help. Web technology assisted all aspects of the project. Things were
done in days that would have taken weeks.
Less can be more in that students may work harder given fewer clues. Collaboration can be forced upon teams by
the sheer number of their tasks. Participants have to share facts to make any sense of the early clues. There is a team at-
mosphere of racing against the rest of the nation or the world. The clues had to be strategically doled out in a sequence
to give enough to build excitement while not ruining the game by giving too much away. As with radio station program-
ming, this is much easier to keep on top of with a database-automated system. These systems make the game more scal-
able and allow the games-masters to maximize variety and engagement. Web 2.0 provides such tools for both administra-
tion and collaboration. An individual teacher could do the whole thing via paper & pencil or email but if other schools
participated in a larger competition the computer version makes the game scalable.
It can easily be seen how even an international competition could be staged using currently-free resources such as
YouTube, Facebook and social networks. These sites analyze popularity in terms of number of visits. This could readily
be incorporated into a scoring system such that participants in an international game uploaded their presentations to a
social media hosting site. The hit rate could create a short list for judges. The big expense of judging can often be bring-
ing them together either physically or virtually to ensure they view the same entries. A social media site could serve as
the otherwise-expensive rst round of the contest by allowing votes to create the short lists. In essence, the anonymous
folksonomy viewers of these sites serve as rst-round judges. This relieves the contest judges from viewing hundreds
or thousands of entries as they need view only those that have passed a rst round which would consist of objective cri-
teria such as time of submission, format, length, copyright clearance, and number of hits generated. These dont ensure
quality but they eliminate those that would have been deemed ineligible without wasting judges time in administrative
considerations that can be automated. While hit rate popularity can be manipulated and scammed, it is a fair requirement
for a rst round. If all entrants are limited to common identiers in the code that describe their entries (eg. school music
competition entrant #302) they may well all benet from the hits of spammers in that the search ranking of the overall
competition may rise, ensuring that those unable to promote their entry draw some views as well.
Folksonomy ratings and hits are some objective and simple scoring systems for the rst stage of a Web 2.0 con-
test. Another objective and simple scoring system to administer is the race. The only parameter in a typical race is time.
If contestants start at different times, as would be the case in an international competition spanning different time zones,
this can be calculated as the time elapsed between their receiving their clue and their submission of a correct response.
The main complication with such a system can be collaboration such that a participant in an earlier time zone transmits a
clue to someone in a later zone, giving them an articial edge in the race. But the objective popularity and race elements
merely qualify or disqualify participants to be judged in the next phase of the overall contest. In the Lost City, a race ele-
ment was scored in terms of proximity of their responses to the information available from the clues -ie- the distance by
air of the Lost City location was used as the point system for accuracy of identication of the location of the lost city. A
simple additive-subtractive scoring system was also used for number of identications of age of ruins, technology of the
people, type of buildings etc that were discernable from the clues provided.
Thus is there now potential to host an international competition from a desktop.
Arts Unit (2010), Schools Spectacular, Specialist Programs, Schools, NSW Department of Education and Training. Sydney.
Benjamin, T. (1980a) The Use of Games in Assessment, Proceedings of the Fifth Annual Brain Impairment Conference.
Benjamin, T. (1980b) Games as Behavioural Measures. Proceedings of The Conference of the Australian Behaviour Modica-
tion Association,.
Benjamin, T. Games: (1981) Concepts. Proceedings of the Sixth Annual Brain Impairment, Conference,.
Christenfeld, Nicholas (1996) What makes a good sport? Nature V. 383 24 Oct 96 p662.
Clark, Richard E. (1994). Media will Never Inuence Learning. Educational Technology Research and Development, 42(2), 21-29.
Clark,R.E. & Sugrue,B.M. (1991) Research on instructional media, 1978-1988 in G.J.Anglin (ed.) Instructional technology:
past, present, and future ch.30 Libraries unlimited: Englewood, Colorado pp.327-343.
Katz, E. Making movies in the Classroom, The Clearing House, XI (November, 1936), p. 156.
Kirriemuir, J., & MacFarlane, A. (2004). Literature review in games and learning (No. 8). Bristol: NESTA Futurelab.
Mayer, R. E. (2001). Multimedia Learning, Cambridge University Press. New York.
Mori, Masahiro (1970). Bukimi no tani (The Uncanny valley). Energy, 7, 3335. (Japanese)
Nichol, J. and Watson, K. (2003). Editorial. special issue:The Rhetoric and reality: the present and future of ICT in education.
British Journal of Educational Technology 34, 2, UK.
Nielsen, C. (2008). Murder under the Microscope. In Proceedings of World Conference on Educational Multimedia, Hyperme-
dia and Telecommunications 2008 (pp. 6357-6360). Chesapeake, VA: AACE.
Nielsen, C. (2011). Murder under the Microscope - a model for best practice. [32278] In AACE Global Learn Asia Pacic
2011 Melbourne
Russell, T. L. (1999). The no signicant difference phenomenon. Ofce of Instructional Telecommunications, North Carolina
State University,
Virvou, M., Katsionis, G., & Manos, K. (2005). Combining Software Games with Education: Evaluation of its Educational Ef-
fectiveness. Educational Technology & Society, 8 (2), 54-65.
Play Games to Learn: Pre-service Teacher Development 101
Play Games to Learn: Pre-service Teacher Development
ALAN AMORY
[C]omputer and video games have the potential to prepare students to learn to use the techniques of communities
of innovation. Epistemic games emphasize ways of learning that stress immersion in practice, supported by structures in
knowledge. As such, they lead to expertise, professional-like skills, and innovative thinking. (Salmani Nodoushan, 2009,
Salmani Nodoushan (2009) also suggested that epistemic (knowledge) games, as proposed by Shaffer and Gee
(2005), are about the facts of life and have the potential to train children in specic ways. However, Kellner and Share
(2005) posited that all media, which include computer video games, are constructed using signs and symbols that are not
necessarily windows of the world and that are decoded by individuals in many ways - media are embedded with specic
values and are always related to power and/or prot. Furthermore, Torres and Mercado (2006) stressed that mass media is
not objective, politically neutral or representative of a balanced position. Therefore, Torres and Mercado argued that me-
dia is primarily used to reproduce and maintain dominant cultural values. Mitchell (2008), in discussing McLuhans the
medium is the message, suggested that contemporary media theory is driven by an obsession with the war machines;
technological innovations are concerned with coercion, aggression, surveillance and propaganda; and that we need to ask
who is behind the media. Gibson (2008) argued that what one does with the media determines its meaning or message.
Therefore, the facts that are part of any epistemic game may represent only one of many ways of knowing and without a
critical appraisal may train children to advance specic ideological positions. Thus, hegemonic practices are reproduced.
While Torres and Mercado (2006) and Kellner and Share (2005) proposed that critical media literacy needs to be
part of teacher education, it would be useful to provide teachers with appropriate thinking and practical tools to support
the use of media, including computer video games, in the classroom. Therefore, the aim of this paper is to explore how
pre-service teachers understand and use computer video games in the classroom to develop a theoretical framework for
learning with video games. Furthermore, this papers supports the position taken by Amory (2010) who argued that the
most appropriate theoretical framework to support the use of educational computer video games in the classroom are
constructivist learning theories, as articulated variously by, among others Vygotsky (1933/1978) and Piaget (1977). More
specically, the contemporary theoretical descendant of Vygotskys work, namely Cultural Historical Activity Theory
(CHAT) can be used both as an analytical frame to design learning tasks that include video games and as a means to un-
derstand tool-mediated knowledge construction through game-play.
CULTURAL HISTORICAL ACTIVITY THEORY
CHAT originated from the earlier work of Vygotsky (1933/1978) and Leontev (1978) and is described as the rst
generation activity system by Engestrm (2001) who wrote that in such a system stimuli and responses are transacted
through cultured mediated acts. Engestrm (2001) further outlined that Leontev introduced the idea of individual and
collective activity to overcome the individualistic focus of the rst generation activity system and thereafter expanded the
system to include components to support social interactions (second generation Fig. 1). Through the interlocking of a
number of activity systems (third generation) Engestrm (2001) extended the system to support cultural diversity.
102 Amory
Actor Object
Community Division of Labor
Figure 1. Activity system diagram (redrawn from Engestrm, 1987).
In any activity system Outcomes result from Actors interrogating Objects by means of Tools (physical pencils
and technological artifacts; or psychological signs and symbols). Tools mediate interactions through the activity
context that includes a Community, Division of Labor and associated Rules (Engestrm, 2000, 2001; Barab, Evans &
Baek, 2004; Roth & Lee, 2007). Objects, as cultural entities, are the prime unit of analysis within an activity system
(Engestrm, 2001), embody communal social practices that transform and further develop during human activity (Stet-
senko, 2005), and, in conjunction with motive, give the system coherence (Engestrm, 2000). Socially created Tools are
inseparable from the associated activity and are part of the purpose, relevance and value appropriated to them by the Ac-
tor (Robbins, 2005), and may become Objects, or Outcomes, of activity (Roth & Lee, 2007). The object of the activity
and social tool-mediation are explored in the next two sections.
THE OBJECT OF THE ACTIVITY
While the object of activity is key to activity theory, Nardi (1996) and Kaptelinin (2005) argued that there are often
different meaning associated with the term object. First, Kaptelinin (2005) explained that the Russian words objekt (ma-
terial things existing independently of the mind, p 6) and predmet (target or content of a thought or an action, p 6)
both translate to object in English. Kaptelinin argued that the object of activity refers to predmet that is more subjective
and, with respect to, the subject-object interaction the object refers to the objekt, which is more objective. Nardi (2005)
stated that the rst meaning is related to that which is to be realized (p 39) and the second could be seen as the object
of desire (p 40). Second, for Leontev the concept of the object of activity is different to that of Engestrm; for Leontev
the object of activity is acted upon predominantly by individuals and activities are individual or collective related to mo-
tivation, while for Engestrm the object of the activity is always production (Kaptelinin, 2005). Both the positions taken
by Leontiev and Engestrm are used in this paper to explore the use of video games in learning and teaching. Lastly,
there is often confusion regarding the object and the motive associated with the activity. If the Object and Motive are
separated (Kaptelinin, 2005), then when we instantiate an object we formulate it, and realize an object when we reach as
outcome (Nardi, 2005).
SOCIAL TOOL-MEDIATION
Mediation is one of the central themes integral to Vygotskys concepts of learning and development. Tool-mediation
is grounded within sociocultural practices (Doehler, 2002), supports human development and freedom to improve the hu-
man condition (Stetsenko, 2004), and learning is always mediated by human psychological and cultural tools (Levykh,
2008). For Shaffer and Clinton (2006), tools and thoughts have the same ontological status where neither could exist
without the other. Wertsch (2007) explained how Vygotsky used mediation in a variety of ways and categorized Vy-
gotskys formulation of mediation as either extrinsic or intrinsic. Extrinsic mediation is when Vygotsky spoke in the
idiom of psychology, especially about what we would today view as a form of behaviorism, or perhaps cognitivism and
it is when an individual overtly and intentionally introduces a stimulus means into and ongoing stream of activity (p
180) that is obvious and non-transitory. Implicit mediation, on the other hand, is less obvious and difcult to detect but
it involves signs, especially naturally language, whose primary function is communication (p 181) and does not read-
ily become the object of consciousness or reection (p 185). Whether mediation operates through direct intervention or
through language and signs, individual transformation by necessity includes both extrinsic and intrinsic modes of media-
tion. In addition, Edwards (2008) argued that mediation is hierarchical where the tools range from simple and material
to the sophisticated (for example, technological systems and ideologies) and that tools support humans to master their
world and thereby transform themselves. However, mediation not only refers to the nature of what goes on between
people but also to the process of co-creation between the social world and the internal world of idea, feelings, and
personal development (Edwards, 2008, 174). Therefore, mediation is an integral part of social transformation. For Ed-
wards (2008) mediation includes two aspects: it explains how the social becomes internalized within the personal and
describes how these internalizations are related to the developmental dynamics of human consciousness. All social
mediation is a cultural processes passed on from one generation to another (p 175).
Therefore, the outcome on the object of an activity is socially constructed through tool-mediated activity that can
only be situated with an individuals culture. These tenants of social constructivism formulated within an activity system
are now used to explore the use of computer video games in teaching and learning.
EDUCATION VIDEO GAME ANALYSIS
The use of video games in the classroom could be viewed either as a material object that exists independently of the
mind (objekt), or as a target of an action or thought (predmet).
In a recent meta-analysis of computer games as learning tools Ke (2008) found that much of the empirical evidence
provided little support that playing games lead to learning -- most of the research analyzed was anecdotal, descriptive,
or judgmental. However, a large proportion of the articles (73%) analyzed by Ke (2008) compared conventional instruc-
tional methods with stand-alone pedagogical instruments or drill-and-practice (trivial) games. Similarly, research using
video games as educational tools in the teaching of medicine showed little evidence for the effectiveness of these games
in learning (Akl, Gunukula, Mustafa, Wilson, Symons & Moheet, 2010). Again, games used in these studies could be
dened as trivial and were played individually by students. There are a number of problems associated with these studies.
First, the use of empirical testing methodologies (i.e. comparing students who played video games with those that did
not) rather than other more appropriate approaches such as design experiments (see Amiel and Reeves, 2008). Second,
the games were used as a material entity independent of the mind (objekt). Third, video games consumption supports the
reproduction and maintenance of dominant cultural values, as suggested by Torres and Mercado (2006). It is therefore
not surprising that when video games function as the objekt of a learning exercise and function primarily as a delivery
mechanism (Jonassen & Reeves, 1996; Schrader & McCreert, 2008), students develop visual intelligence (Greeneld,
2009) and ludic skills (Oliver & Carr, 2009) rather than developing critical thinking, imagination and reective skills as-
sociated with a curriculum.
However, there are examples where the use of video games in the classroom resulted in meaningful learning that
can be associated with the curriculum. Game play led to learning when the students were part of a game development
design team (Waraich & Brna, 2008) or when they created their own games (Robertson & Howells, 2008). In the latter
report, participants were enthusiastic, determined to complete the tasks, worked both individually and collectively, and
could apply what they learnt to other situations. In addition, Rieber, Davis, Matzko and Grant (2009) found that when
designing their own games the design elements of narrative, competition, and challenge relate in some way to the so-
cial dynamics of the daily relationships of the children participants. Verenikina, Herrington, Peterson and Mantei (2008)
showed that group play supported imaginative make-believe as an important learning strategy used by young children.
Seagram and Amory (2006) reported that groups of players who discussed the game puzzles develop a deep understand-
ing of the embedded concepts the longer the participants discussed certain knowledge domains, the richer were their
descriptions. Similarly, the use of quests in a virtual world improved performance and knowledge acquisition (Barab et
al, 2005). Kim, Park and Baek (2009) showed that meta-cognitive strategies, such as recording, modeling and thinking
aloud, inuenced social problem solving abilities and academic performance in a Massively Multiple Online Role Play-
ing Game. In each of these examples the video games were part of the learning activity but did not function as the objekt.
Rather, the object of the activities, the predmet, were to design or make a game (Robertson & Howells, 2008; Waraich &
Brna, 2008), stimulated imagination (Verenikina et al, 2008), mediate problem-solving through discussion (Barab et al.
2005; Seagram & Amory, 2005) and the use of heuristic tools in a virtual world (Kim et al, 2009). Therefore, the learning
outcomes were mediated through the use of a video game, a game environment, or through the puzzles/quests embedded
in a game. Shaffer and Clinton (2006) argued that tool-mediation, in their case a video game acted as the tool, is the
fundamental ontological unit of activity (p 289). In addition, these examples highlight the need of collaborative practice
to support learning.
Video games connect people socially and appear to be an important part of the educational gaming experience
(Rieber, Davis, Matzko & Grant, 2009). Foko and Amory (2008) support the importance of social collaboration during
game-play: students from disadvantaged backgrounds only showed improvement in understanding photosynthesis and
respiration when they played an educational game in pairs and when the game puzzles stimulated social dialog. However,
Schrader and McCreert (2008) argued that collaboration and mentoring is more likely to support novice game-play and is
of little important in achieving game objectives by players at the higher levels of game competency.
Video games, and media, as part of a contemporary classroom, are cultural artifacts that support learning when they
are not the object of a lesson (a learning from position) but when they function as a tool to mediate the learning task or
predmet (a learning with position). In addition, social collaboration fosters independent knowledge construction when
the technological artifact functions as an heuristic. This framework is referred to as an object-tool-social framework. The
primary object of this study is to investigate the use of game-mediate learning with pre-service teachers to evaluate the
use of a socially mediated knowledge construction where computers games function as extrinsic and implicit mediators.
According to Stake (1995) case studies can be divided into three types dependent on the aim of a study, including:
intrinsic case studies that investigate the uniqueness of the cases; instrumental case studies that are concerned with ad-
vancing theory; and collective case studies that make use of any number of cases as part of an instrumental case. This
study makes use of a collective case study approach to evaluate the use of the object-tool-social framework in the use of
video games in learning and teaching. The unit of analysis for the individual cases is how pre-service teachers understand
the use of video games in the classroom. The unit of analysis of the collective case is to build theory on how to use video
games in teaching and learning. Two individual cases (Creswell, 1998) form part of this study. In the rst case, third year
undergraduate Bachelor of Education students played an educational adventure game as the nal authentic task (Reeves,
Herrington & Oliver, 2004) in their professional development course. In the second case, postgraduate students read-
ing for their professional teaching diploma played another educational game as part of the practical component of their
In each case study the educational game was used in a collaborative learning process, as suggested by Amiel and
Reeves (2008), and not as the artifact, or tutor, for instruction. The unit of analysis is thus not the technological artifact
itself (the game in this case), but rather the process of students engaging with the technological tool to develop insights
into the biology of cancer, malaria, tuberculosis and HIV/AIDS (Seagram, 2005) [case 1] or to address misconceptions
related to Mendelian genetics (Baxter, 2008) [case 2]. The study was bounded by time (participants played the game for
a minimum of 10 hours), place (a computer lab at the University of Johannesburg) and by the participants. The research
inquired into the techniques and approaches of the use of games for teaching and learning in order to improve the design
of such learning events (Amiel & Reeves, 2008).
The research methodology in this study makes use of an eclectic-mixed methods-pragmatic approach (Reeves &
Hedberg, 2003) and thus includes qualitative and quantitative methods. Students texts were coded using content analysis
and descriptive statistical analyses were done using PASW Statistics (SPSS) version 18 from IBM.
With respect to the rst case, the third year students (n=184) were introduced to the theories related to authentic
learning and the object-tool-social framework. Both these theoretical frameworks were used in the design of the course.
Playing the educational game on diseases was the nal authentic task of the course and students were asked to install
the game, play in pairs and to try and complete the game. They were also told that they needed to solve all the puzzles
to obtain four cards and four keys along the way. During game play they were asked to think about the motive for play-
ing the game (in other words, they had to identify the object of the activity) and, using the frameworks they had used in
the course, they were asked to analyze the activity associated with their game play. There nal examination assessment
was a portfolio of work. One of the components of the portfolio was to select three of the course tasks and, and using the
frameworks, detail what they had learned during the course. Their performances in a number of the authentic tasks and
the relationship between their performances in these tasks (ANOVA), and the tasks they selected for their portfolio were
quantitatively analyzed for this research. Their written submissions on the game play tasks and their examination portfo-
lio submissions were quantitatively analysed deductively against the frameworks to gain insights into what they learned
through their game play.
A small group (n=11) of postgraduate students (case 2) played an educational game on genetics (Baxter, 2008)
over ve weeks, playing two hours per week. During game play two faculty members supported the students by guiding
their path through the game and facilitating discussions between the group participants. After the students had nished
the game a focus group interaction was held. During this session the students were introduced to the object-tool-social
framework of Vygotsky and asked how this framework related to the game play and the design of their learning experi-
ence. This case study made use of a pre- and post-test instrument (Baxter, 2008) to identify misconceptions related to
genetics the quantitative part of this case study to determine if collective game play helped students overcome mis-
conceptions related to genetics. During classroom interactions and the focus group sessions I kept notes that form part of
the qualitative analyses of this case study.
CASE ONE
Third year students performed the best in the mid semester test (72.0 1.7%), followed by the design of a computer
local area network for a school (69.2 1.2%), evaluation of the educational game (65.3 1.2%), the use of an interactive
whiteboard in a classroom (64.9 1.2) and the review of a chapter for publication (63.1 1.9%) (Table 1). Their perfor-
mances in the authentic learning and classroom design tasks were poor (less than 60%). However, many students selected
these tasks (mind map 14.2%; authentic learning 13.6%; and classroom design 10.8%) to illustrate what they had
learnt. In addition, 19.2 % of the students selected the interactive whiteboard and 16.4% the design of a computer labora-
tory to illustrate their knowledge. Only 8.2% of the group selected the game as one of the tasks to illustrate the knowl-
edge they gained during the course.
In order to better understand the relationship between student performance and the tasks they selected, their average
performance on each task was analyzed using ANOVA. Average performance was found to be signicantly different. The
signicance of the Levene test was less than 0.05 showing that the variances in performance for tasks were signicantly
different. Consequently, the Tamhane post hoc test was used to determine those tasks that were similar and those that
were statistically different in term of performance. Each task was therefore compared with all the other tasks and those
that are similar are shown in Table 1 (Similarity). Student performance appears to consist of two groups with one task
the chapter review intersecting both groups. The rst group includes those tasks with above average performance and
the second group where performance was less satisfactory. Apart from the interactive whiteboard and the design of the
computer laboratory tasks, most students selected the second group to illustrate what they learnt in the course.
Table 1. Performance by third year students in course work authentic tasks and examination portfolio tasks. Column 1
lists authentic task, column 2 lists percent of group choosing specic task as relevant for their learning, column 3 lists av-
erage percentage obtained by group for the task, column 4 the standard error, and the last column the statistical similar-
ity in performance of the different tasks (ANOVA F=22.61, p<0.001; Levene=13.71, p<0.001; Post hoc test = Tamhane).
These results suggest that participants could be divided into two general groups based on the assessment of each
tasks and their opinions as to what they learnt from the different tasks. Students who understood the two theoretical
frameworks, that is they scored higher for the task, were more likely to select the tasks in which they performed better.
This included the game evaluation activity. However, many students selected tasks that were directly related to their pro-
fessional practices. It is interesting to note that a small percentage selected the test as an example to what they learned.
This might be due to the nature of the test, where they had to analysis teaching activities from a text book in relationship
to the theoretical frameworks.
Content analyses of students game assignments and their examination scripts highlighted a number of interesting
points. While they were specically asked to identify the object of the activity, many students also included comments
regarding tool mediation, collaborative learning and authentic tasks.
A number of different positions were taken regarding the object of activity. Many students indentied playing the
game to learn about the diseases as the object of the activity. For example, [t]he object of the game was/is to teach about
the cause, effect and symptoms of diseases such as HIV/AIDS, TB and malaria and is om leeders deur middle van
praktiese metodes meer te leer oor siektes (is to teach students more about diseases using practical methods). Also,
solving puzzles was seen as the object of the activity: Deur die voltooing van die puzzles is die speler bedig on te leer
en navorsing te doen, sonder om dit te besef (Through the completion of the puzzles the players, without being aware,
is learning and undertaken research) and [t]he motive of playing this game being the object of the activity is that our
minds were stimulated because when we were playing we came across puzzles where we had to ll in missing answers.
Only one student realized that the primary object of the activity was to evaluate [the] game for learning.
The game also mediates ones learning process, as we had to gure how to play it without any instructions and
this game is a very good learning tool illustrate that some students understood the role of tools to mediate learning
extrinsically.
Statements such as [t]he game was not easy though because it challenged our mental agility in a lot of instances.
But because we did it as a pair the activity was manageable and we got to learn a lot from each other, was an opportu-
nity to interact socially as well as cognitively and social collaboration appears to be an integral part of the development
of insights and knowledge development show that the students clearly understood the importance of working together to
solve complex problems.
Many of the participants identied the game as authentic as it was set in Africa and was applicable in any context;
because the diseases addressed in the game are the same chronic illnesses that continue to affect our communities and
it uses a real life example of an intern treating patients in a village.
Third year pre-service teachers clearly appreciate the important of socially interactions in solving problems and the
use of authentic tasks; they did not fully understand tool mediation. However, they saw the value of using the game as a
tool to interact with the puzzles that led to knowledge construction.
This case study highlighted student understanding of authentic tasks and object-tool-social frameworks as important
components of learning and teaching. Many of the participants clearly understood collaborative puzzle solving and ar-
gued that educational games could play important roles in the learning process. However, this case study did not investi-
gate the learning that takes place during collaborative game play, which is the primary focus of the next case study.
CASE STUDY TWO
The multiple choice instruments used in the pre- and post testing in this case study were developed and evaluated by
Ivala (1999) to identify genetics misconceptions held by rst year university students. These identied misconceptions
were later used by Baxter (2008) to develop a set of learning objectives for the design and development of an adventure
video game. Baxter designed the game story, puzzles and environment to specically address these misconceptions. The
testing instrument included eighteen questions to test misconceptions around genetics.
For this case study the pre-testing scores of postgraduate participants (n=11), all biology graduates, were poor (29.8
4.7% correct responses). After 10h of game play eight participants remained and they overcame some of their miscon-
ceptions related to genetics (44.4 6.5% correct responses). The improvement in their score was statistically signicant
(t-test = -3.69%, p(T<=t) one-tailed < 0.001). While the participants increased their understanding of many concepts,
there was no improvement in some of the questions testing misconceptions (for example, questions 1 and 2) and in two
there was a decrease in understanding (questions 10 and 11) (Fig. 1).
Pre-test Post-test
Figure 1. Pre- and post-test results of the individual question use to identify misconceptions related to Mendelian genetics.
During the post-test focus group discussion the design of the learning with the video games was discussed: the puz-
zles acted as a device (tool) to facilitate discussion between the pair of team players and thereby mediated learning. One
student noted that the faculty facilitators helped students solve one of the particularly difcult puzzles. But, in analyzing
what the facilitators did the participants realized that they were not told the answer, but shown a way in which they could
solve to puzzle for themselves. The participants understood the concept of tool-mediated learning and one said the game
design was an affective establishment and maintenance of Vygotskys zone of proximal development. When asked if
it was appropriate to upload PowerPoint lectures onto the Universitys on-line learning management system, most of the
participants answered that the presentation was not about mediating learning, but about consumption of information.
Showing that that understood the concept of tool mediation in another situation.
Misconceptions are often deep seated and difcult to overcome. However, game play, in this case study, allowed
students to overcome some of their misconceptions related to genetics and to understand the use of tool-mediation in the
learning process. It is argued that for such learning to take place during game play, the game (or game components such
as the puzzles) facilitated learning through the establishment of a zone of proximal development, and that social collabo-
ration supported problem solving. In other words the game and game puzzle functioned as a tool to mediate knowledge
The two individual intrinsic cases reported here are part of a collective instrumental case study undertaken to ad-
vance the theory of the use of video games in learning and teaching. First, pre-service teachers found the object-tool-
social and authentic frameworks useful in the development of their professional teaching skills. Second, the use of these
frameworks helped students to overcome some of their misconceptions related to genetics. The Vygotskian concept of
social tool-mediated knowledge construction provided an appropriate framework for the use of video games in teaching
and learning. The introduction of game puzzles into a learning activity acted as the extrinsic mediator, while the dis-
cussions between players intrinsically mediated their understanding. Even epistemic (knowledge) games (Salmani No-
doushan, 2009; Shajjer & Gee, 2005) could be useful in the development of knowledge, but such cultural artifacts need
to function as tools to mediate an understanding of the world we live in and not instruct learners to see the world from
the perspectives of the authors of these cultural artifacts.
Akl, E. A., Gunukula, S., Mustafa, R., Wilson, M. C., Symons, A., Moheet, A., & Schnemann, H. J. (2010). Support for and as-
pects of use of educational games in family medicine and internal medicine residency programs in the US: a survey. BMC
Medical Education, 10(1), 26.
Amiel, T., & Reeves, T. C. (2008). Design-based research and educational technology: Rethinking technology and the research
agenda. Educational Technology & Society, 11(4), 29-40.
Amory, A. (2010). Learning to play games or playing games to learn? A health education case study with Soweto teenagers.
Australasian Journal of Educational Technology, 26(6), 810-829.
Barab, S. A., Evans, M. A., & Baek, E. O. (2004). Activity theory as a lens for characterizing the participatory unit. In D.H.
Jonassen (Ed.), Handbook of research on educational communities and technology (pp. 199-214). Washington, D.C.: As-
sociation for Educational Communication and Technology.
Barab, S., Thomas, M., Dodge, T., Carteaux, R., & Tuzun, H. (2005). Making learning fun: Quest Atlantis, a game without guns.
Educational Technology Research and Development, 53(1), 86- 107.
Baxter, D. (2008). Addressing genetics misconceptions with an educational game. Unpublished Masters dissertation, University
of KwaZulu-Natal, Durban, South Africa.
Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among ve traditions. Thousand Oaks, London, New
Delhi: Sage Publications, Inc.
Doehler, S. P. (2002). Mediation revisited: The interactive organization of mediation in learning environments. Mind, Culture,
and Activity, 9(1), 22-42.
Edwards, M. G. (2008). Every today was a tomorrow: An integral method for indexing the social mediation of preferred fu-
tures. Futures, 40(2), 173-189.
Engestrm, Y. (2000). Activity theory as a framework for analyzing and redesigning work. Ergonomics, 43(7), 960-974.
Engestrm, Y. (2001). Expansive Learning at Work: toward an activity theoretical reconceptualization. Journal of Education and
Work, 14(1), 133-156.
Foko, T., & Amory, A. (2008). Social Constructivism in Games Based Learning in The South African Context. In World Confer-
ence on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 5757-5764). Vienna, Austria: AACE.
Gibson, T. (2008). Double Vision: Mc Luhans contributions to media as an interdisciplinary approach to communication, cul-
ture, and technology. MediaTropes, 1, 143-166.
Greeneld, P. (1994). Video games as cultural artifacts. Journal of applied developmental psychology, 15(1), 3-12.
Ivala, E. (1999) Identifying Learning Misconceptions in Genetics. Unpublished Masters Thesis, University of Natal, Durban,
Jonassen, D. H., & Reeves, T. C. (1996). Learning with technology: Using computers as cognitive tools. In D. H.
Jonassen (Ed.), Handbook of research on educational communications and technology (pp. 693-719). New York: Macmillan.
Kaptelinin, V. (2005). The Object of Activity: Making Sense of the Sense-Maker. Mind, Culture, and Activity, 12(1), 4-18.
Ke, F. (2008). A Qualitative meta-znalysis of computer games as learning tools. In R. Ferdig (Ed.), Handbook of research on ef-
fective electronic gaming in education (Vols. 1-3, Vol. 1, pp. 1-32). IGI Global.
Kellner, D., & Share, J. (2005). Toward critical media literacy: Core concepts, debates, organizations, and policy. Discourse:
studies in the cultural politics of education, 26(3), 369-386.
Kim, B., Park, P., & Baek, Y. (2009). Not just fun, but serious strategies: Using meta-cognitive strategies in game-based learn-
ing. Computers & Education, 52(4), 800-810. doi:doi:10.1016/j.compedu.2008.12.004
Leontev, A. N. (1978). Activity, personality, and consciousness. Englewoods Cliffs: Prentice-Hall.
Levykh, M. G. (2008). The affective establishment and maintenance of Vygotskys zone of proximal development. Educational
Theory, 58(1), 83-101.
Mitchell, W. J. T. (2008). Addressing media. MediaTropes, 1, 1-18.
Nardi, B. A. (1996). Context and Consciousness: Activity Theory and Human-Computer Interaction. MIT Press.
Oliver, M., & Carr, D. (2009). Learning in virtual worlds: Using communities of practice to explain how people learn from play.
British Journal of Educational Technology, 40(3), 444-457.
Piaget, J. (1977). The development of thought: Equilibration of cognitive structures. Viking Press.
Reeves, T. C., & Hedberg, J. G. (2003). Interactive Learning Systems Evaluation. Englewood Cliffs, New Jersey: Educational
Technology Publications.
Reeves, T. C., Herrington, J., & Oliver, R. (2004). A development research agenda for online collaborative learning. Educational
Technology Research & Development, 52(4), 53-65.
Rieber, L. P., Davis, J. M., Matzko, M. J., & Grant, M. M. (2009). Children as critics of educational computer games designed
by other children. Handbook of research on effective electronic gaming in education, 1234-1256.
Robertson, J., & Howells, C. (2008). Computer game design: Opportunities for successful learning. Computers & Education,
Roth, W. M., & Lee, Y. J. (2007). Vygotskys Neglected Legacy: Cultural-Historical Activity Theory. Review of Educational
Research, 77(2), 186232.
Salmani Nodoushan, M. A. (2009). The ShafferGee perspective: Can epistemic games serve education? Teaching and Teacher
Schrader, P. G., & McCreery, M. (2008). The acquisition of skill and expertise in massively multiplayer online games. Educa-
tional Technology Research and Development, 56(5), 557-574.
Seagram, R. (2005). Use of constructivism in the development and evaluation of an educational game environment (PhD). Uni-
versity of KwaZulu-Natal, South Africa.
Seagram, R., & Amory, A. (2006). An Assessment of Learning Through the Use of a Constructivist Learning Environment. In
E. Pearson & P. Bohman (Eds.), World Conference on Educational Multimedia, Hypermedia and Telecommunications 2006
(pp. 2165-2172). AACE.
Shaffer, D. W., & Clinton, K. A. (2006). Toolforthoughts: Reexamining thinking in the digital age. Mind, Culture, and Activity,
Shaffer, D. W., & Gee, J. P. (2005). Before every child is left behind: how epistemic games can solve the coming crisis in educa-
tion. WCER Working Paper No. 2005-7. Wisconsin Center for Education Research (NJ1), 18.
Stake, R. E. (1995). The art of case study research. Sage Publications, Inc.
Stetsenko, A. (2004). Tool and sign in the development of the child. In The essential Vygotsky (pp. 501-512). New York, New
York: Kluwer Academic/Plenum Publishers.
Stetsenko, A. (2005). Activity as object-related: Resolving the dichotomy of individual and collective planes of activity. Mind,
Culture, and Activity, 12(1), 70-88.
Torres, M., & Mercado, M. (2006). The need for critical media literacy in teacher education core curricula. Educational Studies,
Verenikina, I., Herrington, J., Peterson, R., & Mantei, J. (2008). The affordances and limitations of computers for play in early
childhood. In World Conference on Educational Multimedia, Hypermedia and Telecommunications (pp. 3091-3100). Pre-
sented at the ED-MEDIA, Chesapeake, VA: AACE.
Vygotsky, L. (1933). Mind in society. The development of higher psychological processes. Cambridge, MA: Harvard University
Waraich, A., & Brna, P. (2008). A narrative centred Informant Design approach for interactive learning environments. Interna-
tional Journal of Continuing Engineering Education and Life Long Learning, 18(2), 181-196.
Wertsch, J. (2007). Mediation. In H. Damiels, M. Cole, & J. V. Wertsch (Eds.), The Cambridge Companion to Vygotsky (pp.
178-192). New York: Cambridge University Press.
Beyond Immersion Meaningful Involvement in Virtual Worlds 111
Beyond Immersion Meaningful Involvement in Virtual Worlds
ANDREW CRAM, JOHN G. HEDBERG, AND MAREE GOSPER
There are growing numbers of educational simulations being developed using three-dimensional multi-user virtual
environments (MUVEs). One reason touted for the adoption of MUVEs in education is that virtual worlds provide an im-
mersive experience for the learner (de Freitas, Rebolledo-Mendez, Liarokapis, Magoulas, & Poulovassilis, 2010; Dede,
2009). This immersive experience can facilitate learning by enabling multiple perspectives, situated learning and transfer
(Dede, 2009). There remains scope for further research, however, to understand how learners become immersed within
a virtual environment. One aspect of a users experience within a virtual environment that is argued to inuence their
perception of immersion is the involvement of the user in meaningful activities within that environment (Calleja, 2007;
Witmer & Singer, 1998). People are involved when they are directing their attention and energy towards a set of mean-
ingfully related activities and events (Witmer & Singer, 1998, p. 227), and deep involvement in an activity is character-
ised by the shortening or disappearance of distance between the user and the environment (Calleja, 2007, p. 254) that
occurs when immersed in a virtual environment.
In an educational simulation, learners become involved in activities that facilitate their progression towards the de-
sired learning outcomes. However, distracted or otherwise off-task learners may progress through the simulation without
becoming involved in the experiences necessary for the intended learning outcomes. Likewise, learners may be meaning-
fully involved in a sequence of learning activities within an educational simulation, but not feel the shortening of distance
that characterises immersion. Nonetheless, focusing on involving the learner in a sequence of meaningful activities and
events is likely to be an effective way of immersing the learner in the simulation.
This paper reports on a research study that seeks to describe participants experiences of immersion within an open-
ended problem solving simulation using a MUVE. This involves evaluation and renement of a framework for describing
a users involvement in the educational simulation, focusing on classifying different modalities of involvement, dened
as the different aspects of the experience that the user attended to and experienced involvement with. This framework is
developed through modication of Callejas (2007) Model of Digital Game Experience, and considers six modalities of
involvement: tactical, narrative, performative, shared, spatial and affective. After describing how MUVEs may be used
for educational simulations, the paper examines the relations between immersion, involvement and education, with a fo-
cus on the six modalities of involvement. Then, the simulation and research methodology are described, followed by the
results and a discussion of links between the participants involvement within the experimental simulation and their feel-
ings of immersion.
MUVES FOR EDUCATIONAL SIMULATIONS
Simulations are used in formal education to scaffold and guide learners through a series of targeted experiences that
are intended to facilitate the learners achievement of certain learning outcomes. Learning within an educational simula-
tion, similar to learning within everyday situations, occurs as participants accumulate experiences and come to recognise
and understand the value of conceptual and physical tools for the achievement of goals (Barab et al., 2007; Gee, 2003).
Using MUVEs, educators can design situations and narratives in which learners control an avatar within a three-dimen-
sional environment interacting with embedded information, virtual objects and other users.
The activities and events that may be designed within different MUVE platforms depend on the types of represen-
tations and user interactions that are afforded by the software. While each MUVE platform has inherent constraints in
terms of perceptual delity and the types of user interactions that are afforded, it is possible to conceptualise four rep-
resentational opportunities that simulation designers may take advantage of: space, time, place and user representation
(Hedberg, Cram, Lumkin, & Eade, 2010). For example, the Taiga Park virtual world in Quest Atlantis assists learners to
understand the processes of erosion and eutrophication by considering the spatial relationships of the river, logging camp
and shing areas (Hickey, Ingram-Goble, & Jameson, 2009). Although not achieved in Taiga Park, it would be possible
to design a MUVE simulation that represents the processes of erosion and eutrophication over time.
112 Cram, Hedberg and Gosper
Space relationships between points and the reasons why they are linked or related.
Time the space represented within a 3D virtual environment may be transformed
over time, to simulate physical relationships in four dimensions.
Place a point in space that has cultural and other meaningful attributes that situate
the thinking.
of users
each user can be represented by an avatar, who can perform embodied
actions such as traversing through space and engaging in non-verbal
communication.
Designers can choreograph learning trajectories through a MUVE by manipulating the learners starting point and
the sequence through which information and interactions are made available (de Freitas & Neumann, 2009). The chal-
lenge for designers is to calibrate the simulation narrative so that the activities and events experienced by the learner fa-
cilitate and encourage progress towards achievement of the intended learning outcomes. Simulation narratives can be lin-
ear, with a single sequence of activities and events for all learners, or non-linear, in which case the sequence of activities
and events is determined by the learner. Non-linear narratives may differ in the extent to which the activities and events
are pre-embedded, compared with user-generated. Pre-embedded activities and events, which often use scripted bots,
are easier to align with learning outcomes and provide targeted support materials. User-generated activities allow learn-
ers to contribute to the narrative, potentially generating more personally meaningful activities that encourage creativity,
although it may be a challenge to provide support materials and align the learners activities with the intended learning
outcomes.
With different combinations of narrative and representations, educational designers can create a wide variety of
learning experiences. Taiga Park (in Quest Atlantis) and River City are examples of exploratory, inquiry based MUVEs
(Hickey et al., 2009; Ketelhut, Nelson, Clarke, & Dede, 2010). Within these, students follow pre-embedded, non-linear
narratives, within a simulated setting to complete a socio-scientic inquiry. The learning activities include collecting
and interpreting data, setting and testing hypotheses, and forming conclusions and recommendations. Other approach-
es involve role play, and design and construction activities. In role plays, learners take different positions and perspec-
tives, and interpret the situations they encounter from a particular point of view. For example, learners may research one
of their role models, then assume that role in discussions with other students within the MUVE (Mayrath, Traphagan,
Heikes, & Trivedi, 2009). In design and construction activities, learners create and transform virtual objects to build an
artefact according to a specic design brief. An example is for learners to design and construct their ideal university cam-
pus buildings, in order to elaborate their sense of place and architectural style (Mayrath et al., 2009). These approaches
are not mutually exclusive; combinations of exploratory, role play and design and construction approaches may be in-
cluded within a single simulation.
IMMERSION AND INVOLVEMENT IN MUVES
Interest in immersion within three-dimensional virtual environments has its origins in research with virtual reality
systems using head mounted display and CAVE technologies (for example, Slater & Wilbur, 1997; Witmer & Singer,
1998). Following the increased prevalence of desktop virtual environments, this interest in immersion has been extended
to research on video games (Calleja, 2007; McMahan, 2003) and educational virtual worlds (Dede, 2009). Throughout
this time, there have been a wide variety of denitions applied to the term immersion (and the related term presence), and
it is necessary to clarify usage of the term: the feeling of immersion within a MUVE occurs when the user is absorbed in
the experience, and results in a shortening or disappearance of distance between the user and the environment (Calleja,
2007, p. 254). This usage is similar to how the term presence is used by Witmer and Singer in relation to a virtual envi-
ronment (1998, p. 225): experiencing the computer-generated environment rather than the actual physical locale, as
well as other constructs in the HCI literature based on the notion of ow (Csikszentmihalyi, 2002), such as cognitive ab-
sorption (Agarwal & Karahanna, 2000).
A wide range of variables have been identied in the telepresence literature that impact feelings of telepresence, and
that may inuence feelings of immersion (Lee, 2004). In relation to MUVEs, however, the discussion has been much
more restricted. de Freitas et al. (2010, p. 70) contend that MUVEs are immersive because the re-embodiment of the user
as an avatar promotes a feeling of presence. Dede (2009) takes a broader view, arguing that immersion occurs through
the conjunction of sensory, actional and symbolic factors within the MUVE design. In an approach that is closely related
to Dedes, we explore the relations between a users involvement (direction of attention and energy) and feelings of im-
mersion. A users involvement in a MUVE will occur through the sensory, actional and symbolic factors, although a
change in the analytic focus will facilitate the foregrounding of different aspects of the users experience. The objective
is to evaluate the ways in which a users immersion in a MUVE simulation is mediated by their involvement in the activi-
ties and events that comprise the simulation.
Involvement, as dened by Witmer and Singer (1998, p. 227), is experienced as a consequence of focusing ones
energy and attention on a coherent set of stimuli or meaningfully related activities and events. Essentially, learners are
involved in whatever they are directing their energy and attention towards. This can be further claried with consider-
ation to different narrative types. In a linear simulation narrative, all learners will progress through the same sequence of
activities and events. However, their attention within these activities and events may be directed to different aspects of
their experience. For example, people with different levels of expertise may attend to different patterns within a situation
(Chase & Simon, 1973), which may modify the form of activity they are involved in and therefore lead to different expe-
riences. The notion of different forms of activity may become clearer below, with a discussion of the different modalities
of involvement. Within a non-linear narrative the differences between what learners are involvement in may become even
more pronounced, as each learner may complete a different set of activities and events.
Considering immersion through the lens of involvement requires a framework that allows analysis of the ways in
which the user directed their attention and energies. The challenge is to nd a categorisation scheme that is comprehen-
sive enough to cover all aspects of users experiences, yet still facilitates data analysis that is useful and relevant to the
evaluation of experiences of immersion within educational simulations. A suitable framework may be grounded in the
Digital Game Experience Model (Calleja, 2007), originally generated from qualitative thematic analysis of interviews
with 3D video game players. This model denes six modalities of involvement that describe different aspects of the ex-
perience that users attend to and experienced involvement with.
Tactical all decision making within the game.
Performative relates to all modes of avatar or game piece control.
Affective this category is not dened within the Digital Game Experience Model; within this paper it covers all aspects
of emotional arousal through engagement with the simulation.
Shared covers all aspects of communication with and relation to other agents in the game world.
Narrative deals with all aspects of engagement with the designed narrative. This is a narrower denition than used
in the Model of Digital Game Experience, which included the ow of players personal experiences in the game
world and was too broadly dened as all involvement related to the ow of personal experiences.
Spatial related to locating oneself within a wider game area than is visible on the screen.
This paper reports on the following questions:
1. To what extent do these six categories (tactical, performative, affective, shared, narrative, and spatial) comprehensively
describe learners experiences of immersion within the MUVE educational simulation?
2. What relationships are found between the users experiences of different modalities of involvement and their perception
of immersion within the simulation?
Data were gathered from nine participants who completed a role play simulation within a MUVE. The participants
were volunteers who were university employees with varying experience and condence with virtual worlds and problem
solving. Each of the participants went through a one hour group training session, consisting of three group activities de-
signed to allow the participants to practice the skills that would be required in the scenario. Within one week of the train-
ing session, the participants individually completed the scenario. The participants were asked whether they felt immersed
within the scenario, and to further explain their response. The data were categorised using the six modalities of involve-
ment, to evaluate the comprehensiveness of the framework and explore the relations between immersion and involve-
ment. Further details on the scenario design and data analysis are below.
Scenario design: The role play simulation was developed using the Activeworlds MUVE technology. The participants
assumed the role of Operational Health and Safety (OH&S) manager in a manufacturing facility, and interacted with
two actors, who played the Operations Manager (Sarah) and a Director (Tim) within the same company. The scenario
was choreographed to elicit a response from the participant to a situation occurring in the facility. This situation was
intended to present an ethical dilemma to the participant, through consideration of the implications of initiating a safety
audit within the facility. Initiating a safety audit would enable a review and improvement of worker safety conditions and
practices, but would also involve the facility being temporarily shut down with contracted workers being placed on half
pay for the duration of the audit. This narrative was designed to provide participants with a meaningful set of activities
and events, such that the participant was required to exert their agency in order to progress through the simulation. The
scenario was designed to be approximately 20 minutes in length. A background information sheet was provided prior to
the start of the scenario, to orient the participant with the role and context. Participants were able to clarify details before
starting the scenario.
The virtual environment was set up with a four level ofce building, surrounded by a landscaped outdoor area. The
scenario took place on the ground and top oors of the ofce. The ground oor consisted of a reception area and two of-
ce areas, one for the OH&S manager and one for the Operations Manager. One ofce presented three wall charts with
data on the recent history of accidents within the plant, the current progress of machine inspections, and the planned ini-
tiatives to improve worker safety. The Directors ofce was on the top oor. Within the scenario, participants were able
to chat using audio or text, move their avatar around the environment, adjust their camera perspective, trigger gestures for
their avatar, and interact with the buildings front doors and elevator.
Data analysis: To answer the questions posed above, a qualitative coding analysis was performed on the participants
reections of their experience of immersion within the scenario. These data were obtained by asking participants the fol-
lowing question in a semi-structured interview conducted after the participant had completed the scenario: Did you feel
immersed in the scenario? Please explain. Participants were prompted to provide a detailed explanation, and to avoid
guiding the responses the term involvement was not used. The responses were recorded using a microphone and tran-
scribed for analysis. The six modalities of involvement were used as categories for data coding. The description used to
guide coding for each category is detailed in the Discussion section below.
An outline of the modalities of involvement reported by each participant is presented in Table 1.
Table 1: Categorisation of Modalities of Involvement within participants perception of immersion
# Reported
Coding details
114 Yes: Absolutely Performative: felt more immersed when she could manage the actual world
and the environment. However, at the start she felt less immersed when she
was unsure of her ability to control the environment.
Tactical: felt that the ability to utilise her background experience to respond to
the issues as an OH&S manager, contributed to her feeling of immersion.
Affective: initial nervousness decreased the level of involvement.
Narrative: felt more immersed as she felt surer of the role she was playing,
without having to check the background information sheet.
162 Yes: Yes I did Tactical: felt that the way she responded to issues and assumed the OH&S
role within the scenario contributed to her feeling of immersion.
Shared: considering the perspectives of the other people within the scenario
contributed to the feeling of immersion.
Narrative: noted that there was limited information available within the sce-
nario, but this didnt signicantly inhibit her feeling of immersion.
362 Partial: if it was a
yes no answer Id
Performative: the ability to interact with objects within the world, and use the
avatar in a natural way contributed to the feeling of immersion.
526 Yes: I felt totally
Shared: the other people within the scenario contributed to the feelings of
immersion.
598 Partial: I sup-
pose partially
Shared: the embodied interactions with Sarah contributed to feelings of immer-
sion. However, the lack of other people, for example other employees, was felt
to have inhibited immersion.
Narrative: showing the charts on the wall assisted immersion.
775 Yes: Yes I did,
it had a lot of
Tactical: considering the decisions in terms of the OH&S role and the ethical
implications was related to high levels of immersion.
Narrative: the consideration of the OH&S role contributed to feelings of immer-
sion, through the need to make decisions.
Shared: reported that the high latency in aural communication didnt signi-
cantly affect feelings of immersion.
780 Yes (partial):
Oh yeah you do
Narrative: the amount of information accessible within the scenario was felt to
be insufcient for the participant to feel totally immersed.
844 Yes: Yes, quite
Narrative: The participant was able to take on the role, and reports that this
contributed to her feelings of immersion. Specic elements in the scenario that
assisted this were the initial brieng before the scenario and the discussion of
the background information with Sarah.
Spatial: the tour of the ofce space contributed to the feeling of immersion.
850 Partial: I did to a
certain extent
Shared: the actors who were present in the simulation contributed to feelings
of immersion, however did not become meaningfully involved in relation to the
employee who were referred to but not present within the simulation.
Affective: did not feel emotionally engaged at all.
Tactical: did not feel that her decisions were consequential, as they did not
involve real people.
The data was also reviewed to identify any other relevant themes, with specic focus on parts of the data that were
not coded with one of the six frames of involvement. This was done to check whether the six categories of involvement
were sufcient to comprehensively describe the participants reections of immersion. One theme emerged, relating to
the perceptual delity of the visual representations within the virtual environment. This covered the delity of representa-
tion and the verisimilitude of the objects and people that were present in the environment.
Table 2: Categorisation of Perceptual Fidelity within participants perception of immersion
# Coding of Perceptual Fidelity
362 looking at the detail in representation of the shtank and the buttons in the lift was im-
mersive; while the relatively low delity of the graphics compared with modern computer
games, including not having the avatars mouth move when talking, was reported to inhibit
598 the gap between the physical presence and the purposeand indeed the decisions, for
example the inclusion of a shtank rather than other company related artifacts such as
miniature tractors.
775 delay in vocal responses due to latency in the audio channel between the participant and
the other actors (somewhat negative, but surprisingly immediate: #775).
850 felt immersed to a certain extent as the environment was real-ish, without having bells
and ambulances and stuff like that.
Tactical: covers all decision making within the simulation. For coding, this category involves all aspects of the partici-
pants experience that relates to problem solving, including the consideration of the ethical issues.
Involvement in decision making was central to many of the participants perceptions of their immersion within
the scenario, but was reported in relation to both high and low levels of immersion. The three participants (#114, #162
and #775) who related decision making to high levels of immersion highlighted the way they approached the decisions
through the role of OH&S manager. Within this scenario, then, the assumption of the role provided was a critical factor
for participants to relate tactical involvement and high levels of immersion. Two of these participants (#114 and #162)
also mentioned how they were able to use their background experiences within their decision making. For these two par-
ticipants, there are clear parallels between their experience and the notion of projective stance proposed by Gee (2008) in
relation to video games. Gee also argues that game players actions result from the integration of the goals of the game
character and the game player, framed within the constraints of the actions that the character may enact within the game
world. Furthermore, when the goals and actions mesh, the game player gains deep pleasure through feeling mastery and
control; feelings which are theorised to be related to experiences of involvement, immersion and presence (Agarwal &
Karahanna, 2000; Witmer & Singer, 1998).
Another factor, consideration of personal morality and ethics, was reported by one participant (#775) as an aspect of
decision making that was related to a high level of immersion. As all of the participants, except one (#850), had consid-
ered ethical issues within the scenario, it appears that the assumption of a role while making decisions is a more impor-
tant aspect of immersion than the content of the decision.
Links between Tactical and Shared emerged, as one participant (#162) discussed how she considered the perspec-
tives of the other agents when making decisions, while another participant (#850) felt that the decisions she was making
did not implicate real people (referring to the workers who were impacted by the decision, but who did not make an
explicit appearance within the scenario) and that this inhibited immersion. Both participants were making decisions and
Tactically involved in the simulation, but their perceptions of the other people in relation to the decisions was more sig-
nicant in determining their feeling of immersion.
Affective: Calleja (2007) does not provide any denitive description of this modality of involvement. The denition
used to guide coding and analysis was all aspects of emotional arousal through engagement with the simulation. Par-
ticipants emotional responses were a signicant inuence on the level of immersion perceived, although only two par-
ticipants mentioned these when reecting on immersion. Both of these participants related their emotional response, or
lack thereof, to a low level of immersion within the scenario. It was evident that other participants experienced signicant
emotional responses during the scenario, however they did not explicitly report that this impacted their level of immer-
sion.
Participant #114 reported feeling nervous at the start of the scenario, when she was not feeling immersed. However,
as she gained condence in being able to manage the controls and the situation, and presumably felt less nervous, she
also became more immersed within the scenario. This nding is in contrast to research that reports that inducing anxi-
ety in users of virtual environments, by leading them to believe there are dangerous snakes hidden in the environment,
increases feelings of presence (Bouchard, St-Jacques, Robillard, & Renaud, 2008). The difference in the ndings may
be explained by considering the focus of attention in each case. Participant #114 in this research felt nervous about the
experiment as a whole and her ability to control the situation, while the participants reported in Bouchard et. al. (2008)
felt anxious in relation to something that was within the environment. It is not simply the occurrence of an emotional
response that deepens feelings of immersion, but what the participant is involved in that triggers the emotional response.
In contrast, participant #850 noted that she felt no emotional links to the other people in the scenario, and that this
was a signicant aspect of her feeling of experiencing a low level of immersion. It is not clear in this case whether an
increased level of immersion would have facilitated an emotional response, or alternatively if an emotional response had
been elicited then she would have felt greater immersion. These reports do suggest, however, that being able to feel some
emotional arousal is important to feelings of immersion, but that arousal needs to emanate from activity that is directed
within the simulation rather than activity that is directed at the simulation.
Shared: covers all aspects of communication with and relation to other agents in the game world (Calleja 2007). For
coding during data analysis, this was operationalised as any reference to communication with and/or the presence of
other agents, whether an actor that was directly interacted with, or a third party whos existence was implied within the
scenario, for example the other workers. References to other agents were a signicant factor of participants experiences,
with six of the nine participants discussing other agents in their reections on immersion. These references were varied
in nature, covering the presence or absence of embodied agents, the consideration of the role and background of the other
agents, the general believability of the agents and the impact of the high levels of latency in the audio channel.
One participant (#598) reported that seeing the actor, Sarah, next to the information charts on the wall, contributed to
his level of immersion. On the other hand, this participant also reported that having only two embodied actors within the
scenario detracting from his level of immersion, as more people would be expected within the ofce space. The absence
of other embodied agents affected another participant (#780) in a different way, focussing on how that lack of other peo-
ple meant that she could not obtain information desired for calibrating her response to the situation. For this participant,
the issue was not a lack of presence, but a limitation on her ability to act as she would in a similar situation in everyday
activity. The latter participant (#780) reported a higher level of immersion than the former participant (#598), indicating
that a focus on how the presence or absence of agents impacts activity and decision making, rather than focusing on the
embodied presence of the agents as an end itself, may be characteristic of an increased level of immersion. Further sup-
port for this claim is found in the reections of another participant (#162), who reported that one of the factors relating to
her high level of immersion in the scenario was how she considered the role and background of Sarah, including whether
Sarah was part of a union, when she was responding to the situation within the scenario. For this participant, immersion
meant considering the identity and background of the other agent, rather than simply being engaged in embodied interac-
tions with another agent.
Participant #850, who did not feel immersed within the scenario, highlighted another potential effect of the presence
or absence of agents within the scenario. This participant reported that while she felt that the agents she interacted with
were real virtual people, who had positions, she didnt feel that the workers impacted by the audit were real people
(these workers were indirectly referred to, but were not embodied by agents within the scenario), and thus her decision
making did not seem consequential. In this case, having embodied interactions with the workers may have contributed to
this participants meaningful involvement in the scenario. Another participant (#526) who felt a high level of immersion,
simply stated that the people were believable.
Throughout out all the scenarios, a high level of latency (delay between one person talking and the other person
hearing) in the audio communication resulted in sporadic breakdowns in discourse, as the participant and actor often
talked over the top of one another and subsequently had to repair the conversation in order to continue. However, this
apparently did not signicantly impact feelings of immersion. Only one participant (#775) mentioned the latency when
reecting on their level of immersion, reporting that while the latency did cause issues for communication, the scenario
experience was still surprisingly immediate. This nding is consistent with other research that reports that users en-
gagement within 3D virtual environments is often resilient to potential disruptions or breakdowns in their activity (Cheng
& Cairns, 2005).
Performative: relates to all modes of avatar or game piece control (Calleja 2007), and was used to code any references
to using the virtual world interface, moving the avatar or interacting with an object by clicking on it. Two participants
discussed their immersion with reference to the performative involvement with the scenario.
Participant #114 initially felt nervous about managing the actual world and the environment, but became less ner-
vous and more immersed as she gained condence. She did not discuss any specic aspects of avatar control, and this
experience is discussed in the section concerning the Affective modality of involvement. The other participant (#362)
reported three specic instances of avatar control that he felt were immersive: when he walked into the lift, he wanted to
turn around because he wanted to face outward like I would in a real lift, rather than continue to face the wall; interact-
ing with the buttons in the lift, to change oors; and when he was talking with Sarah he I wanted to turn around and face
Sara while she was talking. I didnt want to be listening like on a telephone. With the exception of the interaction with
the lift buttons, the instances of avatar control that were signicant to participant #362 occurred in relation to something
else, either the lift space or Sarah. This nding does support the stance of de Freitas et al. (2010) that immersion is re-
lated to the embodiment of the user within the virtual environment through the control of the avatar, although for most of
participants within this study the performative aspects of their experience were not signicant enough to discuss in rela-
tion to their immersion.
Narrative: includes all aspects of engagement with the designed narrative (Calleja 2007). This category included any
reference to the scenario background or information within the scenario that contributed to the progression of the par-
ticipant through the scenario. This category was prevalent in most of the participants reections on immersion. Only
one participants reection did not include any reference to the designed narrative. This may be expected with a loosely-
scripted role play simulation, in which the designed narrative plays a critical role in progressing the user through the sce-
nario. This modality of involvement may be less relevant in exploratory or construction based simulations.
Several aspects of the participants interactions with the designed narrative were related to high levels of immersion,
included how the scenario was set up through the initial brieng, overall context, being shown the building, and talk-
ing about the background information, injuries and machine inspection reports (#844), interactions with the actors and
charts of data (#850 and #598 respectively), and being able to engage with the scenario without referring to the scenario
background documentation (#114). One participant (#775) had the general comment that it kind of put together a very
believable scenario. Between them, these reections cover all aspects of the designed narrative, indicating that for role
play scenarios, at least, it is critical to include background information about the situation and that interacting with infor-
mation that further progresses the story also encourages immersion.
Limitations on the amount of information available within the scenario were reported by two participants (#598,
#780) as inhibiting their feeling of immersion. However, another participant (#162) reported that although she was aware
that there was less information available within the simulation than would be available in a similar situation in everyday
activity, this did not negatively impact her feelings of immersion. This equivocal nding indicates the existence of medi-
ating factors in determining the impact of limited information on level of immersion, but does not indicate the nature of
these factors.
Spatial: related to locating oneself within a wider game area than is visible on the screen. This category was not par-
ticularly relevant for describing participants involvement within the simulation. This may be because the scenario was
linear in its progression through the space, so locating oneself within the wider area was not particularly meaningful to
participants activities within the scenario. One participant (#844) indicated that being shown around the building helped
set the scene and assisted feelings of immersion. A factor which was reported to reduce immersion was being prevent-
ed from going to other locations such as rst aid, which were referred to but were not explicitly included in the simulated
(reported by participant #780). However, participant #114 also encountered this limitation within the scenario, but did
not deem this signicant enough to mention in her reection of immersion. This difference in impact on immersion is
similar to the way that different participants reacted to the limitations of information, discussed within the Narrative cat-
egory above.
Perceptual Fidelity: As outlined in the results above, the six categories relating to involvement were insufcient to com-
prehensively code the data. A further category, relating to the perceptual delity of the experience, was necessary to
categorise data which had not been classied as involvement. This category covers both the representational delity, and
the verisimilitude of the objects that were present in the environment. The delity of the simulation representations was
important to participant #362, who reported immersion when looking at the detail in the representation of the shtank
and the buttons in the lift, but that the relatively low delity of the graphics compared with modern computer games, in-
cluding not having the avatars mouth move when talking, inhibited his immersion. Representational delity was also im-
portant to participant #850. The verisimilitude of the objects present within the environment was signicant to participant
#598, who related his low level of immersion within the simulation to the gap between the physical presence and the
purposeand indeed the decisions, for example the inclusion of a shtank rather than other company related artifacts
such as miniature tractors.
The need for a further category to comprehensively describe users immersion, in addition to the modalities of in-
volvement, is in line with prior literature. Witmer and Singer (1998) originally argued that perceptions of presence (syn-
onymous with the term immersion as it is used within this paper) was related to both the active involvement of the user,
and the sensory envelopment obtained through the simulation (referred to as immersion by Witmer and Singer). How-
ever, the category of perceptual delity covers more than the sensory envelopment, as it also captures perceptions of the
verisimilitude of the environment in terms of the presence or absence of objects.
In this scenario the perceptual delity of the simulation was relatively unimportant to most of the participants, while
the coding of modalities of involvement suggests that all participants found that the way they were involved within the
scenario was signicant to their experience of immersion.
There is a categorisation issue in relation to the comments by participant #775 about the latency of the audio. The
communication issues caused by the high latency in the audio channel could be categorised as Shared as they relate to
other agents within the simulation, or as Perceptual Fidelity. Further modication of the categories would not strengthen
the contribution made by this article, however it is important to note this for future research in this area.
A signicant factor in users immersion within a MUVE simulation is how the user is involved within the experi-
ence. For many of the participants in this study, the experience of immersion could be described through reference to the
active direction of their attention and energy on a set of meaningfully related activities and events. The categorisation
scheme of six modalities of involvement provided an effective way of comprehensively describing these participants
experiences of immersion, and it was shown that involvement in the scenario could occur through any of the modalities.
The reections on immersive experiences of four participants required an additional category, relating to perceptual del-
ity, for comprehensive description. These participants placed importance on how accurately the simulation represented
the OH&S scenario, compared with a similar situation in everyday activity. Three of these participants reported feeling
only partially immersed in the simulation, tentatively suggesting that being highly immersed within a simulation leads
users to attend to and reect on their experience of meaningful involvement, rather than the perceptual delity of the
As data were gathered from participants reections of their perception of immersion, the analytic focus was on the
overall perception of immersion rather than the temporary uctuations in immersion that occur throughout participation
in the simulation. Further research could explore the micro-level of how different forms of involvement relate to immer-
sion throughout the scenario.
Certain characteristics emerged for involvement within the scenario that was related to high levels of immersion:
decision making through the lens of the role play identity, using background experience and considering the perspectives
of the other agents; embodied interactions with the other agents; controlling the avatar in relation to the environment
and other agents within the environment; participating within the narrative to progress through the simulation, and being
spatially oriented within the environment. On the other hand, characteristics of emotional involvement that were related
to low levels of immersion also emerged: lack of an emotional response; or an emotional response directed at something
outside the simulation narrative, rather than something within the virtual environment. Additionally, there were equivo-
cal ndings relating to the impact of the absence within the simulation of information, people and places that would
normally be present in a similar situation that occurred in everyday activity. Some users who encountered these absences
reported that it negatively impacted their immersion, while for other users it was apparently not an issue. This indicates
the existence of a mediating factor, possibly relating to the simulation design or alternatively the immersive tendencies of
the individual (Witmer and Singer, 1998).
The simulation used in this research study involved a role play aiming to elicit a response to an ethically nuanced
situation, and as would be expected the Tactical, Shared and Narrative modalities of involvement featured heavily in rela-
tion to the participants levels of immersion. Other educational approaches using MUVEs for simulations will probably
feature different patterns of user involvement. However, it is likely that there are some generalisable characteristics of
MUVE simulation designs that either encourage or inhibit immersive experiences. Focussing analysis on users modali-
ties of involvement is a promising approach to uncover these characteristics.
Agarwal, R., & Karahanna, E. (2000). Time ies when youre having fun: Cognitive absorption and beliefs about information
technology usage. MIS Quarterly, 24(4), 665-694.
Bouchard, S. p., St-Jacques, J., Robillard, G. v., & Renaud, P. (2008). Anxiety Increases the Feeling of Presence in Virtual Real-
ity. Presence: Teleoperators & Virtual Environments, 17(4), 376-391.
Calleja, G. (2007). Digital Game Involvement: A Conceptual Model. Games and culture, 2(3), 236-260.
Chase, W. G., & Simon, H. A. (1973). Perception in chess. Cognitive Psychology, 4(1), 55-81.
Cheng, K., & Cairns, P. A. (2005). Behaviour, realism and immersion in games. Paper presented at the CHI 05 extended ab-
stracts on Human factors in computing systems.
Csikszentmihalyi, M. (2002). Flow: The Classic Work on How to Achieve Happiness. Sydney, Australia: Random House Austra-
lia (Pty) Ltd.
de Freitas, S., & Neumann, T. (2009). The use of exploratory learning for supporting immersive learning in virtual environ-
ments. Computers & Education, 52, 343-352.
de Freitas, S., Rebolledo-Mendez, G., Liarokapis, F., Magoulas, G., & Poulovassilis, A. (2010). Learning as immersive experi-
ences: Using the four-dimensional framework for designing and evaluating immersive learning experiences in a virtual
world. British Journal of Educational Technology, 41(1), 69-85.
Dede, C. (2009). Immersive Interfaces for Engagement and Learning. Science, 323(5910), 66-69.
Gee, J. P. (2008). Video games and embodiment. Games and Culture, 3(3-4), 253-263.
Hedberg, J. G., Cram, A., Lumkin, K., & Eade, J. (2010). Design and Construction in Virtual Environments for Real-World
Learning. Paper presented at the IODL & ICEM 2010 Joint Conference and Media Days.
Hickey, D. T., Ingram-Goble, A. A., & Jameson, E. M. (2009). Designing Assessments and Assessing Designs in Virtual Educa-
tional Environments. Journal of Science Education and Technology, 18(2), 187-208.
Ketelhut, D. J., Nelson, B. C., Clarke, J., & Dede, C. (2010). A multi-user virtual environment for building and assessing higher
order inquiry skills in science. British Journal of Educational Technology, 41(1), 56-68.
Lee, K. M. (2004). Why presence occurs: evolutionary psychology, media equation, and presence. Presence: Teleoperators &
Virtual Environments, 13(4), 494-505.
Mayrath, M. C., Traphagan, T., Heikes, E. J., & Trivedi, A. (2009). Instructional design best practices for Second Life: a case
study from a college-level English course. Interactive Learning Environments.
McMahan, A. (2003). Immersion, Engagement, and Presence: A Method for Analyzing 3-D Video Games. In M. J. P. Wolf & B.
Perron (Eds.), The Video Game, Theory Reader (pp. 67-86). New York, NY: Routledge.
Slater, M., & Wilbur, S. (1997). A framework for immersive virtual environments (FIVE). Presence: Teleoperators & Virtual
Environments, 6(6), 603.
Witmer, B. G., & Singer, M. J. (1998). Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence: Tele-
operators & Virtual Environments, 7(3), 225-240.
Virtual Research Arena: Presenting Research in 3D Virtual Environments 121
Virtual Research Arena: Presenting Research in 3D Virtual Environments
MIKHAIL FOMINYKH AND EKATERINA PRASOLOVA-FRLAND
The use of 3D Collaborative Virtual Environments (CVEs) such as Second Life for educational purposes has been
constantly increasing during the recent years (de Freitas, Rebolledo-Mendez, Liarokapis, Magoulas, & Poulovassilis,
One of the reasons is the potential and possibility of such environments for supporting collaborative work with vari-
ous types of content, as discussed in several studies (Arreguin, 2007; Atkins, 2008; Hwang, Park, Cha, & Shin, 2008;
Nederveen, 2007). Most CVEs allow advanced content manipulation, uploading, creating and sharing 3D objects and
other media, such as text, graphics, sound and video. The term content can be understood more widely than media
objects, as we have discussed in (Prasolova-Frland, Fominykh, & Wyeld, 2010b). As it is noted in (Bessire, Ellis, &
Kellogg, 2009), content can be objects, places, activities or any valuable information or experience. CVEs allow creat-
ing complex interactive content and use it collaboratively for various purposes. 3D CVEs allow learning communities to
create content and leave traces of their activities that become part of the shared repertoire of the community through the
process of reication (Wenger, 1998).
Another important reason is an opportunity for participants to interact in a way that conveys a sense of presence
(Park, Hwang, & Choi, 2009), lacking in other media (Kelton, 2007). Users are represented by avatars and act in a shared
3D space that gives them awareness of each others actions. Communication is usually presented in the form of gestures,
text-based chat and in-voice chat and allows using CVEs for meetings, performances and role-playing (Sant, 2009).
These opportunities result in a number of benets for establishing and supporting learning communities (Bronack et al.,
2008) and in the potential for supporting cross-cultural understanding and collaboration (Wyeld & Prasolova-Frland,
A growing number of education- and research-intensive institutions have started using CVEs for presentations and
promotions, conferencing, sketching, training and other purposes. For example, promotion of the organization is one
of the primary reasons for nonprots establishing their presence in CVEs (Bettger, 2008). Conducting presentations in
CVEs is becoming more popular and common and although the technology has some limitations, the potential is appar-
ent and highlighted for example in (Yankelovich & Kaplan, 2008). Advanced universities are building full-scale, highly
realistic virtual campuses with various functionality (Prasolova-Frland, Sourin, & Sourina, 2006). Other organizations
that are using CVEs include research centers, libraries and museums. In the industry, many companies, such as IBM, Sun
and Cisco, are using 3D CVEs and investing in research and development of new environments.
The virtual world Second Life is one of the most successful CVEs at the moment (www.secondlife.com). It remains
one of the most stable, developed and populated, though it has certain limitations, as stressed for example in (Crowther &
Cox, 2008) and (Bowers, Ragas, & Neely, 2009).
In this paper, we investigate the possibilities of CVEs for learning communities and continue exploring how to sup-
port interconnected aspects of city life in an integral virtual environment, experimenting with the area of education and
research. In particular, we focus on visualizing and promoting research projects and engaging general public. We present
a qualitative analysis of data from an exploratory case study that involved students from a graduate cooperation technol-
ogy course, researchers and general public.
Despite the great opportunities of CVEs for visualization and the importance of presenting and promoting research,
there are few studies in this area (Djorgovski et al., 2010) and the body of knowledge on educational studies in CVEs has
not developed enough (Campbell & Jones, 2008). Therefore, the main goals of this paper are: rst to demonstrate that
CVEs can be successfully used for presenting and promoting research projects and guide education- and research-inten-
sive institutions in this area and second to present an improved framework of the Virtual Research Arena (VRA) that is
designed to integrate research community into society with its different aspects.
The paper is organized in four sections. In the next section we present the concept of Virtual Research Arena and
outline its initial use in Norwegian Science Fair and in the practical exercise of cooperation technology course. In the fol-
122 Fominykh and Prasolova-Frland
lowing section we discuss the results of the studies, showing the impact, value and possible application of the VRA and
the ideas behind it. In the last section, we outline directions for the future development of the Virtual Research Arena and
conclude the paper.
VISUALIZING RESEARCH PROJECTS
The work presented in this paper was conducted in the Virtual Campus of Norwegian University of Science and
Technology (NTNU) in Second Life. Despite the criticism we mentioned, Second Life platform was chosen as it is the
most common technology of choice for such educational projects, including other pre-existing Norwegian projects, such
as Second Norway. The campus and previous studies there were described in (Prasolova-Frland, Fominykh, & Wyeld,
2010a; Prasolova-Frland, et al., 2010b).
VIRTUAL RESEARCH ARENA OVERVIEW
In this paper, we present initial results of implementing and using Virtual Research Arena a framework for creating
awareness about educational and research activities, promoting cross-fertilization between different environments and
engaging the general public. The idea of the Virtual Research Arena emerged after we were invited to participate in an
annual scientic festival and present our work there. Our goal was to attract scientists who wanted to demonstrate their
work on the festival in a virtual mode and to build visualizations of their projects (Fig. 1).
Figure 1: Virtual Research Arena in the Virtual Campus of NTNU
In the previous research, we were exploring collaborative work on 3D content in a virtual campus and virtual city
context. The VRA contributes to the conceptual framework Universcity, in which we seek to integrate different aspects
of city life, such as culture, society, education and entertainment (Fominykh et al., 2010). We consider Universcity as an
integral/holistic organism, since in reality all these aspects are interconnected. The Universcity framework has 4 layers
that correspond to the aspects of city life. Each layer has its own specics and major infrastructure elements or facilities
(Fig. 2). These elements of the environment are designed using a tool called Creative Virtual Workshop or CVW that we
previously proposed and described in (Fominykh, et al., 2010). In the core of CVW lies collaboration around 3D content
that includes creating, sharing, exhibiting, annotating and other manipulations. CVW functions as a pattern for creating
infrastructure elements/facilities of the Universcity in such a way that they are connected to all the layers. The VRA
functionality was designed based on the basic ideas of CVW.
Figure 2: Virtual Research Arena in the Universcity context
In the following, we present the use of VRA in Norwegian Science Fair and in the cooperation technology student exer-
cise.
PRESENTING RESEARCH PROJECTS AT NORWEGIAN SCIENCE FAIR
Norwegian Science Fair in the city of Trondheim is a part of Norwegian Science Week, an annual festival. The goal
of this event is to present science projects to the general public. In Trondheim, which is recognized as a student city and
a technological capital, the festival is organized in pavilions on the central city square. At this science fair, a number of
researchers present their work in appealing yet simple ways.
In September 2010, a virtual science fair was erected in Second Life to mirror the one in reality. One of the major
city landmarks King Olav Tower, was reconstructed in the virtual science fair on the virtual central square, in the
same place where the fair was organized in reality, to create a familiar atmosphere for the local visitors. At the same time,
the Virtual Science Fair could be visited from around the globe through Second Life. In this way, it contributed to creat-
ing a meeting place for researchers, students and public. Moreover, while the physical pavilions at the fair were decon-
structed at the end of the event after two days, the virtual pavilions have been preserved and available for future use.
Furthermore, the Virtual Science Fair was presented at the fair in real life as one of the projects. The visitors in the
real life could come to the pavilion and immerse themselves into the virtual extension of the fair, exploring a number of
projects (Fig. 3). Such a mix of real and virtual is especially interesting and should be further improved, according to the
Figure 3: Real-life pavilion of the Virtual Science Fair in Second Life
The Virtual Science Fair in Second Life has 8 pavilions, each presenting a research project from NTNU and other
research environments. Most pavilions presented major ideas of projects with posters, slides, video clips and links to web
pages. Also, some interactive elements were used, such as teleports to other regions in Second Life, interactive models
and feedback boxes. The following virtual pavilions were presented:
Virtual Eidsvoll an educational region in Second Life for studying Norwegian history;
ArTe New Media Art research and dissemination activities at the intersection of art and technology;
Middelalderens Nidaros i virtuell virkelighet a reconstruction of medieval city in virtual reality;
EU project TARGET a 3D virtual serious game;
WAVE Women Academics in Virtual Environments;
Multi-lingual text annotator Typecraft a free online tool for language experts and anthropologists;
Digital stil a project advertizing social networking and mobile technologies;
vAcademia an educational virtual world.
The number of people who visited the real-life pavilion of the Virtual Science Fair shows that this topic is interesting
for the general public. An article dedicated to the Virtual Science Fair published in a national newspaper VG is also a
sign of interest.
VISUALIZING RESEARCH PROJECTS AT COOPERATION TECHNOLOGY COURSE
In the autumn of 2010, we conducted a practical exercise in a course TDT4245 cooperation technology in the Vir-
tual Campus of NTNU. This is a regular exercise and we applied most of the lessons learned from the earlier work, espe-
cially from the previous case study conducted in 2009 and described in (Prasolova-Frland, et al., 2010a).
The data in the recent study was gathered from three sources of evidence: direct observation of students activities
online, virtual artifacts, such as chat log and 3D constructions, and users feedback in a form of group essays. After the
study, the data were qualitatively analyzed.
The recent study was carried out with 25 students in 7 groups, 2-4 students in each, both regular NTNU students
(master and PhD level) and international students, participating in the NTNU international master program. The students
were asked to build a visualization representing any research project and present it at a joint session by role-playing. This
method is based on constructionism (Harel & Papert, 1991) an educational philosophy, which implies that learning
is more effective through the design and building of personally meaningful artifacts than consuming information alone
(Bessire, et al., 2009; Harel & Papert, 1991). Constructionism is related to the social constructivist approach (Vygotsky,
1978), where the main idea is that learners co-construct their environment and understanding together with their peers.
We also applied role-playing, which is a widely used and effective learning and teaching method. It implies an active be-
havior in accordance with a specic role (Craciun, 2010; McSharry & Jones, 2000).
Prior to the competition, the students had a tutorial on Second Life in a classroom (for those who were located in
Trondheim). Following this, they came online into Second Life, presented their project proposals, identied building
spots and received additional training. The total building period was about 1 month. During this time we were available
for questions and assistance both in a real-life computer-room and online in Second Life. Assessment was based on par-
ticipation in the construction effort and on a group essay where the students reected on their experience.
This year the exercise was conducted in conjunction with an International Summer School on Collaborative Tech-
nologies, Serious Games and Educational Visualizations, organized by the EU TARGET project (http://www.reachyour-
target.org/). The summer school provided 2 virtual events: a seminar Using Virtual Worlds to Improve Business Presen-
tation Skills by Judith Molka Danielsen and a seminar TARGET EEU A step toward new e-learning technologies,
by Albena Antonova. Each of the seminars attracted 20-30 participants from different countries (China, United States,
UK, Russia, Bulgaria etc.). TARGET announced a prize for the best student project, which was later divided between
two groups that got an equal amount of votes. Almost nished student constructions were available in the Virtual Campus
of NTNU during the Norwegian Science Week and demonstrated on the Virtual Science Fair.
During the joint session the students presented their projects in the form of role-plays, evaluated each others con-
structions and received feedbacks from the visitors. The following provides an overview of the session.
Group 1 created a programming history museum. The group constructed 4 oating platforms symbolizing eras of
programming and presenting important concepts. Each platform has interactive schemes or challenging quests (for ex-
ample reconstruction of a motherboard and if-then loops) as well as slides explaining the topic. For a role-play presenta-
tion group members appeared as robot-like avatars, guiding the visitors through the museum and explaining the central
concepts.
Group 2 visualized the effect of Kung-Fu training on health. The group created a very realistic and authentic Chi-
nese-inspired environment that impressed the public, including Chinese visitors. The building was decorated with tradi-
tional Chinese furniture, an animated statue of Buddha, an authentic replace, a gong and other elements. Posters and
slides on the walls provided information about the martial art and its inuence on the human health. The leader of the
group has an authentic avatar.
Group 3 created and presented a research project BP Solar Energy the biggest solar skin in Norway. The solar
skin is located at the NTNU campus south wall and provides an additional energy supply. The students constructed a
piece of wall with an interactive virtual solar skin that could be switched on and off, emanating light and thus visualizing
conversion of the solar energy. The construction also included posters and slides providing more information about the
project and presenting group members. The students also prepared a scenario and performed a role play highlighting the
importance of solar energy and presenting the project in an informal yet informative way (Fig. 4).
Groups 4 presented the work of the designer Enzo Mari called Autoprogettazione and who was known by using
simple pieces of wood for constructing furniture. The group built several pieces of furniture and placed them in a work-
shop. The construction also included a presentation area showing a video clip and posters, providing additional informa-
tion. During the performance, one of the group members impersonated Enzo Mari.
Group 5 tried to visualize the idea of proposing prototypes and selecting the best solution. The construction included
a room with a set of random interactive objects, and a half-working voting system. The presentation was done by simply
naming the prototypes and soon became a discussion on the CVE technology.
Group 6 created a visualization of a concurrent design methodology by constructing a Concurrent Design Facil-
ity. The students sought to re-create real-life design facilities and built a room with a few tables and large screens on
the walls for different expert groups. An additional screen displayed the central aspects of the presented methodology.
During the presentation the group members played the key roles of facilitator, session secretary and customer, while the
public was invited to be members of the expert groups. In such a manner, a demonstrative session How to make a good
project presentation in a virtual environment was played.
Group 7 visualized a project called ArTeNTNU that aimed at increasing knowledge about the interdisciplinary in-
tersection between digital art and software technology. The students built a simple 2-oor building, lling it with posters,
slides and web links with information about the artifacts created within the projects.
One of the TARGET seminars was held between the presentations. Most of the students and a number of interna-
tional visitors participated in the event.
During the discussion in the end of the session, the central question debated was usefulness of 3D virtual environ-
ments for presenting projects. A group of students argued that using tools like Second Life requires too much time and
effort, even though the presentation is more vivid and appealing. Another group was less critical and proposed that there
is a number of cases where using a 3D environment is feasible and the effort spent is rewarding. Analyzing the chat log
showed that the students learnt a lot about advantages and limitations of using CVE technology for collaboration and
moreover they understand more clearly the roles of other tools and technologies.
Figure 4. BP Solar Energy project: role-play project presentation
After the sessions, the students had 2 weeks for reecting on their activities in group essays. We provided a guide-
line for this task in the form of a set of points to discuss. According to the guideline, the students had to talk over po-
tential use of their constructions, a number of aspects related to collaborative work and learning and other topics. In this
paper, however, we explore the one related to the VRA design.
Evaluating general usefulness and the potential of the Virtual Research Arena in group essays, the students provided
different opinions. Positive feedbacks were related to conceptual opportunities of the VRA, while the criticism was most-
ly focused on some imperfections of the current design and limitations of the technology. The potential of the VRA was
mostly seen in promoting presented research environments by creating a socializing and gathering place around project
presentations. Increased awareness among researchers, students, university departments, research groups, institutions and
general public was emphasized as a way for promoting collaboration and an important opportunity. In the current VRA
design, the students appreciated appealing reconstructions of real-life places.
[Essay citations]:
VRA is a cost efcient, social place to meet researcher colleagues, and discuss with them in a natural setting.
Its easy to create a small interesting taste of a topic in VRA, and then link further to external information on the web.
We like that there are some physical and design similarities with the real Norwegian Research Week event.
Many students expressed their appreciation for the global nature of the VRA and potential for supporting collaboration
between researchers, students and general public.
People from other cities can take a look of what NTNU and Trondheim has to offer. VRA can be a source to trigger
the willingness to visit Trondheim and NTNU.
This is an extraordinary way to promote collaborations among different projects. Using this approach new cross
boundary projects may come out.
Visitors/Students from other places can also nd it useful to discover the inner working of the university, visit some
of the buildings and know the activities developed in the university by students and teachers.
The negative impression was based on a general frustration about the early stage of the VRA development. Some were
disappointed that proposed functionality is not yet implemented.
VRA does not provide any support for research activities, but it does provide the users with an interactive experience
The problems in this kind of technology are time required to make a presentation and a lot of system resources to
use it smoothly.
The quality of objects is too undeveloped to fairly illustrate all types of research projects.
Several technical comments were related to navigation problems, overage of objects and complexity for inexperienced
In this section, we discuss how the development of the Virtual Research Arena changed the students experience. Further-
more, we discuss the context the VRA and summarize the major implications for presenting research projects in CVEs.
VIRTUAL RESEARCH ARENA IMPACT
Analyzing the studies presented in the paper, we noticed a change in how the participants reected on their experi-
ence. In comparison to the previous study in the undeveloped campus (Prasolova-Frland, et al., 2010a), we observed
several trends that were related to the development of the VRA and improvement of the study settings.
According to the observations and feedbacks, the students were inspired by the other constructions in the Virtual
Campus, intentionally or not. They could explore existing project visualizations both from the previous year cooperation
technology course and the Virtual Science Fair. There was no plagiarism since the topics were different, but the students
could grasp some interesting and effective solutions and estimate approximate effort required. Reconstruction of several
real places in the Virtual Campus helped the students to adapt to the environment and feel comfortable, as almost all the
groups noted in the essays. According to the feedbacks, attracting attention to the neighborhood of the Second Norway
region in Second Life encouraged some of the students to explore the region and expect visitors to their own projects.
Within TARGET summer school 2 virtual seminars were organized, which provided the students with an outlook of
the latest trends in the area of CVEs. On the seminars, some recent and current projects were presented, from which the
students could learn more about the practical use of the technology. Besides that, the students could experience how the
virtual lecturing works in general, its benets and limitations. Moreover, the summer school attracted some international
participants, which resulted in a bigger and more independent audience. Another difference of the study this year was ini-
tiation of a prize for the best project.
The introduction of role-playing as a presentation method had also inuenced the students experience, according to
their feedbacks. First of all, calling this activity role-playing placed more emphasis and improved the general attitude
of the students. Although not all the groups did really play roles, the overall quality of the presentations increased. As
distinct from last years presentation, this time none of the groups reported the problem identifying who is presenting.
The students prepared scenarios, some wore authentic avatars and many used voice chat in addition to the text. Moreover,
the audience was expecting a play and therefore more focused. In many plays, the presenters engaged the audience into
the play, which was appreciated both by the visitors and the students.
The VRA helped the students to extend their understanding of cooperation with the CVE technology. Discussing the
possibilities and the future of the CVEs in essays, most of the groups mentioned their potential for supporting social net-
works and collaboration among various groups of people, institutions and countries. Describing scenarios of use for their
own constructions, the students often considered them as a part of the Virtual Science Fair that is closely connected to the
university and local community.
The study demonstrated the range of possible topics that can be visualized and also the variety of presentation meth-
ods. The topics included research projects or concepts from both technical disciplines and humanities. A number of dif-
ferent metaphors were used, including a museum, a gallery, a meeting room and a workshop. Construction presentations
revealed the possibilities for immersing visitors into the project environment or process, live discussions and demonstra-
Our observation of the students work and their feedbacks can be summarized as a set of recommendations for pre-
senting research projects in CVEs. The following recommendations are developed for teachers, instructors and techni-
cians working with the CVE technology.
Demonstrate the possibilities of the technology, including interactive elements, various types of content and
ways of presenting information.
Provide tutorials introducing technology basics and building resources for composing structures from ready-
to use blocks.
Involve presenters and visitors from different social groups, such as researchers, students and general public.
Support activities in a virtual environment with real-life events or places to attract more visitors for both
virtual and real-life environments.
CONCEPTUALIZING VIRTUAL RESEARCH ARENA
The results of the presented study contribute to 3 major areas that are connected by the VRA: rst, collaborative
work on 3D content; second, virtual campus as an environment for learning, researching and socializing; and third, virtu-
al city as an environment integrating different aspects of city life. In the following, we attempt to form a concept of VRA
out of our observations, experience and case study data.
Figure 5. Virtual Research Arena context
Collaborative work on 3D content is the major activity that Virtual Research Arena supports. Tools and features that
provide this support within the VRA are designed based on the basic ideas of CVW. The VRA has a virtual workplace
equipped with tutorials and tools, providing assistance for control and navigation, communication and work with content.
The workplace is linked to a library with ready-to-use 3D objects, textures, scripts and other resources. To provide sup-
port for sharing and presenting content, there is a virtual stage (under development), equipped with corresponding facili-
ties, such as a slide-show screen and a place for presenting 3D constructions. The stage is surrounded by a virtual gallery
(under development), which contains and exhibits constructions.
Virtual Campus framework was elaborated based on the results of the presented study. The Virtual Campus of
NTNU was used as a venue for the study. It provided appealing atmosphere, tools and facilities for seminars, meet-
ings and discussions. Besides that, the campus contains crystallized activities or traces (Wenger, 1998) from past events,
creating a cultural component of the environment and a base for further development. In the virtual campus context, the
VRA is a place, where students and researchers can try out their ideas, express themselves, create visualizations and ex-
hibit them.
Universcity framework was improved based on the results of the study. In the Universcity context, Virtual Re-
search Arena and Virtual Campus are infrastructure elements. They represent the layer of education and research, which
is considered for supporting educational/research activities and networks. At the same time, the VRA is connected to
all other layers: cultural, social and entertainment. The VRA contributes to the cultural layer by attracting international
visitors in the virtual environment and after that perhaps in real life as well. In the virtual city, the research arena is em-
bedded into the architectural/cultural environment and replicates a real place. The VRA contributes to the social layer by
connecting research environment and the general public. It allows scientists to present their work to the public, facilitates
communication and creates awareness about the local and international research. For many people, visiting events within
the VRA is an entertainment, since one of the main goals of the arena is presenting scientic models and projects in an
interesting and engaging way. The results of the study can be generalized and used for designing other infrastructure ele-
ments in all the layers.
CONCLUSIONS AND FUTURE WORK
In this paper we present the results of a case study conducted to evaluate the Virtual Research Arena framework.
Conducting events both in the virtual environment and in reality, we collected empirical data and feedbacks from partici-
pants, including university students, researchers and general public. The results show the potential and possibilities of the
VRA for supporting collaborative work with 3D content in the research area. In addition, the experience with developing
and studying the VRA contributed to the Virtual Campus and Virtual City frameworks.
Future work will include more studies in the area of collaborative work and learning in 3D CVEs and further devel-
opment of the Virtual Research Arena framework. We are planning to use the arena more widely in the city life and con-
necting it more clearly to other Universcity layers. Furthermore, we have in focus strengthening the link between the
virtual environment and reality and attracting more participants from various society groups.
Arreguin, C. (2007). Reports from the Field: Second Life Community Convention 2007 Education Track Summary. New York:
Global Kids.
Atkins, C. (2008). Virtual Experience: Observations on Second Life. Paper presented at the 1st International Conference on
Computer-Mediated Social Networking (ICCMSN), Dunedin, New Zealand, 717.
Bessire, K., Ellis, J. B., & Kellogg, W. A. (2009). Acquiring a professional second life: problems and prospects for the use
of virtual worlds in business. Paper presented at the 27th CHI International Conference extended abstracts on Human fac-
tors in computing systems, Boston, MA, USA, 2883-2898.
Bettger, S. (2008). Nonprots in Second Life: Promoting Causes Inside Second Life for Real World Impact. Retrieved from
http://npsl.wiki.techsoup.org/le/view/Non+Prots+and+Second+Life+Report+1008.pdf
Bowers, K. W., Ragas, M. W., & Neely, J. C. (2009). Assessing the Value of Virtual Worlds for Post-Secondary Instructors: A
Survey of Innovators, Early Adopters and the Early Majority in Second Life. International Journal of Humanities and So-
cial Sciences, 3(1), 40-50.
Bronack, S., Sanders, R., Cheney, A., Riedl, R., Tashner, J., & Matzen, N. (2008). Presence Pedagogy: Teaching and Learning in
a 3D Virtual Immersive World. International Journal of Teaching and Learning in Higher Education, 20(1), 59-69.
Campbell, C., & Jones, S. (2008). Using Second Life as an Educational Instructional Tool with Pre-service Education Students:
A Work in Progress. Paper presented at the 20th World Conference on Educational Multimedia, Hypermedia and Telecom-
munications (Ed-Media), Vienna, Austria, 3638-3642.
Craciun, D. (2010). Role playing as a Creative Method in Science Education. Journal of Science and Arts, 1(12), 175-182.
Crowther, P. S., & Cox, R. J. (2008). Building Content in Second Life Issues Facing Content Creators and Residents. Paper
presented at the 1st International Conference on Computer-Mediated Social Networking (ICCMSN), Dunedin, New Zea-
land, 28-34.
de Freitas, S., Rebolledo-Mendez, G., Liarokapis, F., Magoulas, G., & Poulovassilis, A. (2009). Developing an Evaluation
Methodology for Immersive Learning Experiences in a Virtual World. Paper presented at the 1st International Conference in
Games and Virtual Worlds for Serious Applications (VS-GAMES), Coventry, UK, 43 - 50.
Djorgovski, S. G., Hut, P., McMillan, S., Vesperini, E., Knop, R., Farr, W., et al. (2010). Exploring the Use of Virtual Worlds as
a Scientic Research Platform: The Meta-Institute for Computational Astrophysics (MICA). In F. Lehmann-Grube & J. Sa-
blatnig (Eds.), Facets of Virtual Environments (Vol. 33, pp. 29-43): Springer Berlin Heidelberg.
Fominykh, M., Prasolova-Frland, E., Morozov, M., Gerasimov, A., Bellotti, F., Gloria, A. D., et al. (2010). Universcity: To-
wards a Holistic Approach to Educational Virtual City Design. Paper presented at the 16th International Conference on
Virtual Systems and Multimedia (VSMM), Seoul, Korea, 371-374.
Harel, I., & Papert, S. (1991). Situating Constructionism Constructionism: Ablex Publishing Corporation, USA.
Hwang, J., Park, H., Cha, J., & Shin, B. (2008). Effects of Object Building Activities in Second Life on Players Spatial Reason-
ing. Paper presented at the 2nd International Conference on Digital Game and Intelligent Toy Enhanced Learning, Banff,
Canada, 62-69.
Kelton, A. J. (2007). Second Life: Reaching into the Virtual World for Real-World Learning. ECAR Research Bulletin, 2007(17).
McSharry, G., & Jones, S. (2000). Role-Play in Science Teaching and Learning. School Science Review, 82(298), 73-82.
Nederveen, S. v. (2007). Collaborative Design in Second Life. Paper presented at the 2nd International Conference World of
Construction Project Management, The Netherlands, 1-6.
Park, S., Hwang, H. S., & Choi, M. (2009). The Experience of Presence in 3D Web Environment: An Analysis of Korean Second
Life. Paper presented at the 13th International Conference on Human-Computer Interaction. Part IV: Interacting in Various
Application Domains, San Diego, CA, USA, 387-395.
Prasolova-Frland, E., Fominykh, M., & Wyeld, T. G. (2010a). Virtual Campus of NTNU as a place for 3D Educational Visual-
izations. Paper presented at the 1st Global Conference on Learning and Technology (Global Learn Asia Pacic), Penang,
Malaysia, 3593-3600.
Prasolova-Frland, E., Fominykh, M., & Wyeld, T. G. (2010b). Working on Educational Content in 3D Collaborative Virtual
Environments: Challenges and Implications. Paper presented at the 13th International Conference on Computers and Ad-
vanced Technologies in Education (CATE), Maui, Hawaii, USA, 183-190.
Prasolova-Frland, E., Sourin, A., & Sourina, O. (2006). Cybercampuses: design issues and future directions. Visual Computer,
22(12), 1015-1028.
Sant, T. (2009). Performance in Second Life: some possibilities for learning and teaching. In J. Molka-Danielsen & M.
Deutschmann (Eds.), Learning and Teaching in the Virtual World of Second Life (pp. 145-166). Trondheim: Tapir Academic
Vygotsky, L. S. (1978). Mind in society: the development of higher psychological processes. Cambridge, MA: Harvard Univer-
sity Press.
Wenger, E. (1998). Communities of Practice: Learning, Meaning, and Identity. New York/Cambridge: Cambridge University
Wyeld, T. G., & Prasolova-Frland, E. (2006). Online 3D CVE performance of T.S. Elliots coctail party: an example of virtual
stage. Paper presented at the 5th International Conference on Web-Based Education (WBE), Puerto Vallarta, Mexico, 31-
Yankelovich, N., & Kaplan, J. (2008). Improving Virtual World Presentations. Paper presented at the Computer Supported Col-
laborative Work (CSCW), Virtual Worlds, Collaboration, and Workplace Productivity Workshop, San Diego, CA, USA.
The research presented in this paper is nanced by Program for Learning with ICT and Department of Computer
and Information Science, NTNU and EU FP7 TARGET project (grant agreement N 231717). The authors wish to thank
Rune Aunien, Albena Antonova, Judith Molka Danielsen as well as all the students and researchers participating in the
The Curriculum as an Ontology - A Human Centered Visualization Approach 131
The Curriculum as an Ontology - A Human Centered Visualization Approach
SIMONE KRIGLSTEIN AND RENATE MOTSCHNIG-PITRIK
Although curricula are often only available in textual form (e.g., a detailed course description) graphical representa-
tions can provide a clear understanding of the curriculum structure along with various dependencies and relationships be-
tween courses and modules. The results of the study from Kriglstein (2010) conrm that students and lecturers consider
visual representations of curricula to be easier to understand and memorize than textual representations. Furthermore,
curricula visualizations allow a faster orientation which helps students to plan their study and lecturers in order to orga-
nize their courses.
In various communities and applications, ontologies have been introduced to describe concepts and their relation-
ships in a human-readable way which can be used as a skeletal foundation for a knowledge base. In particular, the visu-
alization of curricula via ontologies is a good possibility to represent the structure and the dependencies between courses
clearly and allow one to share, exchange, reuse, analyze and extend curricula. The mapping of the clear modular struc-
ture of curricula to an ontology such as OWL (W3C, 2004) seems natural in so far as modules of the curricula can be vi-
sualized as classes, courses can be dened as instances, and the dependencies between courses can be described as object
properties. Nevertheless, it is a real challenge for the visualization to represent the ontology in a way that is genuinely
useable for its users! Therefore the authors designed and studied a variant of the human centered design process to apply
it in the development of an ontology visualization tool to represent curricula as OWL Lite ontologies.
Within the eld of software development approaches, human centered design is a well-known strategy which in-
volves users already in the early stages of the development process to get a better understanding about their needs, tasks
and expectations. The process starts by identifying the needs of students and lecturers as well as which expectations they
have in regard to curricula visualization. Based on this information the requirements can be dened. The next step is to
analyze if an existing visualization tool meets the dened requirements. If no tool is available, the development of a visu-
alization tool starts. Finally, the tool is evaluated to verify if it meets the dened requirements successfully.
Based on these observations, this paper presents the core parts of a case study on a human centered approach which
shows the design process of a visualization tool to represent curriculum ontologies. The case study is a result of the doc-
toral thesis from Kriglstein (2011) and deals with the visualization of a modular computer science bachelor curriculum.
There exist several related research works which present different visualization approaches to make the structure of
curricula for students and lecturers more transparent. For example, Sommaruga and Catenazzi (2007) present a visualiza-
tion approach which visualizes a bachelor curriculum in 3D. The modules of the bachelor curriculum are presented as
blocks and specic attributes of the module dene the size of the corresponding block. For example, the height of the
block reects the number of credits. Although the results of the evaluation showed that the learning time was very short,
the 3D visualization was unfamiliar to users. Another approach is presented by Kabicher et al. (2008), which combines
the textural description of the curricula with a node-link representation to show the curriculum structure and module
dependencies. Modules are visualized as rectangles and the relationships between modules are visualized as arrows.
However, it was developed for lecturers only to help them to systematically describe and to coordinate their modules and
courses and hence is not tuned to students needs.
This paper is structured as follows. In Section 2, the bachelor curriculum of computer science is described. In Sec-
tion 3, we present the user analysis and requirements of students and lectures. Finally, the analysis of existing ontology
visualization tools and a short description of the newly developed tool are discussed in Section 4.
132 Kriglstein and Motschnig-Pitrik
STRUCTURE OF THE CURRICULUM ONTOLOGY
This section presents a short overview on the specication of the bachelor curriculum of computer science as an on-
tology. This ontology example can be helpful for further curriculum ontologies and is presented in detail in (Kriglstein,
2011). The curricula of the University of Vienna are based on the requirements of the Bologna Process (Commission,
2010) and are dened by a clear modular structure. Each curriculum contains modules and courses. A module species
a set of courses and can contain further modules (submodules). Based on the clear hierarchical structure of the modular
structure it is easy to map the curriculum to an OWL Lite ontology. Classes of the ontology represent modules and the
courses of each module are dened as instances of the corresponding class. For example, the diagram in Figure 2 shows
the module WGI-Scientic Foundations of Computer Science with its two courses Basics of Scientic Writing and Formal
Basics of Computer Science.
Figure 2: The diagram presents the module WGI-Scientic Foundations of Computer Science with its courses Basics of
Scientic Writing and Formal Basics of Computer Science.
The subClassOf relationships of the ontology describe the hierarchical relationships between modules and their sub-
modules. The diagram in Figure 3 shows as an example that the module PA-General Basics with its submodules HCI-
Human-Computer-Interaction and Psychology, MOD-Basics of Modeling, PM-Project Management and WGI-Scientic
Foundations of Computer Science.
Figure 3: The diagram shows the module PA-General Basics and its submodules.
Additionally to the hierarchical relationships it is also necessary to represent the dependencies between the modules/
courses. There exist two types of connections which are dened as object properties in our curriculum ontology:
Formal dependencies: dene the prerequisites of a course.
Content dependencies: show which courses are helpful or interesting for other courses.
Datatype properties of the ontology dene additional information about the course (e.g., course type, ECTS points,
term, short description of the course). For example, the diagram in Figure 4 shows that the course Modeling is a formal
prerequisite for the course Database Systems. Whereas Database Systems is a formal prerequisite for the course Software
Engineering. The content of the course Database Systems depends on the course Software Architectures and Web-Tech-
nologies. Furthermore, the diagram represents four datatype properties (course type, ECTS points, term and content) for
the course Database Systems.
Figure 4: The diagram shows the formal prerequisite for and the content depends on relationships as object properties
and the course information (e.g., ECTS points) as datatype properties for the course Database Systems.
USER ANALYSIS AND REQUIREMENTS
User Analysis
For successful curriculum visualizations it is important to nd out what different user groups in our case students
and lecturers expect from the visualization and what kind of support they are looking for. In order to nd this out we
analyzed the results of the study by Kriglstein (2010), who asked 137 students and 19 lecturers to get more information
about how students and lecturers inspected curricula and what they expected from a graphical representation of their cur-
riculum. Moreover, we asked 41 students (school) who wanted to enroll at a university to nd out what they expected
from a graphical representation of a curriculum. In a nutshell, the responses of students (university and school) and lec-
turers showed that, in the rst place, they wanted to get a better overview about the whole course offer and a better un-
derstanding of the structure and the dependencies between modules and courses (Kriglstein 2011).
Based on these ndings, we created user proles for a student from a university, for a rst-year student, for a student
(school) who wants to enroll at a university, for lecturers and for a rst-year lecturer. Although user proles are a good
way to get brief descriptions of users characteristics (e.g., age ranges, experiences, employments, attitudes) (Courage
& Baxter, 2005), they do not include much detail information about the needs of the different user groups. Therefore, we
decided to design personas to get a better picture about the different user groups. Cooper describes personas as a pre-
cise description of users and what he wishes to accomplish (Cooper, 2004, p. 123). Personas are ctional persons, who
represent the real users and help to see for which persons the visualization is designed. They help to identify the scope of
the visualization as well as possible design problems during the design process (Cooper, 2004; Courage & Baxter, 2005).
For an effective usage, a right choice of personas is important, because if universal personas are developed, they are too
unspecic and less believable (Courage & Baxter, 2005; Stone, Jarrett, Woodroffe & Minocha, 2005). Therefore we used
a plot diagram to identify the important personas (see Figure 5).
Figure 5: Plot diagram for the identication of personas.
Table 1: Personas for student, rst-year student and lecturer (based on (Kriglstein, 2011)).
Primary Personas Secondary Personas
Advanced student Lecturer First-year student
Name: Tom Name: Sue Name: Liz
Age: 22 Age: 35 Age: 18
Term: 4 Teaching course:
Term: 1
Expectation: He wants to get
an overview about
the whole course
offer and to see the
formal dependencies
between the courses.
Expectation: She wants to see
the dependen-
cies between her
course and the
other courses.
Expectation: She wants to
see the courses
which are
the rst term.
Scenario: Tom plans to change
his specialization,
because he is more
interesting in medical
informatics. There-
fore, he wants to
inform which courses
are the same for
all specializations
and which courses
are especially only
necessary for medi-
cal informatics.
Scenario: Sue wants to
use modeling
languages for the
specication of
several database
examples and
therefore she
wants to check
which previous
about modeling
language the
students have.
Scenario: She wishes
to have ad-
ditional graphical
support to see
the structure of
in combination
with the textual
The dots present the user proles and they are plotted as function of ontology and curriculum expertise. The axis
which represents the ontology knowledge denes how familiar the user is with ontologies and the axis which represents
the curriculum knowledge describes how familiar the user is with the curriculum. For the user proles First-year student
and Student (school) the structure of the curriculum and the main components of ontologies are unfamiliar. Therefore,
both user proles are grouped to the persona First-year student. In contrast to the persona First-year student, the user
prole Student denes the persona Advanced student. Advanced students know the structure of the curriculum better and
have basic knowledge about ontologies, because it is a topic in a course of the bachelor curriculum of computer science.
The user proles Lecturer and First-year lecturer describe the persona Lecturer, because both user proles have similar
knowledge about the curriculum and as the ndings of the study from Kriglstein (2010) show, lecturers for the bachelor
curriculum of computer science have at least basic knowledge about ontologies.
The personas Advanced student and Lecturer present the primary personas and the persona First-year student denes
the secondary persona, because they were neither familiar with ontologies nor with such visualizations and therefore a
graphical representation of ontologies alone would be insufcient. The assumption that ontology visualizations can only
be used as additional support for students who presented the persona First-year student was also conrmed by the study
from Nina Forst (2010). In her work Forst evaluated two different visualization tools which represented the bachelor cur-
riculum of computer science with students who represent the persona First-year student.
Furthermore, the personas were extended with scenarios to show in which context users would use the visualization
and to get an overview about possible tasks. The described scenarios concentrate primarily on nding specic courses/
modules and identifying different dependencies between courses/modules. Table 1 presents examples of personas (stu-
dent, rst-year student and lecturer) which are based on the described personas in Kriglstein (2011).
Requirements are measurable benchmarks for the evaluation, which specify design goals and primarily describe
what a visualization should provide (Maguire & Bevan, 2002; Stone et al., 2005). User requirements concentrate on the
needs of students and lecturers in regard to curriculum ontology visualizations. Based on the ndings of the user analysis
(students and lecturers), the following user requirements were specied (Kriglstein, 2011):
Overview about the structure: The visualization should give a clear overview about the curricula so that
students and lecturers can see the whole course offer. The structure of the curriculum with its modules and the
differentiation between compulsory modules and modules of each specialization should be clearly visible.
Relationships: It is essential that curriculum visualizations should provide a clear representation of the different
relationships between modules and courses. The ndings of the user study from Kriglstein (2010) about students
and lecturers expectations on graphical representations of curricula showed that they ranked the overview about
the curriculum structure with their dependencies as very important.
Detail information: Additionally to the general view about the structure, it is also necessary that users get more
detail information about each course quickly. Although the results of the user study from Kriglstein (2010)
showed that students and lecturers stated that detail information about courses is important, they ranked the
priority lower than the overview with their interconnections.
Planning/decision-making: The visualized information should support users in their planning and help them to
make good decisions. For example, the visualization can help them to nd out which courses are recommended
for a specic term.
Easy to learn and to understand: The visualization should present the information in an understandable way
and students as well as lecturers should interpret the data correctly. Furthermore, the visualization should be easy
to learn, to memorize and to use. Otherwise it has for the students and lecturers no additional benet to the textual
document and therefore the acceptance to use such curricula visualizations is low.
To meet the above-mentioned requirements, it is important to consider the following elements for the ontology visu-
alization (Kriglstein, 2011):
Classes: Classes represent the modules of the curriculum, which reect the basic framework of the curriculum
ontology. Therefore, it is necessary that students and lecturers can have fast access to the classes and can get a
good overview. For a clear representation of classes it is important that they are visually distinguishable from
instances.
Instances: Instances represent the courses in the ontology and therefore they frequently play an important role for
students and lecturers. A clear visibility of instances along with readable names is necessary to nd them quickly.
subClassOf relationships: A hierarchical structure of classes is particularly necessary to see modules with
their submodules. It helps students and lecturers to get a better orientation and to nd modules (classes) with
their courses (instances) more easily to support a quick scan with less cognitive effort. Furthermore, a clear
visualization of subClassOf relationships allows students and lecturers to compare modules with their submodules
(e.g., they can see if the modules have more or less submodules, which can be important for their planning).
Object properties: Additionally to the subClassOf relationships, object properties dene different relationship
types between modules/courses (e.g., formal dependencies or content dependencies). It is important that the
different object properties are visually distinguishable, so that students and lecturers can directly see the different
dependencies in the visualization.
Datatype properties: Datatype properties relate courses (instances) to datatypes to give students and lecturers
the possibility to get additional information to each course (e.g., ECTS points). Therefore, it is necessary to nd a
design solution to represent the datatype properties directly of the corresponding course.
ANALYSIS OF ONTOLOGY VISUALIZATION TOOLS
Different ontology visualization tools were analyzed in regard to the curriculum ontology. The tools were repre-
sentatives of different visualization approaches OntoViz (Sintek, 2007) and TGVizTab (Alani, 2003) for the node-link
approach as well as Jambalaya (nested graph view) (Storey et al., 2010) and CropCircles (Wang & Parsia, 2006) for the
container approach. In general, node-link approaches present objects as nodes and the relationships between them as
edges. In contrast to the node-link approach, container approaches concentrate primarily on the representation of the
hierarchical structure and present child nodes nested inside their parent node. Figure 6 shows how the different visualiza-
tions tools represent the curriculum ontology.
CropCircles presents the hierarchical structure of the ontology. The circles represent the modules (classes) and every
submodule (subclass) is nested inside its module. However, there exists no information about the courses (instances), de-
pendencies (object properties) or general information of the course (datatype properties).
Jambalaya (nested graph view) visualizes the modules (class) as yellow rectangle and courses as purple rectangles.
The submodules (subclasses) and courses (instances) are nested inside their corresponding module. Users have the possi-
bility of zooming to navigate to the detail information of a course (datatype properties and object properties) (Kriglstein,
2008). However, if users zoomed in a module then only the submodules and courses of this module are visible. Hence,
they cannot compare different modules with their submodules and courses at the same time (Kriglstein, 2011). Further-
more the labels of classes or instances are often difcult to read, because they are sometimes too small or they overlap
(Kriglstein, 2008). The nested graph is combined with a node-link representation to additionally visualize the object
properties. Each relationship type has its own color and the connections between modules are summarized in a red arrow,
which shows the direction of the relationship and the thickness of the arrow varies in regard to the number of bundled
connections. However, the node-link representation visualizes only the connection between modules (classes) and not
between courses (instances). Furthermore, the visualization can easily overcrowd depending on the number of presented
relationships and the visible connections constrain further interaction in the visualization. For example, connections over-
lap classes, and therefore it is difcult to select the class without clicking the connection before (Kriglstein, 2008). The
red arrows only show a predened number of individual connections in a tooltip and users cannot jump directly to the
courses (Kriglstein, 2011).
OntoViz uses a node-link approach which visualizes modules (classes) and courses (instances) as rectangles and
relationships (subClassOf relationships and object properties) as edges. Additionally, OntoViz uses the color black for
modules and the color red for courses. The subClassOf relationships are colored in black and the object properties are
blue, thereby OntoViz does not make further differentiations between the different types of object properties (Kriglstein,
2008). Furthermore, users have the possibility to view additional information about a course (datatype properties). The
interface of OntoViz is very technical and therefore abbreviations for the drawing settings are used which are difcult to
understand for students and lecturers. However, the settings are very important to choose which information is shown in
the graph.
TGVizTab visualizes the ontology as node-link representation. Modules (classes) and courses (instances) are visual-
ized as nodes and courses are colored purple for a better differentiation. Relationships (object properties and subClassOf
relationships) between modules and courses are displayed as edges (Kriglstein, 2008). In contrast to OntoViz, TGVizTab
does not make visual differentiations between object properties and subClassOf relationships. Users have to move the
mouse over the respective edges of the relationship to see the corresponding labels. Additional information for a course
(datatype properties) is not shown.
Figure 6: Different visualization tools which present the curriculum ontology.
Summarizing, the ndings of the comparison analysis showed that no tool existed that would meet all the require-
ments described in the previous Section. Consequently, different existing tools would need to be combined in order to
properly support students and lecturers tasks. Since using multiple tools doesnt appear to be practically viable we de-
cided to develop a new tool called Knoocks (Knowledge Blocks).
DESCRIPTION OF KNOOCKS
Based on the identied need to develop a tool to support ontology visualizations for curricula in response to stu-
dents and lecturers needs, a visualization tool (called Knoocks) was developed with the focus on classes, instances,
datatype properties and object properties. Knoocks combines a container approach for instances to represent the hier-
archical structure and a node-link approach for dependencies between classes/instances to clearly differentiate between
subClassOf relationships and object properties. In contrast to existing ontology visualization tools it was important to
develop a tool which should give users the possibility to simultaneously see the overview of the ontology structure and
a detail view for the instances in connection with their classes and properties. The basic layout of Knoocks with its three
main components main window, preview window and toolbox is shown in Figure 7. The detail view only shows one
block at a time and presents the courses (instances) of the corresponding modules (classes) with their properties. A block
(as shown in Figure 7) is constructed for each main module. Modules which are connected by subClassOf relationships
are grouped in a hierarchical left to right manner. For example the module SWE- Software Engineering in Figure 7 is a
submodule of PI-Information Technology. Clicking on a course (instance) in the detail view opens separate tables for
additional information of the selected course (datatype properties) and for the different dependencies (object properties)
between the selected course and the other courses (which can also be seen in Figure 7 for the course Database Systems).
The table allows users to directly jump to a connected course and the detail view automatically focuses on the new
selected course. The overview (see B in Figure 7) shows all blocks which are arranged in a circular pattern to see the
overall structure of the ontology. Dependencies (object properties) between courses are presented as curves and colors
are used to visually differentiate between dependencies different. The thickness of the curve reects the number of con-
nections between the courses. The individual connections can be inspected in detail by clicking on or hovering the mouse
over a curve. The intensity of the color of modules in the overview reects the number of courses in the module (darker
color represents more courses). Furthermore, users have the possibility to individually arrange the blocks. A switch but-
ton gives users the possibility to choose if the overview or the detail view is in the main window and a double click on
a block automatically opens the detail view of this block in the main window. Moreover, the toolbox contains functions
which allow users to interact with the ontology (e.g., important modules can be highlighted or users can search specic
modules or courses). For a thorough description of Knoocks and its development process see also (Kriglstein, 2011).
Figure 7: The main components of Knoocks : A) toolbox, B) preview window and C) main window.
This section gives a short overview about the most important insights which were gained during the user tests (for a
detailed description of the evaluation of Knoocks see also (Kriglstein, 2011)). The human centered development process
of Knoocks required four iteration cycles. To develop a visualization tool which meets users requirements, Knoocks was
tested with students and lecturers after each iteration cycle to detect usability problems and to check if the design idea
was understandable, which functionalities needed further improvements and if users missed certain important features.
The rst prototype of Knoocks was tested by six students (three students who presented the primary persona and three
students who presented the secondary persona). The testing session for each student took about 40 minutes and concen-
trated on getting feedback if the block representation was understandable. The focus of the evaluation of the second ver-
sion of Knoocks was on detecting misinterpretations or unclear elements. The prototype was tested with 15 students and 7
lecturers who presented the primary personas (test session for each participant took 90 minutes). In contrast to the previ-
ous user tests, the third version of Knoocks were tested with three lecturers (primary persona) who were ontology experts
and each testing session for each lecturer took about 180 minutes. Finally, the fourth version was tested with 9 students
and 7 lecturers who presented the primary personas (each testing session took about 120 minutes) and with four prospec-
tive students (secondary persona) who wanted to enroll at a university (each testing session took about 70 minutes).
For the evaluation of Knoocks, the following methods were used: observations in combination with thinking aloud,
semi-structured interviews and task scenarios. The focus of tasks for the user tests was to detect specic modules and
specic courses as well as to analyze dependencies between courses and to get also detail information of a course. While
participants solved these tasks, they were observed and encouraged to think aloud to make their behavior and decisions
more transparent. After they nished the tasks they were asked if the visualization was helpful and if they have sugges-
tions for improvements.
The results of the user tests showed that the concepts of Knoocks and especially the layout of the blocks was clear.
The participants liked the layout of the blocks which allowed them to quickly see the submodules and the courses for
each module. Furthermore, they found the combination of the overview and the detail view very helpful and they liked
that they had the possibility to select which view was in the main window. The observation and thinking aloud protocols
showed that the participants found Knoocks and its functionalities intuitive and easy to learn. They stated that the repre-
sentation of dependencies in the overview gives a fast impression of the relationships between courses. Furthermore they
stated that the table representation of dependencies and additional information of a course in the detail view gives them a
clear overview.
Additionally to the usability evaluation, Knoocks was also compared with other visualization tools (among others:
Jambalaya (nested graph view) and OntoViz). To make the tools comparable, the participants got similar tasks. The re-
sults of the comparison study showed very well that usability played an important role for the successful acceptance of
such visualization tools and also if participants interpret the curriculum visualization correctly. For example, in the case
of OntoViz, the meaning of the abbreviations for the drawing settings was incomprehensible for students and lecturers
and based on false interpretations of these abbreviations, they drew graphs without the necessary information to answer
most of the tasks (e.g., to nd the additional information of a specic course). Although they also liked the layout of the
nested graph in Jambalaya, they pointed out that it was harder for them to get a general overview, because of the repre-
sentation of the dependencies. They also had problems to navigate with the zoom function. They wanted to zoom in, but
the zoom out button was selected or they selected the zoom in button and they assumed that no further submodules levels
were available, because the visualization showed no reaction.
Describing of curriculum as an ontology is a good possibility to clearly show the structure and dependencies. Graph-
ical representations of such curriculum ontologies support students and lecturers to understand the structure of the cur-
riculum with their dependencies between courses and modules easier. This improves the management of the study pro-
grams. For a positive acceptance of such curriculum visualizations, it is necessary that the design of the visualization is
usable. Therefore the needs of users play an essential role to make a working with such visualizations more effective.
Therefore, the human centered design process a well-known approach in human computer interaction to involve poten-
tial users already in the early design and development phase to understand their needs, tasks, environment and expecta-
tions was adapted.
The paper presented the core parts of a case study on a human centered approach to develop a visualization tool
which can be used to represent a curriculum ontology. The starting point was to understand the use of context (e.g., what
students and lecturers want to do with a curriculum visualization) and to specify which information of the curriculum
had to be visualized. Therefore, clear user proles were necessary to specify the typical user groups. For a deeper un-
derstanding of the different user groups, personas were designed. In contrast to user proles, personas include typical
scenarios which allow to see the curriculum visualization from the viewpoint of students and lecturers during the whole
development process of the visualization tool. Based on this information, requirements were specied. For the visualiza-
tion of curriculum ontologies, it was important that the tool contains classes (they dene the modules of the curriculum),
instances (they are used to specify the courses) and properties (e.g., to show the dependencies between courses or to
specify additional information of a course). Furthermore, the case study demonstrated that usability was central for the
acceptance and quality of the visualization tool, because the modications based on the results of the user tests signi-
cantly simplied the work with the tool. While the Knoocks prototype implementation is working under C# a Protg
plug-in is being implemented in order to allow Knoocks to become available for a broader group of students and lectur-
ers in tertiary education.
We sincerely thank Gnter Wallner from the University of Applied Arts Vienna for the critical feedback and for the
support of the development of the visualization tool Knoocks.
Alani, H. (2003). TGVizTab: An Ontology Visualisation Extension for Protg. Proceedings of the Knowledge Capture Work-
shop on Visualization Information in Knowledge Engineering, Sanibel Island, Florida, USA.
Commission, E. (2010). The Bologna Process. Retrieved 16.11.2010, 2010, from http://ec.europa.eu/education/policies/educ/
bologna/bologna_en.html
Cooper, A. (2004). The Inmates Are Running the Asylum: Why High-tech Products Drive Us Crazy and How to Restore the
Sanity (2nd ed.): Sams.
Courage, C., & Baxter, K. (2005). Understanding Your Users: A Practical Guide to User Requirements Methods, Tools, and
Techniques: Morgan Kaufmann.
Forst, N. (2010). Human Centered Design. BSc thesis, University of Vienna, Austria.
Kabicher, S., Derntl, M., & Motschnig-Pitrik, R. (2008). Towards an Active Curriculum for Computer Science. Proceedings of
the World Conference on Educational Multimedia, Hypermedia & Telecommunications (ED-MEDIA 2008), Vienna, Aus-
tria.
Kriglstein, S. (2008). Analysis of Ontology Visualization Techniques for Modular Curricula. Proceedings of the 4th Symposium
of the Workgroup Human-Computer Interaction and Usability Engineering of the Austrian Computer Society on HCI and
Usability for Education and Work, Graz, Austria, pp. 299 - 312.
Kriglstein, S. (2010). Curriculum Visualization: User Study about Experiences and Expectations. Proceedings of the World Con-
ference on Educational Multimedia, Hypermedia &Telecommunications (ED-MEDIA 2010), Toronto, Canada, pp. 775 -
Kriglstein, S. (2011). Human Centered Visualization of Ontologies. Doctoral thesis, University of Vienna, Austria.
Maguire, M., & Bevan, N. (2002). User Requirements Analysis: A Review of Supporting Methods. Proceedings of the IFIP 17th
World Computer Congress - TC13 Stream on Usability.
Sintek, M. (2007). OntoViz. Retrieved 04.12.10, 2010, from http://protegewiki.stanford.edu/index.php/OntoViz
Sommaruga, L., & Catenazzi, N. (2007). Curriculum visualization in 3D. Proceedings of the 12th International Conference on
3D Web Technology, Perugia, Italy, pp. 177 -180.
Stone, D., Jarrett, C., Woodroffe, M., & Minocha, S. (2005). User Interface Design and Evaluation. San Francisco: Morgan
Kaufman.
Storey, M. A., Callendar, C., Lintern, R., Ernst, N., & Best, C. (2010). Jambalaya. Retrieved 04.12.10, from http://protegewiki.
stanford.edu/index.php/Jambalaya
W3C. (2004). OWL Web Ontology Language Overview. Retrieved 18.12.2008, from http://www.w3.org/TR/owl-features/
Wang, T. D., & Parsia, B. (2006). CropCircles: Topology Sensitive Visualization of OWL Class Hierarchies. Proceedings of 5th
International Semantic Web Conference (ISWC 06), Athens, USA, pp. 695 - 708.
Students Perceptions of Online Learning and Instructional Tools 141
Students Perceptions of Online Learning and Instructional Tools: A Qualitative Study of
Undergraduate Students Use of Online Tools
DAVID A. ARMSTRONG
The number of students taking at least one fully online class from an accredited university in the United States has
grown signicantly over the past decade. Between 2002 and 2007, the number of online students jumped 145%, from
1,602,970 to 3,938,111. Moreover, of the 17,975,830 students enrolled in degreegranting postsecondary institutions in
the US in 2007, 21.9% were taking courses online (Allen & Seaman, 2008). This upward trend in online enrollment,
which is expected to continue well into the second decade of the 21
century, clearly poses a number of challenges to
the education community (Allen & Seaman, 2008). How will universities handle such a rapid increase in the number of
online students? What alternative course delivery methods will best meet online students needs? To date, much research
has been focused on the former question and, in particular, on the technical aspects of online education such as access
and information delivery. Research in the area of the latter subject is growing but has overlooked one critical aspect that
needs to be understood if electronic learning (elearning) is to be made more effective in the future: how do student per-
ceptions impact their actions, approaches, and learning within the online educational environment?
The purpose of this study was to describe undergraduate students experiences and perceptions of online courses
based on interviews, observations, and online focus groups. I describe (a) motivational and learner characteristics within
online classes, (b) the positive and negative aspects of online courses as experienced by students, (c) what instructors can
do to improve the teaching of online courses, and (d) how undergraduate students perceptions of the online learning en-
vironment and the tools used affects the selection of their approach to learning.
SIGNIFICANCE OF THE PROBLEM
Research into the effectiveness of online instruction has looked primarily at individual implementations of instruc-
tional methods within a single class or set of classes taught by a single instructor (Means et al., 2009). Where this study
differs is that it investigates online instruction in the typical faculty-developed course, that is, approaching online in-
struction from the student perspective in a mix of typically delivered and designed classes. These classes were not the
exceptional online class designed to investigate a new or innovative online practice, they were simply what Edventures
(2009) would term the current state of online instruction.
This study provides a rich, complex, and detailed picture of students within the online learning environment. By
organizing the analysis of data and content around approaches to learning, learnercentered tools can be developed that
promote deep learning approaches in undergraduate students during online learning experiences. Results from this study
yielded recommendations for changes in the design of online and elearning that encourage student learning that is
aligned with faculty, student, and institutional perceptions of online education. Faculty may be expected to improve their
online instruction through a clearer insight into the effects of course management tools.
Developing effective online learning environments is becoming a challenge for many universities. Current trends in
education, which include shrinking funding, have spurred greater competitiveness among universities as they seek new
ways to attract students not only in traditional environments but also in the online environment. In both, it is important to
maintain academic integrity and to ensure high levels of student learning and by achieving a better understanding of stu-
dents needs in relation to their learning, online education can be improved an its value as an educational tool increased.
By investigating ways that students perceive and interact with the learning environment, it may be that the design of the
online learning environment can be better developed to support learning.
142 Armstrong
From a business of education standpoint, it is essential to remember that practitioners of education should not only
be concerned with the number of degrees awarded but also the quality of student learning obtained in achieving those de-
grees. Thus, the focus of this study was on the students, who they are and how best they can be served.
DESIGN, COLLECTION, AND EVALUATIVE FRAMEWORK
The methodology used in this study was derived primarily from research into student learning and the selection of
approach, in the tradition of Marton and Sajjo (1976), Entwistle and Ramsden (1983), Biggs (1987), Prosser (1999), and
Ramsden (2002). Central to this approach is the perspective of the student regarding both the process and outcomes of
learning and instruction. Qualitative data-collection techniques were used to obtain and describe undergraduate student
views on online instruction, online learning tools, and instructional processes. Three stages of data collection were used
in this study these were (a) one-on-one open-ended interviews, (b) think-aloud observation, and (c) online focus groups.
The main data collection was student interviews. Data from think-aloud observations and online focus groups were used
to conrm ndings from the interviews. Data were collected between the Summer and Fall academic sessions of 2008 at
two sites. Additional data were collected in the Summer of 2010. This study will continue and be updated with data col-
lection resuming in the Summer of 2011.
The sample consisted of 16 undergraduate students who had completed or were enrolled currently in an online
course at one of the two universities. Students were recruited to participate in one or more of the data-collection meth-
ods; these were 11 in the interview process, 8 in the think-aloud observations, and 8 in the online focus groups: 5 in one
group and 3 in the other group. Student participants were mostly in their mid20s; 10 were female, and 6 were male.
Three students participated in all three data-collection methods, ve students participated in two of the data-collection
methods, and eight students participated in only one data-collection method.
All students were drawn from religiously afliated universities in Northern California. Both universities (S1 and S2)
are primarily undergraduate universities, whereas university 2 (S2) has a more diverse population both in age and ethnic-
ity. The graduate populations at both schools were not included in this study. University 1 (S1) is a mediumsize, private
university with a student population of approximately 8,500: about 5,000 undergraduate students and 3,500 graduate stu-
dents. The undergraduate population has a male to female ratio of 45% to 55%, and about 35% of undergraduate students
identify themselves as persons of color. Almost 60% of undergraduates are from California, with the others coming from
throughout the United States and more than a dozen foreign countries. Between 65% and 70% of undergraduate students
receive some form of nancial aid: scholarships, grants, or loans. University 2 (S2) has an undergraduate population of
approximately 5,500 and a graduate population of approximately 3,300. The ethnic breakdown for S2 is as follows: Eu-
ropean American 39%, Asian American 20%, Latino or Hispanic American 15%, International 7%, African American
4%, Native Hawaiian or Pacic Islander 2%, and Native American 1%, with 11% unidentied.
Faculty participation in this study was not a requirement. Two of the faculty from S1 met with me prior to the start
of data collection. The purpose of this meeting was to discuss the upcoming course offerings and the data-collection pro-
cess. Currently, the majority of online course offerings at S1 are within the College of Arts and Sciences academic sum-
mer programs; the remaining offerings are in the business school and the law school. Students in S1 for this study were
primarily from the College of Arts and Sciences and the School of Business undergraduate programs. Students from S2
were drawn from business and nursing.
The framework of approach to learning is used to analyze the data collected for this study. Three approaches to
learning as described in the literature are called deep, strategic, and surface. Strategic learning is sometimes called
approaching, depending on the researcher and the nature of the study. Deep learning is dened as examining new facts
and ideas critically, tying them into existing cognitive structures, and making numerous links between ideas (Rosie,
2000). The deep learner is able to retain information and to organize materials in a variety of ways that aid in making
meaningful connections that promote learning. Characteristics of deep learning include: looking for meaning, focusing
on the central argument or concepts needed to solve a problem, interacting actively, distinguishing between argument
and evidence, making connections between different modules, relating new and previous knowledge, and linking course
content to real life. The strategic learner is a student who intends to achieve the highest grade possible through effective
time management and organized study methods. Students exhibiting a strategic approach are focused on the assessment
process (Entwistle & Ramsden 1983).
I examined participants responses in interviews, think-aloud observations, and online focus groups; categorization
of responses was based on the tools mentioned, statements of value, and perceptions of positive or negative effect on
learning. The think-aloud observations and online focus groups served to conrm or add insights to data collected during
the interview process. Sixteen undergraduate students who had completed or were enrolled in an online course at one of
two universities participated in the study. Of the 16 students, 11 participated in the interview process, 8 in the observa-
tions, and 8 in online focus groups. Three students participated in all three data-collection methods, ve students partici-
pated in two of the data-collection methods, and eight students participated in only one data-collection method.
Analysis of the data from interviews, think-aloud observations, and online focus groups produced ve major nd-
ings. These ve ndings are (a) the role of communication in shaping perceptions and actions of students, (b) how tech-
nology is used not the technology determines its value, (c) the role of course organization for students success, (d) ap-
proaches to learning are shaped by students perceptions as are students determination of academic quality, and (e) stu-
dents use nonacademic resources because of ease and familiarity.
The role of communication in online learning took many forms and was dominate in every data-collection method.
Although students took online courses because they wanted independence and self-regulation, they also stated a desire
for concise directions on everything from assignments and assessments to when and how to access course information.
The expectations for communication went beyond just a need for direction. All of the participants expresed a view that
faculty was missing from the educational conversation. How instructors communicate online was perceived to a limita-
tion of online learning. When communication was perceived lacking, participants lower their approach learning electing
for more strategic or surface learning.
Participants did not perceive the negative attributes of technology to be inherent in the technology so much as to
its use and implementation. What participants expected was that communication technologies would be used in ways
familiar to them and in providing a timely response to participants educational needs. Indeed, poor technology imple-
mentation was mentioned in association with the lack of organizational structure found in some online instruction. In
interviews, think-aloud observations, and online focus groups, participants expressed the perception that faculty lacking
in technology skills were likely to use or implement technology in a way that resulted in confusion.
All 16 participants stated that the main reasons for pursuing online instruction were exibility and selfcontrol with-
in the learning environment. Participants perceived online learning to be a convenient alternative to traditional classroom
learning but indicated that convenience came with a price: in gaining independence, selfdirected learning, they were
losing direction from and communication with instructors. In some instances, this tradeoff was perceived to decrease the
educational and academic value of the learning experience. For these participants, academic value was perceived to come
from interaction and engagement from peers and faculty. Participants indicated that without necessary direction from fac-
ulty online learning allows for an approach to learning that is more surface or strategicoriented than is the case in the
traditional facetoface classroom experience.
The resources provided by universities for students research and information gathering were perceived to of less
value then nonacademic tools. The use of nonacademic database sources was especially true when participants were
asked to use online databases to perform research. During the think-aloud observations, participants used Google and
Wikipedia before those resources provided by the university. When asked to explain their use, participants stated that
Google and other free tools are familiar and do not have the access restriction placed on them that university systems
have. Additionally, participants stated that the university tools were cumbersome and hard to navigate.
In summary, tools used for communicating or conducting research were not as important as the communication it-
self. Perceptions of value for any tools used depended not on the tool but on the speed and consistency of communica-
tions. Participants did not perceive the negative or positive attributes of tools or technology to be inherent to the technolo-
gy itself, but to its use and implementations. When faculty were perceived to be unresponsive, it was not the tool that was
perceived to be of little educational use but the level of communication. When faculty were perceived missing from the
educational conversation the academic quality was perceived diminished compared with face-to-face instruction. When
the academic quality was perceived low, participants exhibited a strategic or surface approach to the learning.
RECOMMENDATIONS FOR FUTURE RESEARCH
Although this study conrmed past research results (Cotton, 2006), it is believed that a more thorough study will
provide additional data on students perceptions and use of the online environment in the promotion of learning. Al-
though the study was limited to two religiously afliated institutions with limited online programs, a larger study would
offer results that could be applied more generally. This study was conducted in primarily short 5-week summer online
courses with limited enrollments. The exceptions to the summer courses were two short 5-week courses at University 2
(S2) in the Fall of 2009 and the OMIS course taught by me during the regular 10 week sessions at University 1 (S1). It
is possible that selection of a small range of online courses produced a limited range of course interactions. Because of
this limitation, a larger study conducted in a wider range of disciplines may produce a different set of results. One such
study, just completed and published, showed similar ndings, yet was able to look more closely at students approach to
learning through a wider set of interviews and other data-collection methods (Ellis & Goodyear, 2010). Ellis and Good-
year were able to draw a relationship between online discussion and student approach to learning. In particular, Ellis and
Goodyear found statistically signicant relationships between deep approaches, cohesive conceptions, positive percep-
tions of the learning context, and higher levels of student performance. Although the collection of student performance
data can be problematic, more studies investigating students measured use of approach in relation to student outcomes in
online learning could prove useful.
Perceptions of communication played an important role in the results of this study. Although this study relied on stu-
dents perceptions of communication and observations of their actions within the online environment, actual communi-
cations were not assessed. Future studies that look at possible links between faculty use of communications, the content
and amount and communications online, and the perceptions of students may be warranted.
This study and others have investigated only a few of the possible relationships between perception of the online
environment, the tools used, students approach to learning, and students perceptions of learning. A more focused inves-
tigation of student perceptions of the design of online learning, including Internet resources, the role of community, and
social networking is needed. Although Internet resources and community and social networking were mentioned in this
study, they may play a larger role in student communication and learning than was described in this study.
An additional phenomenon not investigated fully by this study is the link between perception and outcome. Assum-
ing a link between perception and outcome based on past research may not be sufcient when considering the online en-
vironment. An investigation of the relationships between online perceptions, approach, and outcomes is an area that may
merit further research.
Comparing students expectations and actual use of communication technology with the use and expectation in the
online classroom could inform future studies on student perceptions of online learning. Faculty use and knowledge also
may effect student perceptions. Additional factors not investigated in this study include (a) institutional beliefs around
online learning, (b) the place of online learning in the strategic plan of the university, and (c) the implementations of
teaching standards related to online instruction.
Studies investigating faculty perceptions and training are numerous; however, the link between faculty training in
the use of standards for the development of online courses and student perception and outcomes is not well understood.
What participants say they want in an online course and the standards as written into resources such as the Rubric for
Online Instruction (ROI) are similar. These similarities include more communication, faster response time, and more
engagement with peers and faculty. The standards used in the ROI, and other such tools, are widely used teaching the de-
velopment and assessment of online education across the United States and other countries, but studies linking the stan-
dards contained in the ROI to student perception are limited, as are studies linking the use of the ROI to either student
approach or increased student outcomes. An investigation of the effects of the ROI on perception, approach, and outcome
may provide educators a better understanding of how best to design online education in the future. One possible outcome
of such research may be that including basic teaching strategies not specic to the online environment are not necessary
in tools like the ROI and that including such information diminishes there value to faculty.
Although not specically a limitation, one area of concern within this study is that participants statements of re-
sponse time was not followed up on during data collection. All participants stated that faculty and students were unre-
sponsive at some time yet a precise time was never ascertained. Investigating what is an appropriate response time for
e-mail, discussions, and assignments may be an area of further research.
Allen, I. E., & Seaman, J. (2006). Making the Grade: Online Education in the United States, 2006. Olin Way, Needham, MA
02492-1200: Sloan Consortium
Biggs, J. (1976). On Qualitative Differences in Learning: I-Outcome and Process. British Journal of Educational Psychology,
46, 4-11.
Biggs, J. (1979). Individual differences in the study process and quality of learning outcomes. Higher Education, 8, 381-384.
Biggs, J. (1987). Student Approaches to Learning and Studding: Study Process Questionnaire: Manual. Melbourne: Australian
Council for Educational Research.
Cotton, D., & Gresty, K. (2006,). Reecting on the think-aloud method for evaluating e-learning. British Journal of Educational
Technology, 37, 45-54.
Entwistle, N., & Ramsden, P. (1983). Understanding Student Learning. New York: Nichols.
Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis: verbal reports as data (Revised ed.). London: MIT Press.
Marton, F., & Saljo, R. (1976). On Qualitative Differences in Learning, Outcome and Process. British Journal of Educational
Psychology, 46, 4-11.
Richardson, J. C., & Newby, T. (2006). The Role of Students Cognitive Engagement in Online Learning. The American Journal
of Distance Education, 20, 23-37.
Richardson, J. (2003). Approaches to Studying and Perceptions of Academic Quality in a Short Web-Based Course British Jour-
nal of Educational Technology, 34, 433-442.
Richardson, J. (2005). Students Approaches to Learning and Teachers Approaches to Teaching in Higher Education. Educa-
tional Psychology, 25, 673-680.
Online Discussions: The Effect of Having Two Deadlines 147
Online Discussions: The Effect of Having Two Deadlines
MICHAEL J HERRICK, MENG-FEN GRACE LIN, AND HUEI-WEN TSAI
Discussion forums are the heart of asynchronous online courses. These forums, in all their various implementations,
provide public spaces for students to interact with each other and with their instructor. The written exchange of ideas
online replaces the oral discussions that occur in face-to-face classrooms. Creating a discussion forum which engenders
students to actually discuss ideas is not a trivial endeavor. There are many factors which go into the design of an asyn-
chronous online course. Gilbert and Dabbagh (2005) termed these design elements structuredness and went on to say,
These include facilitator guidelines, posting protocols (eg, number of postings required, length of posting, pacing of
postings), evaluation criteria, grade weight, nature of topic discussed, number of participants, length of discussion, and
degree of instructor or facilitator participation, among others. (p 8)
While teaching an asynchronous online class in the fall of 2009, the authors noted that students seemed to wait until
the assignment deadline to make required postings to the class discussion forum. Doing so afforded their classmates little
time to comment on what they had written and virtually precluded a back-and-fourth conversation to take place on the
weekly topic. Gilbert and Dabbagh (2005) addressed the same impediment. The authors of the current study postulated
that a two-deadline solution, one in which initial responses were due days before peer comments, would serve to better
incite dialogue among the students. The solution was implemented the following semester; its success is the subject of
this paper.
The purpose of this quantitative study is to determine the effect of having two deadlines in an online discussion with
graduate and undergraduate students enrolled in a technology in education course at a major US public university. Spe-
cically, the research is guided by three questions:
1. Do students given two deadlines contribute more often (number of posts) to an online discussion than those only
given one deadline per discussion prompt?
2. Do students given two deadlines contribute more (number of words) to an online discussion than those only
3. Does the dialogue ow better when students are given two deadlines per prompt?
Discussion is one of the most utilized tools in online course delivery. Richardson (2010) argued that students can de-
velop critical thinking stills through online discussions. Many researchers have been trying to answer the question of how
an online discussion can be designed to foster meaningful learning. There are various factors that might inuence how
students are engaged in online discussions. Factors range from discussion topics, if a discussion is graded or mandatory,
how an instructor is involved in the discussion, to personal factors such as availability of time and knowledge about the
topic. Cheung and colleagues (2008) investigated what motivated participants, given the freedom of choice, to contribute
to discussions. In their study, each student owned an individual discussion forum and could invite others to participate.
Their study showed that the most common motivator for students to participate in an asynchronous discussion lies in
their friendship with one another and the feeling that their contributions were acknowledged by the forum owner.
Similarly, Chang (2008) examined the correlation between members backgrounds, discussion content and frequen-
cy, and the quality of their problem-based projects. While most groups showed positive correlation between discussion
frequencies and projects, one group had a negative correlation, meaning this group had poor performance in discus-
sion frequency but with strong work achievement. However, online discussions content-wise had a positive correlation
to project work, even though the degree varied. Most importantly, this research pointed out that an individual members
background, such as technology literacy, played an important role in their work quality.
It is clear that some students will not voluntarily participate in an online discussion when it is not part of the course
grade (An, Shin, & Lim, 2009). An array of variables such as learning styles, preferences in discussion structure, and
148 Herrick, Lin and Huei-Wen
dimensions of interactions have been examined with the general conclusion pointing out the obvious: students who were
self motivated performed best even though instructors presence and facilitation also played an important role in stu-
dents learning experience (Friedman, 2010). Hew (2008) suggested using students as facilitators in an online class and
seeing how different facilitation strategies such as introduction, engagement, and mentoring might foster deeper peer
involvement. Baran and Correia (2009) investigated student-led facilitation strategies and identied three strategiesin-
spirational, practice-oriented, and highly structuredto be the ones that successfully overcome the challenges of instruc-
tor-dominated facilitation in an online asynchronous discussion environment. This study took place in the context of a
masters program of education in which two discussion groups in one online course designed instructional projects and
facilitated asynchronous discussions. Using quantitative data as preliminary analysis, this qualitative study showed that
participants were enthusiastically and actively engaged in the discussions where the student facilitators, respectively, put
weight on eliciting participants personal stories, making associations with the participants practice or real-life profes-
sional situations, and systematically scaffolding the discussions within the participants knowledge of their background
as well as the topic assigned. Assigning different student roles in an online class is not a novel idea. For example, Wise,
Saghaan, and Padmanabhan (2009) studied a set of seven commonly assigned student roles in an online asynchronous
discussion board and developed a set of description of the roles. In addition to role assignment, which provides a struc-
ture that fosters student participation, another instructional approach could be to have students vote for a best posting
weekly. The students who received the highest votes would receive extra credits for the assignment (Meyer, 2008).
Research has been done on changing aspects of asynchronous online course protocols, such as those indicated
above. More interesting ideas on how to design an online discussion are topics of ongoing research. The literature seems
bereft of studies which focus on changing simple procedural points to quantify the effect. The number of weekly dead-
lines is one such point.
Computers in Education is a required course for undergraduate pre-service teachers and an elective for graduate
students in the College of Education at a major US public university. The course is a three-credit class focusing on teach-
ing about technology applications for effective integration into educational settings. Implemented as an asynchronous on-
line class, all class content is housed in the universitys course management system (CMS) while Ning, a social network-
ing site, is used for assignment submissions and discussions. This study focuses on seven specic discussions, engaged
in by ve groups of students, enrolled during two different semesters.
Three groups of students (n = 15, 16, and 10) were enrolled in Computers in Education during the Fall 2009 Se-
mester. Each group was facilitated by a different Teaching Assistant (TA) under the guidance of an Educational Technol-
ogy (ETEC) Department professor. All students in every group accessed the same course materials in the universitys
CMS and all students were given the same scripted prompts for assignment submission. The prompts were categorized as
discussion, skill, or learning unit. The seven discussions, those being used for data collection in this study, were titled: (1)
Who are you?, (2) Copyright & Fair Use, (3) Classroom Management, (4) Teaching Tools, (5) Evaluating WebQuests, (6)
Online Learning with Games, and (7) Edublogs & More. For the discussions, students were given at least a week to post
an initial reply to the prompt and to reply to at least two other students posts. All submissions were required by midnight
Two groups of students (n = 17 and 14) were enrolled in Computers in Education during the Spring 2010 Semes-
ter. One group was facilitated by the ETEC Department professor previously mentioned and the second was facilitated by
a different TA. The same course materials and prompts as the previous semester were used. For the discussions, students
were given at least a week to post an initial reply to the prompt and then two additional days to reply to at least two other
students posts. During this semester, the initial submission was required by midnight on Monday and the replies by mid-
night on Wednesday. The implementation of two deadlines was the only procedural difference between the two semesters
of Computers in Education.
Upon completion of the courses, and with the informed consent of the students, data was collected from the seven
discussions of each group. The number of student and instructor posts in each Ning discussion was tallied. Each student
post in the Ning Forum discussion was examined for response level, number of days posted before the nal deadline,
and number of words in each post. Response Level records the depth of each post in relation to the original discussion
prompt. For example, a Level One (or L1) post indicates a direct initial post to the discussion question, while a Level
Two (or L2) post means a post that reacts to an L1 post. Table 1 describes the variables Response Level, which ranged
from 1 to 4, and Number of Days, ranging from 6 to -6. Microsoft Word was used to count the number of words in each
post so that the denition of word would be consistently evaluated by that software. It is important to note that Num-
ber of Days always refers to the number of days before the nal deadline for both sets of groups; for the group with two
deadlines, Number of Days refers to the number of days before the Wednesday deadline.
Table 2. Description of Response Level and Number of Days
Response Level Number of Days
Level 1 reply to instructors prompt
Level 2 reply to Level 1 post
6 posted six or more days before the nal
5 posted ve days before the nal due
4 posted four days before the nal due
3 posted three days before the nal due
2 posted two days before the nal due
1 posted one day before the nal due
0 posted on the nal due date
-1 posted one day after the nal due date
-2 posted two days after the nal due
-3 posted three days after the nal due
-4 posted four days after the nal due
-5 posted ve days after the nal due
-6 posted six or more days after the nal
While collecting data, some anomalies were observed. The most salient of these occurred because the instructor in
one of the Spring 2010 groups deviated from the scripted lesson plan. She combined Discussions 2 and 3 with other as-
signments, convoluting the data and making those discussions unusable for this study. It was also noted that some of the
Ning Discussion Forums were set up as threaded, showing responses as indented paragraphs below previous posts and
making the determination of Response Level easy, while others were set up as at, quoting previous responses within
every reply and making the determination of Response Level more difcult.
There were a total of 1666 posts made to the seven discussion prompts under study by ve groups. Of these, 1427
were made by students and 239 were made by instructors. Considering only the student posts, there were 829 made in
the three one-deadline groups and 598 made in the two two-deadline groups. The average number of posts per student
(total number of posts/total number of students) and the average number of words per student post (total number of
words/total number of students) were computed for each of the 33 discussions for which data was obtainable (5 groups x
7 discussion topics - 2 rejected discussions). The mean values and standard deviations of these variables are shown, dis-
aggregated by deadline groupings, in Table 2.
Table 3. Descriptive statistics for average posts per student and average number of words per student post.
Average Posts per
Average Words per
Group # discussions Mean SD Mean SD
One Deadline 21 2.88 .60 461.07 131.87
Two Deadline 12 3.29 .41 531.96 163.24
In order to answer the rst research question, Do students given two deadlines contribute more often (number of
posts) to an online discussion than those only given one deadline per discussion prompt?, an independent samples t-test
was conducted to compare the average number of posts per student in the two groups. The mean average posts per stu-
dent in the one-deadline discussions (M = 2.88, SD = 0.60) was signicantly less than the mean average posts per student
in the two-deadline discussions (M = 3.29, SD = 0.41); t(31) = -2.129, p < .05. Students given two deadlines per week
contributed more often to the online discussion than those given only one deadline. About 13% of variance can be attrib-
uted to the difference in deadlines (
= .128).
Turning to the second research question, Do students given two deadlines contribute more (number of words) to an
online discussion than those only given one deadline per discussion prompt?, an independent samples t-test was con-
ducted to compare the average number of words per student post in the two groups. The mean average words per student
post in the one-deadline discussions (M = 461.07, SD = 131.87) was not signicantly different than the mean average
words per student post in the two-deadline discussions (M = 531.96, SD = 163.24); t(31) = -1.362, p = .183. Students
given two deadlines did not contribute any more to the online discussion than those given only one deadline.
The nal research question asks Does the dialogue ow better when students are given two deadlines per prompt?
No single quantitative statistic was observed that measured this quality. Two factors, taken together, can give insight into
dialogue ow: average number of student posts at each Response Level and percentage of student posts by Number of
Days before the deadline. First, consider the number of posts at each of the four levels that contained posts. Table 3 con-
tains the mean average number of student posts and standard deviations at each Response Level. Figure 1 illustrates the
data in a stacked column graph. The two-deadline group, given two days in which to only post replies to their classmates
Level 1 responses, clearly had more Level 2 responses.
Table 4. Descriptive statistics for average number of student posts at each Response Level.
Level 1 Level 2 Level 3 Level 4
Group # discussions Mean SD Mean SD Mean SD Mean SD
One Deadline 21 .99 .14 1.64 .53 .24 .18 .01 .03
Two Deadline 12 .95 .09 2.02 .24 .29 .24 .03 .05
One Deadline Two Deadlines
Level 4 Post
Figure 8. Mean average number student posts at each level, stacked by deadline group.
Examining the signicance of the comparison of average number of student posts at each Response Level, indepen-
dent samples t-tests were performed. The mean average number of student posts at Level 1 in the one-deadline discus-
sions (M = 0.99, SD = 0.14) was not signicantly different than the mean average number of student posts in the two-
deadline discussions (M = 0.95, SD = 0.09); t(31) = .694, p = .493. The Level 1 posts for both groups was essentially 1
per student, the value that would be expected as each student makes an initial response to the instructors prompt. Level
3 responses were not signicantly different, t(31) = -.676, p = .504, nor were Level 4 responses, t(14.09) = -1.306, p =
.212. (See Table 3 for means and standard deviations.) It was in Level 2 where the mean average number of student posts
in the one-deadline groups (M = 1.64, SD = .53) was signicantly different than the mean average number of student
posts in the two-deadline groups (M = 2.02, SD = .24); t(31) = -2.331, p < .05. Students given a second deadline, two
days after the rst, in which to respond to their classmates posts made signicantly more Level 2 posts to the online dis-
cussions than those given only one deadline. About 15% of variance can be attributed to the difference in deadlines (
.149).
Examining the Number of Days before the deadline, a summative comparison of the one and two-deadline group
data is illustrated in Figure 2. The two-deadline group consistently had postings completed earlier. By two days before
the nal deadline, the rst deadline for the two-deadline group, 63% of the two-deadline groups comments had been
posted while only 28% of the one-deadline groups comments had been posted. On the day of the nal deadline (Day 0),
39% of the one-deadline groups posts were made bringing their cumulative total to 85% and leaving 15% of the posts to
be made after the deadline. By contrast, only 19% of the two-deadline groups posts were made on Day 0 for a cumula-
tive total of 97%.
Returning to the question of dialogue ow, it can be said that the two-deadline groups had a better ow than the
one-deadline group for two reasons. First, the two-deadline groups posted earlier, completing nearly two-thirds of their
comments two days before the nal deadline, giving their classmates more time to read, consider, and reply to responses.
Second, the two-deadline groups actually made more follow-up, Level 2 responses. The mean average number of L2 re-
sponses in the two-deadline groups was 23% higher than the mean average number in the one-deadline groups.
Postings by Deadline
6 5 4 3 2 1 0 -1 -2 -3 -4 -5 -6
Days Before Final Deadline
One Deadline
Two Deadlines
Figure 9. Posting by deadlines; cumulative percentages.
There were signicant differences between the one-deadline and two-deadline groups, validating the authors ra-
tionale for changing procedures to have students post initial responses by one date and follow-up responses a few days
later. This change of protocol was a minor modication in the overall instructional design of the course. The switch to
two deadlines was made to affect a change in student behavior: to encourage the student to participate more fully in the
online discussion by not waiting until the last minute to post comments. This study shows that a minor change of proto-
col can have a statistically signicant affect on outcomes but it does not indicate if changes in behavior accompany them.
Our feeling is that they do not.
An increased number of posts, their more timely posting and more higher level postings do not necessarily mean that
students conduct a more substantive conversation or interaction. It might only be that students nd it easier to respond
to peers by building on others comments. Thus no guarantee of more interaction among students can be claimed. It also
may happen that students reply to each others posts without any interaction or communication. It was frequently ob-
served, while sifting through the data, that two students could comment on each others posts without actually engaging
in a conversation.
In-depth conversations rarely seemed to take off. No conversations made it to Level 5. Very few students made Level
4 postings and only, on average, one-in-four, made a Level 3 posting per discussion. There was hardly any conversation
going on beyond course requirements. Students made the required postings and moved on to the next assignment. There
is little evidence that they returned to conversations after they had completed required postings.
Still, while performing data analysis, the authors noted several positive aspects of the online discussions under
study. The total number of replies remained consistent throughout the semester in both deadline groups. Students in both
groups began the semester by dutifully replying to the teacher prompt and their fellow students well before the deadline;
nding a way to sustain is important in fostering online discussions. Students initial responses (Level 1) were typically
quite wordy, averaging about 350 words, adequate to address the prompt and provide thoughts for further discussion.
Late Level 2 and higher level responses did not always mean students submitted their required postings late but showed
that some simply wanted to respond to their classmates more even after the deadline; they had not moved on before fol-
lowing up.
While the focus of this study has been on student postings, no analysis of any course can be complete without con-
sidering the instructor. Table 4 shows data by instructor group and lists posts made by instructors as 26, 102, 67, 10, and
34. The wide range of postings, one forum containing 30% instructor posts (Fall 2009 Group 2) and another containing
only 3% (Spring 2010 Group 2) indicates the differences in instructor presence among the groups. Some instructors took
a much more hands-on approach, being totally immersed in the conversation; others were more Spartan in their com-
ments. Despite the scripted and pre-packaged nature of Computers in the Classroom, differences in teaching style were
evidenced simply by the number of instructor posts made to the online discussion forum. For example, the instructor in
Spring 2010 Group 2 posted only 34 messages, a number not considered high compared to Fall 2009 Group 2 and Group
3 instructors. However, Spring 2010 Group 2 produced the highest number of L3 (38) and L4 (4) postings among all ve
groups. Teacher presence is not a simple thing; differences in teaching style most probably cause far more differences
between groups then the truly simple change of deadline protocols.
Table 5. Number of student posts by level and number of instructor posts. Summative data for all seven discussions ex-
cept * which only has data from fve discussions.
Group n Level 1
15 113 162 23 2 0 300 26
16 105 194 32 2 0 333 102
10 66 115 15 0 0 196 67
Group 1*
Total 72 459 837 121 10 0 1427 239
Online discussion forums are an extremely important part of asynchronous online courses. Getting them right is
not a simple task. The authors of this study noticed that students posted too close to the deadline to allow time for a dis-
cussion to develop. They proposed and implemented a simple solution, two deadlines. Statistically signicant changes in
outcomes were observed but neither changes in student attitudes nor achievement were measured in this study. Measur-
ing such a complex human endeavor as conversation with a few easily observable metrics is an over simplication.
This study used a quantitative approach to look at how one change affected an asynchronous online discussion. Fur-
ther research should include both quantitative and qualitative studies into all the facets of online discussions, both in iso-
lation and as integral parts of the total course design. The authors of this study have discussed amongst themselves many
ways to look at the 1666 posts in their data set and proposals for future research. Ultimately, the goal is to determine how
best to facilitate an online conversation so students learn from it.
An, H., Shin, S., & Lim, K. (2009). The effects of different instructor facilitation approaches on students interactions during
asynchronous online discussions. Computers & Education, 53(3), 749-760.
Baran, E., & Correia, A. (2009). Student-led facilitation strategies in online discussions. Distance Education, 30(3), 339-361.
Chang, C. (2008). A case study on the relationships between participation in online discussion and achievement of project work. Jour-
nal of Educational Multimedia and Hypermedia, 17(4), 477-509.
Cheung, W. S., Hew, K. F., & Ling Ng, C. S. (2008). Toward an understanding of why students contribute in asynchronous online dis-
cussions. Journal of Educational Computing Research, 38(1), 29-50.
Friedman, R., Elliot, N., & Haggerty, B. (2010). E-Learning in undergraduate humanities classes: Unpacking the variables. Internatio-
nal Journal on E-Learning, 9(1), 51-77.
Gilbert, P. K., & Dabbagh, N. (2005). How to structure online discussions for meaningful discourse: a case study. British Jour-
nal of Educational Technology, 36(1), 5-18.
Hew, K. F. and Cheung, W. S. (2008). Attracting student participation in asynchronous online discussions: A case study of peer facili-
tation. Computers and Education, 51 (3), 1111-1124.
Meyer, K. (2008). Do rewards shape online discussions?. Journal of Interactive Online Learning. 7.2, 126-138.
Richardson, J. C. & Ice, P. (2010). Investigating students level of critical thinking across instructional strategies in online dis-
cussions. Internet and Higher Education, 13, 52-59
Wise, A., Saghaan, M., & Padmanabhan, P. (2009). Comparing the functions of different assigned student roles in online con-
versations. Paper presented at the World Conference on E-Learning in Corporate, Government, Healthcare, and Higher
Education 2009, Vancouver, Canada.
Calculus in Elementary School: An Example of ICT-based Curriculum Transformation 155
Calculus in Elementary School: An Example of ICT-based Curriculum Transformation
ANDREW E. FLUCK, DEV RANMUTHUGALA, CHRIS CHIN, AND IRENE PENESIS
There has been speculation that children using computers can learn concepts at younger ages than hitherto. Seymour
Papert suggested this would be possible in the area of mathematics as follows:
If you examine the current math curriculum with an open mind at least 80% of it will be recog-
nized as inferior to the examples of mathematical thinking that are made possible by these three
1. free access to computers for much more than a few hours a week
2. a level of technological uency (that has to develop over years) equal to the levels of
reading uency we now regard as basic skill
3. freedom from having to pass tests on nineteenth century knowledge
Many topics that were unteachably abstract in the context of pencil and paper technologies will be
considered as appropriate for children in the context of a digital technology that makes the previ-
ously formal become concrete. (Papert 2000).
Mitch Resnick demonstrated this idea for the acquisition of systems concepts (1998), and we set out to do so in the
area of integral calculus. After the 2007 Australian Federal election, the rst item on the new cabinets agenda was com-
puters in schools. This evoked great media interest, with the Australian Broadcasting Corporation airing the issue. Mary
Gearins interview featured the following comment:
Well, I actually think that this is [Prime Minister] Rudds education Trojan horse. Getting laptops
into the classroom forces the teachers to think about their own skills and their own training and
it forces the educational administrators to think about the curriculum and in the end it will force
them both to change and that, I think, is the real goal (Pesce 2007).
Principals interviewed on the same program expressed their expectations that the Rudd-Labor digital education ini-
tiative (Rudd, Smith & Conroy 2007) would give a real vision for what education might be in the future, and move the
education system forward. The Australian Government committed to a Digital Education Revolution with a focus on
schooling in Years 9-12 where students nationwide were to be provided with computer access throughout every school
day. The government was concerned about lethargic transformation based on information and communication technology
(ICT) in education: while ICT has fundamentally reshaped whole industries, revolutionized production processes and
generated massive improvements in productivity in our workplaces, our education systems have been slower in adapting
(Gillard 2008). Society is being transformed by computers, and learning needs to adapt accordingly (Gosper et al. 2008).
This makes it a problem of national signicance. To rise to the challenge, the research community can, and should, of-
fer some insights into what school pupils can realistically achieve with these new tools. Australian institutions aspire to
systemic transformative uses of educational computers (Fluck 2003). Professor Barry McGaw was appointed Chair of the
new Australian Curriculum, Assessment and Reporting Authority (ACARA) to develop a world-class curriculum for all
K-12 Australian pupils. In this endeavor ACARA needs to draw upon practical ways to transform education using ICT.
This is not ICT literacy, nor is ICT solely a support for other learning outcomes already identied in school curricula.
ACARA needs to examine learning outcomes which can only be achieved through pupil use of ICT. Australia desperately
needs clear directions on what teachers can achieve in this area.
156 Fluck, Ranmuthugala, Chin and Penesis
In addition to this curriculum renewal, Australia has also been facing a critical shortage of engineering, science, and
mathematics graduates (Goldsworthy 2008, p. 2). The shortage has been most acute in engineering with declining enrol-
ments. In Australia only 10% of graduates in 2008 were in engineering; the OECD average was 14% and in China 40%
were in engineering. The National Centre for Maritime Engineering and Hydrodynamics was undertaking a research
project to identify the effect of school pupils perceptions of engineering and technology careers on university enrolment.
Our project supplemented this research, and thus addressed a major issue still faced by many Australian universities.
Integral calculus is key to a career route into the engineering profession, and hence our choice for demonstrating a trans-
formation in curriculum.
The crucial transformational role of ICT in schooling has been underlined by the four types of use identied for the
Australian government in Making Better Connections:
Type A: encouraging the acquisition of ICT skills as an end themselves;
Type B: using ICTs to enhance students abilities within the existing curriculum;
Type C: introducing ICTs as an integral component of broader curricular reforms that are changing not only how
learning occurs but what is learned;
Type D: introducing ICTs as an integral component of the reforms that alter the organization and structure of school-
ing itself (Downes, Fluck et al. 2002, p.23).
This project clearly addressed Type C changes, since it radically reconsidered what kind of mathematics might be
taught at a far earlier age than hitherto. The power of this exemplar is that it will make necessary a re-consideration of
all areas of the curriculum for relevance in a digital age. ICT and associated skills are seen as strategic, and nationally
important for economic, pedagogical and social reasons (Hawkridge 1989). If ICT use in schools is restricted to Types A
or B, then the full benet of high technology investment in schools is limited. However, for uses corresponding to Types
C or D, school cultures will need to change markedly. This transformational view of ICT in schools requires a rethink
about curriculum content, the applicability of prior learning outcomes and criteria to the future lives of pupil, and even
the structure of schooling itself (Fluck 2003; Fluck 2005; Tinker 2000).
Some major reports into the efcacy of ICT as a support for pupil attainment in numeracy and literacy show it
can be limited when used within an inappropriate curriculum. Dynarski et. al. (2007) conducted a large scale scienti-
cally based study of software efcacy aimed at improving print-based reading and mathematics. The ndings showed test
scores using conventional items were not improved by the use of this tutorial-style software. Work for the Australian Re-
search Council came to similar conclusions about the minor impact of ICT in improving learning outcomes in traditional
curricula (Robertson & Fluck 2006). Parr (2000) showed ICT is about as effective as other methods for improving educa-
tion, such as decreasing class sizes, when assessed using non-ICT based tests.
The project focused on teaching integral calculus to school pupils aged ten to twelve years through the use of appro-
priate computer software and delivery techniques relevant to their experiences. For those not familiar with the MAPLE
software we intended pupils to use, its functionality can be summarized as follows. The software can:
accept input of mathematical functions;
manipulate such functions and solve equations algebraically;
perform calculus operations such as differentiation or integration;
calculate the value of an integral between given limits;
graph and visualize function and solution.
In short, the software removes the need to memorize dozens of integration techniques and accurately calculates the
value of a denite integral for a given range. The efcacy of similar software has been demonstrated at undergraduate
level, reducing the time to learn calculus by half (Palmiter 1991).
The basic methodology was an intervention study involving one class (about 25 students) in each of four government
schools. Government schools in Australia educate sixty six percent of pupils, (Australian Bureau of Statistics 2009).
These schools were drawn using purposeful sampling from Tasmania, Victoria, New South Wales and Queensland where
the targeted pupils had already been assigned individual laptop or similar small netbook computers. Facilitators were
trained by the investigators to deliver the mathematical concepts with the aid of MAPLE software and our new tech-
niques. The intervention consisted of eleven one hour periods of intervention instruction from a local facilitator working
in tandem with the regular teacher. A nal assessment of achievement and understanding (another hour, making 12 in
total) culminated the intervention and provided data for analysis.
Our rst stage was a proof of concept demonstration. There is a general perception among teachers and pupils that
integral calculus is one of the more challenging concepts to teach and understand within the school curriculum. Many
carry this perception into university, with a number of undergraduates nding it difcult to master integral calculus in
their rst year (Galbraith & Haines 2000, p 651), which is possibly due to difculties they experienced in school. The
reason for the latter possibly stems from their inability to relate the concepts to everyday life, compounded by the dif-
culty in grasping the associated mathematical manipulations. It was therefore important to identify an issue that is rel-
evant to these pupils, which links to integration, and attempt to pursue this only up to a level appropriate to their skill lev-
el. Our new approach concept was initially tested on a small number of eleven year old pupils in Launceston, Tasmania.
This was considered extremely important for the success of the project as the selected topic should be seen as relevant
to the pupils, while the mathematical concepts and the associated manipulations were found to be within their skill level.
We then rened the teaching sequence. Our new approach uses real world problems best solved using integral calcu-
lus. Therefore each problem contains a scenario from which a function equation can be derived and an area or rotational
volume deduced. It was important pupils saw the relevance of each scenario to their lives, thus catching their interest and
providing engagement with learning of the relevant mathematical concept. An example problem addresses painting a
mural on their bedroom wall. Pupils chose a curve as the boundary, and then calculated the integral to nd the area to be
covered and thus the amount of paint required. We rened our problem series in line with the non-template problem solv-
ing method of Allen (2001) and a realistic mathematics education approach (Gravemeijer & Doorman 1999) with the aid
of a teacher advisor to ensure it ts pupils in the targeted age range. Pupils would utilize the software tool MAPLE (ver-
sion 13 was installed on their personal computer) for algebraic manipulation and calculation of denite integrals but the
problem series ensured they also mastered relevant concepts.
Next, we trained the local facilitators. This stage consisted of recruiting and training the facilitators for each of the
four schools to carry out the intervention delivery. Given the time and resource limitations, we brought them to Laun-
ceston for a full day of training and familiarization. During this period we explained the project, provided training on
MAPLE, and provided instructional sessions on how to use our problem series and Allens non-template problem solving
method to teach the underlying concepts and their application. For example, the explanations covering the equation of a
curve and its integration must be in terms that the pupils can understand and relate to.
The fourth stage was the intervention conducted at the four schools. The facilitators (who were sometimes the regu-
lar teacher) had access to the target classes for two hours per week over six consecutive weeks. During this period, the
facilitators gradually introduced each problem, the mathematical concepts, the software tool, and ways to solve the prob-
lem using it. For the integral problem dealing with the area of a surface bounded by a curve (the mural problem), the
process was presented as follows. In the rst phase the facilitator used the problem to introduce the concepts and solu-
tion techniques related to curves and equations. The next phase introduced integration and its solution. The facilitators
guided the pupils through the solution process, but were also able to ensure they imparted the underlying concepts to the
pupils. During this intervention stage one of the researchers visited each school to ensure that the program was on track
and the required outcomes were being met. An intervention lesson was observed and focus group interviews conducted
with pupils where possible. The research team also monitored the progress at each school via telephone conferences and
email communication with the facilitators.
Finally, pupils undertook a test based on questions drawn from rst year engineering calculus examination papers to
assess their knowledge and skills. The questions were provided on a printed sheet of paper, and a template for working
and answers was provided as a MAPLE le. This le did not contain any of the mathematical material, so pupils were
required to demonstrate their capacity to input mathematical functions through the MAPLE user interface. Some of the
questions are shown in Table 1.
Table 1: Example questions from the culminating assessment task.
The gure shows a sail at-
tached to a mast of height
9 metres. Calculate the
area in order to get enough
material to make the sail.
Integrate the following
between 0 and 2.
The gure shows a drive-
way to a house. Calculate
the area between the
curves to order enough
concrete.
The following is a plan of
a playground that needs
woodchip. Calculate the
area of the playground.
(Hint: Split the area into
separate sections.)
To provide consistency, facilitators e-mailed the MAPLE documents produced by pupils in the culminating assess-
ment to the research team for marking. These were printed, marked and returned to the facilitator with a community
report for reproduction in the local school newsletter. Table 2 contains a summary of the class and school characteristics
and the mean test scores.
Table 2: Demographics, location, school advantage and mean test score for project classes.
State Males Females Youngest Oldest Location ICSEA* Mean
NSW 13 16 10.2 13.3 Metropolitan 993 64%
QLD 14 14 11.5 12.7 Metropolitan 1099 83%
TAS 12 12 12.6 13.5 Provincial 982 66%
VIC 8 8 11.4 12.2 Provincial 988 64%
*Index of Community Socio-Educational Advantage (ICSEA): The mean ICSEA value is 1000 with a standard deviation
of 100. Values below the mean indicate schools with fewer advantages.
Unsurprisingly, the mean score of the more advantaged school was higher, but this is an expected result. More im-
portantly, the mean scores are all above the 50% required to pass the engineering examination, and only eleven of the
ninety seven pupils scored below this level. The greatest age range was in School 1 but the correlation between age and
score was virtually zero (<0.1). Two of the youngest students in this class scored above 75% and the eldest scored below
50%. The results are very encouraging. These pupils working with computers were not selected for their academic prow-
ess and came from schools with average or lower socio-economic advantages. Their performance was similar to that of
students some eight or more years older selected for a university engineering course, but who are tested in a conventional
way (using handwriting only).
If the project had set out to show ten year old pupils at a government school could learn integral calculus as well as
rst year university engineering students, it succeeded. The research team feels that our original intent to show learning
of complex concepts at a younger age through ICT has been fully vindicated and Resnicks work in systems has been
repeated in this area of mathematics.
In addition to the quantitative results we also examined the interviews with pupils and their responses to the question
what is calculus good for? on the test paper. Pupils at School 1 showed a good understanding of calculus, stating it was
useful because:
To work out maths for work or school. It can be used for working out how many kilometers to somewhere
or to measure a garden.
Because it helps you measure and calculate the distance and area of places. eg. how far you throw a ball.
It is useful because you can go home and do calculus. Went [sic] you start it looks hard.
These pupils used 15 screen Acer and Dell laptops nanced by the school and by parents and shared their work at
home. They celebrated their learning because they were able to outwit Mum & Dad or more seriously, measure my
house - gure out how much building material went into it. They enjoyed graphing functions (see Figure 1).
Figure 1: Graph produced by a pupil at School 1. Figure 2: Cup designed by a pupil using MAPLE
software.
School 2 pupils showed a good understanding of calculus, stating it was useful because:
It helps us work out real life shapes and equations. You can work out the area of a swimming pool or a
skateboard ramp.
It is helpful because you can use it in real life situations. Its good software and lets kids know what you
will be using if you become an engineer. If someone knows how to work with all the expressions and what
they mean, MAPLE 13 can be really easy.
One girl said My dads an architect, so I showed him MAPLE. He had a go with it. He said he wished hed had it
when he was studying! My mum doesnt like it because she doesnt understand it. And shes a teacher at this school!
Students went on to design mathematical cups using the software (see Figure 2).
In School 3, pupils liked calculus because:
you can work out the area of a building or something like that.
it can help you gure out hard questions easily.
some of the questions look hard but if you use this its really easy.
for when you get older and you might get calculus in college or university.
It helps you understand things better and using MAPLE it was easier than I thought it would be.
if youre painting a house or a skate ramp it tells you how much paint or whatever it is you need to get. It
can save you money and time.
Pupils in School 4 said:
Integral maths is useful because it can help measure things that are curved. It can also be used to measure
things that are on different angles on different areas. It can be useful for example to nd out how much
cement you need to make a skate ramp.
I nd MAPLE fun to learn and I would recommend it to all the schools. P.S. Its a good way to get out of
maths!!
Pupils took their netbooks home and shared what they were doing. Some had shown it to friends, and seven had
shown it to parents. A few had used MAPLE to do their other homework and saw the potential of using it for other stud-
ies (especially homework) and applications. Students had found new mathematical functions on their own and most felt
they had learnt new things through the project, such as cubic equations and graph plotting.
This project was timely and urgent. This project has demonstrated that ICT can help pupils go beyond the unteach-
ably abstract (Papert 2000) and thus experience curriculum transformation. It has shown what a digital curriculum could
look like, what sorts of learning outcomes can be addressed, and how this might be done. As a nation we cannot keep
tacking computers onto existing learning goals, but we need to revolutionize the teaching process (Gillard 2008) using
methods like this. Frankly, it is obvious that keyboarding is unlikely to improve handwriting. This is not an argument to
discard existing curriculum outcomes, but to envisage new ones so that choices can be made. Schools can then choose to
mix handwriting with computer-based skills of keyboarding and voice recognition dictation. They may also want to in-
clude in curricula new ICT-based learning outcomes in computational chemistry, weather forecasting and other learnings
which cannot be realistically achieved without a computer.
Furthermore, this was a signicant project because it linked ICT-based transformation with the development of a
love of learning mathematics as it is used in science and engineering. With our new approach showing how integral cal-
culus can be used to solve practical real world problems, we used scientic and engineering examples to engage students.
By promoting an interest in the mathematics which lies at the heart of these difcult disciplines, we hoped to address the
shortage of engineering, science, and mathematics graduates. This long term aim is yet to be fully realized, because it
will take several years for a critical mass of schools to adopt our methods and for their pupils to apply to university. Our
method is innovative, because we implemented the project with ICT equipment already available in the nal years of el-
ementary schooling, but we are hopeful that this early intervention is likely to be an effective intervention.
The implications for curriculum design are salutary. Australia is adopting a nineteenth century curriculum in the
emerging face of a twenty-rst century paradigm shift. The debate about how to leverage pupil access to personal com-
puters needs to take the potential demonstrated by this project into account.
We intend to carry the project further in three ways. Firstly, we will analyze our data to a greater extent to examine
any signicant age and gender related factors through the use of Rasch methods. Secondly, we will repeat the project in
a fth school using Maxima, free and open-source software that has similar functionality to MAPLE. However, this is
causing some debate amongst the project team, because Maxima does not have a user interface for input with convention-
al mathematics notation. We know that pupils can use computers to solve problems with integral calculus but will this
learning be affected by the use of a non-traditional notation system? Finally, we will extend the work by delivering and
assess other advanced concepts beyond integral calculus through improved techniques based on the ndings of this proj-
ect. Our further investigations will probe this aspect with the intent of making our methods more accessible to schools.
Allen, D. (2001). Learning integral calculus through non-template problem solving. Primus: Problems, Resources, and Issues in
Mathematics Undergraduate Studies 11(2) 147160.
Australian Bureau of Statistics (2009). Schools Australia 4221.0. Canberra: Author. Retrieved from http://www.ausstats.abs.gov.
au/ausstats/subscriber.nsf/0/05ADAC0812C70C9DCA25775700218CA4/$File/42210_2009.pdf on 17 October 2010.
Downes, T, Fluck, A, Gibbons, P, Leonard, R, Matthews, C, Oliver, R, Vickers, M, & Williams, M. (2002). Making Better Con-
nections. Commonwealth Department of Education, Science and Training.
Dynarski, M., Agodini,R., Heaviside,S., Novak, T., Carey, N., Campuzano, L., Means, B., Murphy, R., Penuel, W., Javitz, H.,
Emery, D, and Sussex, W. (2007). Effectiveness of Reading and Mathematics Software Products: Findings from the First
Student Cohort. Washington, D.C.: U.S. Department of Education, Institute of Education Sciences.
Fluck, A. (2003). Integration or Transformation? A cross-national study of ICT in school education. University of Tasmania:
PhD Thesis. Retrieved from http://eprints.utas.edu.au/232/02/02whole.pdf on 9 February 2007.
Fluck, A. (2005). The realities of transforming education through ICT, invited keynote at Global Project Based Learning Forum
and Exhibition, Kaohsiung, Taiwan, 29-35 (Eng).
Galbraith, P. and Haines, C. (2000). Conceptual mis(understandings) of beginning undergraduates. International Journal of
Mathematical Education in Science and Technology 31, 651678.
Gillard, J. (2008). Address to the Australian Computers In Education Conference, Canberra, 1 October 2008. Retrieved from
http://www.deewr.gov.au/Schooling/DigitalEducationRevolution/Documents/ACECMinistersSpeech_011008.pdf on 17
March 2008.
Goldsworthy, A. (2008). Submission to the review of the national innovation system. Fitzroy, Victoria: Business/Higher Educa-
tion Round Table Ltd. Retrieved from http://www.bhert.com/publications/submissions/2008-04-InnovationReview.pdf on
10 January 2011.
Gosper, M., Green, D., McNeill, M., Phillips, R., Preston, G. and Woo, K. (2008). The impact of web-based lecture technologies
on current and future practices in learning and teaching. Australian Learning and Teaching Council. Available http://www.
cpd.mq.edu.au/teaching/wblt/overview.htm on 16 March 2009.
Gravemeijer, K. and Doorman, M. (1999). Context problems in realistic mathematics education: a calculus course as an exam-
ple. Educational Studies in Mathematics 39, 111-129.
Hawkridge, D. (1989). Machine-mediated learning in third-world schools. Machine-Mediated Learning, 3, 319-328.
Palmiter, J. R. (1991) Effects of Computer Algebra Systems on Concept and Skill Acquisition in Calculus. Journal for Research
in Mathematics Education 22 (2) 151- 156.
Papert, S. (2000) Technology in schools: To support the system or render it obsolete. Milken Family Foundation. Retrieved from
http://www.mff.org/edtech/article.taf?_function=detail&Content_uid1=106 on 17 October 2010.
Parr, Judy M. (2000) A review of the literature on computer-assisted learning, particularly integrated learning systems, and out-
comes with respect to literacy and numeracy. Auckland, New Zealand: Ministry of Education. Retrieved from http://www.
minedu.govt.nz/index.cfm?layout=document&documentid=5499 on 3 November 2002.
Pesce, M. (2007) Transcript: ALP education revolution in the spotlight reporter Mary Gearin. Broadcast 4Dec07, Australian
Broadcasting Corporation.
Resnick, M. (1998) Technologies for Lifelong Kindergarten. Educational Technology Research & Development, 46(4).
Robertson, M. and Fluck, A. (2006) Children online learning and authentic teaching skills in primary education - nal report.
Department of Education, Science and Training.
Rudd, K., Smith, S. and Conroy, S. (2007). A Digital Education Revolution. Retrieved from http://www.alp.org.au/download/
now/labors_digital_education_revolution_campaign_launch.pdf on 31 December 2007.
Tinker, R. (2000). Ice machines, steamboats, and education: structural change and educational technologies. Paper presented
at The Secretarys Conference on Educational Technology, September 11-12. Retrieved from http://www.ed.gov/rschstat/
eval/tech/techconf00/tinkerpaper.pdf on 31 October 2006.
Green Training: Chronicling the Reuse of Government Healthcare Instruction 163
Green Training: Chronicling the Reuse of Government Healthcare Instruction
DAVID TWITCHELL, MITCHELL J. SEAL, AND CHRISTOPHER G. LYNCH
Atop the mountain sits a Guru and a corporate training director.
Training Director: (slightly out of breath, hands a blade of grass to the white haired old man) What is
Green training?
Guru: Hum, (holding up the blade of grass) it is conserving resources through reuse
Training Director: Oh, like taking your own cloth bags to the grocery store?
Guru: (after long pause staring into the distance) Yes, but on a grander scale. Imagine building a tunnel
to get you easily to your destination on the other side of the mountain; and then another tunnel when you
make a return trip.
Training Director: Thats ridiculous! No one would ever do something like that.
Guru: (with just a whisper of a smile) Au contraire, it happens every day.
The Guru is right. (Arent Gurus always right? Isnt that why they are called Gurus?) Most training materials are
not recycled in whole or part. This is so for two reasons; rst the training is not engineered for reuse and secondly, the
infrastructure is obstructive. Have you ever heard of shared training that seriously is worth the time and effort? Too
often it is like driving your stack of newspapers six miles (one-way) to deposit them in a recycle dumpster; where the
cost of gas outweighs any benet that might be gained by recycling your paper. Because the value proposition is not per-
ceived, or is not understood, most training is built virtually from scratchwithout a second thought. Government train-
ing is no exception. However, reusing training is not just the right thing to do it also makes good business sense.
WHAT IS GREEN HEALTHCARE TRAINING?
From the shared training perspective there are only two kinds of training: that designed for sharing and that, that
was not.
For training not designed for reuse, training products would, if possible, need to be deconstructed for maximum
reusability or just shared as is. In this digital age any training asset can exist in multiple forms. Consider a complete
packaged course that can be composed of a dozen lessons; each lesson can be composed of a number of content objects
and each content object composed of assets such as: still images, ash animations, text blocks, sound les, etc. Outside
of the intended audience, an intact course may only have utility for a few users; but when it is disassembled reusability
growsmore users are interested in one or more lessons from the course and still more users would be able to utilize
single content objects; but for maximum reuse it is easy to see that we need to get to the asset level.
This reusability can be engineered or serendipitous (reverse-engineered). To be most accessible content should be
able to be used as the lowest level of aggregation in addition to every congured aggregation level above.
The idea of recycling training is not new. Reusable chunks of training became known as Learning objects around
1990. Recently these objects have been dened as Sharable Content Object (SCO).
Many millions of dollars in train-
ing are built each year to standards that facilitate sharing at all levels.
Chiappe dened Learning Objects as: A digital self-contained and reusable entity, with a clear educational purpose,
with at least three internal and editable components: content, learning activities and elements of context. The learning ob-
12 See: www.ADLnet.gov
164 Twitchell, Seal and Lynch
jects must have an external structure of information to facilitate their identication, storage and retrieval: the metadata
(See Chiappe, Segovia, Rincon, 2007).
Prior to that, learning objects were dened as: any digital resource that can be reused to support learning
(Wiley, 2000). In 1998-99 I worked with a team on a government funded project (Bala, Twitchell, Zhang, 1998) titled:
Intelligent Teaching and Instructional Design/Authoring (ITIDA). The objective was to design an articially intelligent
instructional design and development system that would be facilitated through the use of Instructional Objects. These
objects were essentially independent content chunks enabled with a presentation or interaction strategy. The presentation
of these chunks was facilitated through an equally intelligent presentation system. About that same time the Advanced
Distributed Learning (ADL) group began making signicant progress toward a similar goala model for reusability:
SCORM. Their work has been the foundation for knowledge object granularity. These SCOs form a foundational ele-
ment in the design of reusable courseware. In 1990, M. David Merrill used the term Knowledge Objects to describe a
structure for types of knowledge (See Jones, Li & Merrill, 1990). In 2001 Ellen Wagner stated: A learning object is the
smallest element of a stand-alone information required for an individual to achieve an enabling performance objective or
outcome. It was widely recognized that to make reuse practical would require a content repository and registry and to
make storage and reference systems operable would require robust metadata.
Today, a decade later, there are thousands of organizations that build training content objects, SCOs or learning ob-
jects for reuse or share training as courses or lessons. It just makes sense. We dont need to recreate training over and
over again. However, for every one that has found a way to store, retrieve and reuse training content, in whole or part,
there are a thousand more that have not. This paper explains how the Department of Veterans Affairs has approached
training reuse and has achieved remarkable results. This paper is written to those who are not yet engaged in shared
training and those who are seeking to enlarge their sharing options.
THE IDEAL
Consider a typical scenario that plays out in corporate, government and even academic circles around the globe ev-
ery day: First a training need is validated and objectives are constructed and made operable. Using the objectives a build
vs. buy analysis is conducted. Even though the organization has been doing training for many years previous training is
not usually readily available or if available is not usually usable. Thus, when the analysis is complete the work is gen-
erally constructed with no, or few, existing resources other than the raw training content.
After sharing is made operable the steps are very similar except in the build/buy analysis. It is there that a scan of
internal and external courses, lessons, or objects that might accomplish objectives for the target audience are reviewed.
Lets say, for argument, that the scan yields 2 lessons and 14 objects that are well suited for your audience and objectives,
and that the remaining course development, including transitions between your found content, constitutes 34 percent.
With only 34% remaining to complete the training need your costs are diminished substantially and the time-to-need is
also similarly reduced. But the hidden advantage is that with every new training venture you also add reusable resources
to your repository and the investment reduces future costs more and more over time.
WHAT IS REQUIRED?
An infrastructure that will help facilitate sharing is important but not essential at frst. The frst step is to begin shar-
ing. Demonstrate that it can be done in your organization. Document the savings. There is a culture behind sharing in-
structional resources just as there is an affective culture behind conserving natural resources. You have to have a green
attitude. But let us continue to speak of the ideal. You will need a:
design and development standard
metadata taxonomy
metadata application tool and/or process
registry (catalog and search system)
means of reviewing sharable resources
acquisition process
Let me repeat, you dont have to have all this in place to begin sharing. Eventually, you will need all these ele-
ments, if you are serious about it, but not at rst. For instance the Department of Veterans affairs began an effort to share
training in 2004 with none of the above in place; but the VAs Interagency Sharing Coordinator who was a dedicated,
ingenious and visionary individual dedicated to the necessity of the effort. He began by cataloging the inventory of
training products that could be shared and by forging an agreement with the Department of Defense
and later with other
DoD, IHS, CDC,
FEMA, EPA, DHS,
DoE, HHS, HRSA
DoD, IHS, CDC, FEMA, EPA,
DHS, DoE, HHS, HRSA
DoD, HIS, CDC
DoD, IHS
DoD, IHS, CDC, FEMA,
EPA, DHS, DoE, HHS,
HRSA, OSHA, FBP,
PHS, DHS
DHS, DoE, HHS, HRSA,
OSHA, FBP, PHS, DHS
Figure 10 Programs and Sharing Partnerships
That rst year eighty programs were shared between DoD and the VA for a savings of 1.2 Million dollars. Now
remember this is the rst year of what was to many an experiment. This is predominantly healthcare training which
is fairly standardized by research and best practice models. In year two a memorandum of understanding was created
bringing the Indian Health Service (IHS) into the sharing community. That year 120 programs were exchanged for a
combined cost avoidance of 2.3 Million dollars. In the graph (Figure 1) in 2010 the VA shared 1600 programs across ten
federal sharing partners for a costing avoidance just short of 50 Million dollars. Currently, at the end of the third quarter
of 2011, cost avoidance was just over $79 Million with 2,161 programs shared.
These numbers come from sharing existing trainingtraining content that was not designed or engineered for shar-
ing or reuse; without any real sharing infrastructure. At rst the shared training programs were predominately satellite
and video with little representation from other modalities. The balance shifted to a predominance of e-Learning in 2007
and e-learning has been the predominant media form thereafter. To illustrate: as of third quarter 2011, 1,754 of the 2,161
programs shared were e-learning products. In 2010 there were thirteen federal agencies
within the sharing commu-
13 The Health Executive Council began an effort to encourage collaboration and sharing among DoD and VA organi-
zations. This council is composed of equal representation from the uniformed Services and from the VA. Congress set
up the Joint Incentive Fund to assist in collaboration and sharing by seeding funding through a grant application pro-
cess. To date the VA collaboration and sharing efforts have received four grants with a cumulative total of approximately
$11,000,000.
14 This sharing community consists of eight organizations working under formal sharing agreements and ve work-
nity; this represents a multiplier that also contributes to these numbers. Each agency that adopts and uses extant training
avoids the cost to develop that training.
Qtr)
Figure 11 Cost Avoidance Growth in VA Shared Training Program
Direct Cost avoidance is simply the amount of money that was expended to originally produce the training that does
not need to be expended by the organization receiving the training. The real total cost avoidance is much harder to quan-
tify. You could consider just direct cost avoidanceor you could add to that the indirect costs, like reduced travel and
housing, employees time spent to support contracted development, contract management, time saved, etc. The indirect
costs are wide and varied and very difcult to measure with accuracy. We settled on only reporting direct cost avoidance
because it is a hard documented cost and because it is completely defensible. However there are considerable indirect
costs that could be introduced. These might include the cost of external substitute training, travel related costs, the cost
of employees away from the workplace and customization costs, to name a few.
As you can see we began in 2004, by sharing between ourselves and the US uniformed services and we avoided
costs of 1.2 Million dollars. From there we demonstrated sizable growth, doubling the cost avoidance each year through
2007. With a spike from growth in shared e-Learning products our cost avoidance grew by more than 300% in 2008.
The 2010 numbers show that we are just shy of 50 Million in savings. At the end of 3
quarter (July 1
) cost avoidance
exceeded $79,000,000.
We do not know where this growth will level off; we suspect that growth will continue as sharing takes root and as
we make the process easier and more manageable. Thus far we have not considered collaboratively developed content
content designed and built to meet collective needs, designed and built to be shared and maintained; designed around
ing through less formal understandings. Sharing Partners include the following United States organizations: Depart-
ment of Veteran Affairs (VA), Department of Defense (DoD)(Army, Navy, Air Force), Department of Homeland Security
(DHS), Indian Health Services (IHS), Centers for Disease Control and Prevention (CDC), Federal Emergency Manage-
ment Agency (FEMA), Environmental Protection Agency (EPA), Department of Energy (DoE), Occupational Safety and
Health Administration (OSHA), Federal Bureau of Prisons (FBP), Health and Human Services(HHS), Health Resources
and Services Administration (HRSA), and Public Health Service (PHS).
SCORM, designed for granular use and reuse. In 2011 we have just begin to account for these efforts. As we move to-
ward a more engineered sharing approach and we will see our yearly savings jump again.
We are able to share existing training resources across twelve federal agencies; but the process is clumsy and very manu-
al. Here are the problems that we are now addressing:
There are a wide
variety of LMSs:
Plateau, Saba, Moodle, Ilias; some agencies or organizations have no LMS
at all. When problems arise with a course package, its hard to pin down
whether its the package itself or the receiving agencys LMS.
No simple
fle transfer
mechanism:
VA doesnt have a dedicated FTP server for exchanging courseware. Often,
the same course package will get transferred to multiple agencies in a variety
of different ways, from circuitous internet routs to old fashioned mail. A
related problem also springs from a lack of an effective version control
notifcation process.
No meta-
data attached to
All sharing partners have their own version of an LMS submission form that
needs to be flled in, (e.g.. technical issues [SCORM version, completion data,
bookmarking, etc.] and basic info [audience, keywords for searching, contact
information, etc.]) yet currently there is no centralized way of collecting and
disseminating that information.
No test-drive: Occasionally we go through the diffcult process of transferring a course
package and getting it up on the sharing partners LMS, only to fnd out that it
didnt really meet their needs and their considerable effort was wasted.
No facilitation at
a more granular
Sharing partners will need a way to view the lesson, learning objects, SCOs
and assets in addition to whole, intact courses. This same facility will ac-
commodate the storage and upload of asset, objects and lessons from sharing
partners.
No way to search
There is not an effcient effective way to fnd applicable content at any
granularity. A way to quickly, and effciently, metadata tag at all levels is
essential. If this process is too cumbersome developers and project/product
managers will not take the time.
To address these issues I initiated a proposal and plan in 2006 to care for registry, repository and metadata applica-
tion, (see Twitchell, Bodrero, Good & Burk, 2007) but it has not been until just recently that this proposal has gained
traction and support. We have determined that this system, to be effective, should be independent of any one agencys
servers, especially those behind frewalls or requiring regulated access. It must be open-source so that it is not tied to a
vendor and is both freely distributable and extendable to, and by other agencies. The fundamental architecture has been
slightly adapted from the original proposal but continues to be simple and effcient:
This system is being constructed to be fully operable wherever there is an instance of the system on a server. The
Exchange will be developed in open-source and immediately available to all existing sharing partners. The Exchange is
being built with the cooperation and help of ADL and is intended to be shared as well as to facilitate sharing. However,
the master Exchange will be operated by the Department of Veteran Affairs under the VHA Employee Education Sys-
tems Interagency Shared Training Group (ISTG) to enhance and facilitate the sharing among all healthcare providing
government agencies.
The central features of this architecture include three operations:
(1) Content Repository,
(2) Course Deployment and
(3) Metadata Application.
Role-Based Views
Setup &
Courses Add/
Figure 12 Exchange Architecture
CONTENT REPOSITORY
The content repository consists of a content database for the storage of redundant content packages where content
can be accessed where possible in all confgurations, or level of aggregation, from the course to the asset where courses
are made up of Lessons and lessons are made up of objects and objects of assets. As there has been some controversy
over the size and composition of content objects, we have defned such as the embodiment of enabling objectives where
one single performance is facilitated and measured if possible. This makes for smaller content-objects, but it also facili-
tated what we believe is the most reusable confguration.
COURSE DEPLOYMENT SYSTEM
The course Deployment system facilitates courseware review, testing, and acquisition. This operation employs an
extensive Metadata taxonomy that enables rapid and accurate search and retrieval model. It will also provide a forum for
commenting by interagency sharing partners and subject matter experts, technologists, curriculum planners, etc.
Search utility
Allows agencies to easily locate course materials that are truly relevant to their needs.
Broad, full-text searching as well as detailed search capabilities across the entire metadata record (i.e., simple
search vs. advanced search)
Ensure scalability and speed of searches
Course materials repository
Ensures that all shared training materials are easily accessible from one central location.
Ensure scalability of repository
Provide for full-text insertion and retrieval of metadata records (harvesting) besides what is available through
the rich client
Testing environment:
Allows agencies to fully preview shared training materials to make sure they t their needs.
Full courses can be user-tested
Should deploy both SCORM 1.2 and 2004 courses
Users can search for course content relevant to their interests and add located items to a personalized area
Users can comment on course materials
Administrators can create user accounts and assign courses
Smaller aggregation (lessons, objects, assets) view
METADATA CREATION/EDITING TOOL
The Metadata tool ensures the inclusion and standardization of information that is vital to cataloging and implementing
courses across multiple systems.
User Interface design and development based upon existing mockups provided by VHA/EES
Primarily utilizes VA-designated schema but also can include extended schemas from other agencies
UI form enforces required elements, vocabularies, and data types as much as possible
Allows for templates, where some of the data is prelled based on the template selected; must also contain an
administration system that will allow for creation of templates
Schema validation service to verify XML before storage in online repository
Can run either inside the browser, or as a stand-alone application on multiple operating systems
Data harvesting mechanism for primary standard documents (e.g., LMS submission forms, course brochures,
Can import existing Metadata les for editing purposes
The Exchange system will facilitate sharing and reuse at all levels. Practical application of Course sub-components
such as lessons, learning objects and assets will be both viewable and downloadable providing the appropriate rights are
granted. There are no efforts to disaggregate existing courses to make their sub-components available; however, all new
training efforts will supply assets, objects and Lessons to the repository.
Coordinated Efforts
Integrate the Clearinghouse repository into other existing repositories to allow such things as federated searching.
Package the Clearinghouse system in a way that other government organizations can implement similar systems for
themselves.
The next wave of savings through sharing and reuse of instructional materials will come from intentional premeditated
design efforts. This will occur in three areas:
1. Collaborative development efforts among sharing partners
2. Contracts designed to facilitate sharing and reuse, and
3. Content mining and inclusion
COLLABORATIVE DEVELOPMENT
Among sharing partners collaboration across agencies represents a new paradigm in sharing. Several projects have
demonstrated the creation of a best practice model (see Twitchell & Bodrero, 2008a). Lessons learned from collabora-
tive development provide a workable framework for future efforts (Twitchell & Bodrero, 2008b) and success factors for
the design of sharable training are also available. (Twitchell & Bodrero, 2006a, 2006b)
CONTRACTS WITH SHARING IN MIND
Presently there is no effort to unite the purchasing power of formerly disparate government agencies for purposes
of interagency sharing. Creating a new paradigm for government training contracting will make a signifcant improve-
ment in government healthcare shared training. In the future there will be a growing effort to structure contracts with
government-wide use rights rather than department or sub-organization rights only. This will make contracted training
immediately sharable across agencies. But the ownership and rights clauses also will be modifed to provide for use and
sharing at the most granular levels. In essence government training will be entirely government property allowing any
agency to share, revise, and reuse at all levels; and an Exchange system will help this become operable at a large level.
Many contractors complain that this has not been their business model and that if the retention of proprietary rights
disappears and if all rights are granted to the government as a whole (who paid for them) then they would need to raise
their pricing to ensure a solvent business model. This may initially be the case, but this is a new era of open-source and
sharing and it is past time for responsible resource management of this sort. Eventually those businesses who adapt their
business models will survive and prosper while those who dont will need to adapt in other ways.
CONTENT MINING
The idea of content mining and inclusion is also not new. This is a process of acquiring content through the min-
ing of owned resources. It has been done on a small scale as a practical means of obtaining relevant usable resources for
training ventures. The suggestion here is to take this effort to an institutional level. The idea of inclusion is to tack-on
shot lists to video shoots and to keep a log of needed audio resources to fll in when professional resources are available
for other work. Presently a scenario is needed for an e-learning course; actors are auditioned and hired for a day rate.
When the shoot occurs we fnd that we have a scene set up with an actor doctor and nurse in place and three hours re-
maining on the contract. When a shot list has been pre-prepared the actors and scene can be used to fll orders on the list.
Recently, we had such a video shoot and found that in two hours we acquired more than 600 useful still images for our
asset repository.
VAS UNIQUE POSITION
Healthcare is a discipline that is widely governed by research and best-practice models. The whole industry, by
in large, abides by the same research foundation and standards. This fact makes training particularly ideal for sharing.
The Department of Veteran Affairs is in a unique position to facilitate shared healthcare training. Not only is the VA the
premiere healthcare provider in the government, it is the largest healthcare provider in the world. The VAs homegrown
patient record system is the envy of nations and the model for patient records management around the world. As such,
the VA can be very infuential in facilitating and promoting shared training; but there are clearly other advantages. For
instance, the VA is affliated with over 800 medical schools in the United States and abroad. These schools offer another
tremendous untapped source for training resources (assets objects, lessons) and the sharing of training. If each of these
educational centers were to both contribute to the content repository as well as utilize training courses from such a sys-
tem, the cost avoidance would grow exponentially.
In conclusion, tools and technologies are beginning to facilitate the sharing of training at a whole new level. More
tools are on their way. But tools and systems are not all that is needed, there must be attitude. Thinking green is a
mindset. It is an attitude of conservation and effciency; but it is also an attitude of economybottom line. To be most
effective there is an optimal fundamental infrastructure, but there is also a culture that must be cultivated and grown; it
will take both.
Bala J., D. Twitchell, J. Zhang, (1998) Instructional Authoring System Development for Networked Environments, Final Re-
port submitted to the US Department of Education, Contract #: RW-97-076144, March 1998.
Chiappe, Andres.; Segovia, Yasbley; Rincon, Yadira (2007), Educational Technology Research and Development, Boston: Springer,
pp. 671-681, (http://andreschiappe.blogspot.com/2007/09/que-es-un-objeto-de-aprendizaje-what-is.html).
Jones, M. K., Li, Z., & Merrill, M. D. (1990) Domain knowledge representation for instructional analysis, in Educational Tech-
nology, 30(10, 7-32.
Twitchell, D., and Bodrero, R., (2008) Collaboration Produces Blended Learning, ISPI THE Performance Improvement Confer-
ence 2008, New York City NY, 7 April 2008
Twitchell, D., and Bodrero, R., (2006) Collaborative Development of Sharable Healthcare Training, Distance Learning 2006,
Madison WI, 4 August 2006
Twitchell, D., and Bodrero, R., (2006) ISPI: Critical Success Factors: Design and Development of Sharable Training, eLearning:
The World is Your Classroom, Boise ID, 16 September 2006
Twitchell, D., and Bodrero, R., (2008) Web-based Pharmacy Technician Training: A Case Study in Reusability, VHA/DOD Fa-
cility Based Educators Conference, San Antonio TX, 28 May 2008
Twitchell, D., Bodrero, R., Good, M., and Burk, K. (2007). Overcoming Challenges to Successful Interagency Collaboration,
Performance Improvement:, Vol. 46, no. 3, March 2007.
Wiley, David A. (2000), Connecting Learning Objects to Instructional Design Theory: A Denition, A Metaphor, and A Tax-
onomy, in Wiley, David A. (DOC), The Instructional Use of Learning Objects: Online Version.
Authentic e-Learning in a Multicultural Context: Virtual Benchmarking Cases from Five Countries 173
Authentic e-Learning in a Multicultural Context:
Virtual Benchmarking Cases from Five Countries
IRJA LEPPISAARI, JAN HERRINGTON, LEENA VAINIO, AND YEONWOOK IM
Supporting quality authentic learning is arguably a key factor in online learning establishing a rm foothold in high-
er education. With more ubiquitous use of social media, students have limitless opportunities to access knowledge sourc-
es and interact with experts. Working in innovative borderless learning environments and increasingly networked infor-
mation societies requires new kinds of teacher competence and an entirely new educational approach to be able to meet
these contemporary challenges. Working with intercultural and international students groups requires careful attention to
the internationalisation of teaching methods.
What kind of learning and what types of teaching methods are needed for a learner to genuinely develop the exper-
tise that will be required in the future? The national evaluation conducted by the Higher Education Evaluation Council
in Finland raised the challenge of strengthening authenticity in online education at universities of applied sciences (Lep-
pisaari, Ihanainen, Nevgi, Taskila, Tuominen & Saari, 2008). Traditionally, knowing and doing have been differentiated
in education (Resnick, 1987). Online educational forms often emphasise knowing more than doing. The challenge is to
integrate doing in authentic environments more fully within online education. Teachers want to improve performance in
implementing working life oriented teaching to respond to the needs of rapidly changing, globalised and multicultural
working life. Studies have also indicated that higher education students feel learning is meaningful when it is linked to
authentic and realistic contexts and problems (Herrington & Herrington, 2006; Saari & Leppisaari, 2008).
On the web, social media tools enable students to engage in interaction with peer learners, teachers and working life
experts irrespective of physical locationlocal or international. Students and teachers thus have an opportunity for col-
legial or multi-professional collaboration and an opportunity to collaboratively construct knowledge and solve common
problems and progress towards a shared and disseminated multidisciplinary expertise. Social software, like Web 2.0,
enables people to collaborate through computer-mediated communication and to form learning communities in which
they construct and share knowledge. Purposeful learning communities emerge when individuals share common interests
(Jonassen, Howland, Marra & Chrismond, 2008; Hakkarainen, Palonen, Paavola & Lehtinen, 2004.)
In this paper, we describe outcomes of projects implemented in the Virtual University of Applied Sciences network
that have formed the basis for operational models and tools for virtual peer development of authentic learning (Her-
rington & Oliver, 2000), where teachers were able to evaluate authenticity in their own or a colleagues teaching and
together update their performance. The research context is the Finnish Online University of Applied Sciences (FOUAS),
a virtual cooperative and expert network established by universities of applied sciences (UAS). In 2009-10, an authentic
e-learning development project (the IVBM Project) was carried out in the FOUAS, in which a virtual benchmarking ap-
proach supported development of teachers online pedagogic skills. The model has been evaluated in articles by authors
(2009; 2010). In this paper, we examine the application of design principles of authenticity in the benchmarking material.
ELEMENTS OF AUTHENTIC LEARNING AS EVALUATION CRITERIA
Nine elements of authentic learning proposed by Herrington and Oliver (2000) were applied in authentic learning
benchmarking (see also Herrington, Reeves & Oliver, 2010). They propose that learning is best facilitated in learning
environments that:
1. Provide an authentic context that reects the way the knowledge will be used in real-life
2. Provide authentic activities and tasks
3. Provide access to expert performances and the modelling of processes
174 Leppisaari, Herrington, Vainio, and Im
4. Provide multiple roles and perspectives
5. Support the collaborative construction of knowledge
6. Promote reection
7. Promote articulation
8. Provide coaching and scaffolding
9. Provide for authentic assessment of learning within the tasks
These elements were used to develop an evaluation tool for the project, in the form of a matrix divided into four
columns: the rst column lists the nine elements of authentic learning; the second column expands on each element by
outlining some of its characteristics, the third column suggests a continuum range (values 1-5) for each characteristic
(describing non-authentic through to authentic); and the fourth column is a checklist for evaluators. A continuum al-
lows a picture of how much a learning environment adheres to the nine elements of authentic learning to be gained.
INTERNATIONAL VIRTUAL BENCHMARKING PROJECT (IVBM): DEVELOPING A MULTICULTURAL COMPARISON OF
The virtual peer learning community (cf. Jackson & Temperley, 2007; Lewis & Allan, 2005) established in the
IVBM Project aimed to support teachers in strengthening authenticity in online education and to facilitate their reection
on authentic principles from a multicultural perspective. Good international practices in authentic online courses were
also disseminated.
English language cases were called for from UASs in Finland and these were matched with an international counter-
case. The submitted course was examined in the application form according to Herrington and Olivers (2000) criteria
for authentic learning. There were eight full or partial courses submitted for benchmarking (see Table 1) and these came
from Finland (4), Canada (1), South-Korea (1), Belgium (1) and Wales/UK (1). An Australian expert on authentic learn-
ing acted as project advisor, and there were also observers from Japan.
Virtual work methods and tools enable international peer development of shared authentic e-learning principles. A
Ning environment (http://ibenchmarking.ning.com) was employed as the projects common virtual knowledge collection
and interaction forum. A forum was created for each benchmarking pair in Ning, in which benchmarking sessions and
compactions of the learning process were prepared. Adobe Connect Pro was employed as the online connection (further
ACP). There were two different kinds of participants: 12 teachers participating in the international benchmarking pairs
(4 pairs or groups), and 23 observers. A further 20 people occasionally followed the IVBM groups activity through the
Ning environment.
Benchmarking is a learning process, through which good models are learned from others and development challeng-
es are set for ones own activities (e.g., Jackson, 2000; Karjalainen, Kuortti & Niinikoski, 2002). The IVBM activity was
ve-phased: 1) initiation, 2) preparation for benchmarking, 3) benchmarking session, 4) post mortem discussion and 5)
conclusion. Teachers described authenticity in the course they had submitted to the IVBM process. Benchmarking pairs
comprised teachers who self-evaluated their own (and peer evaluated their colleagues) course applying the authentic
learning evaluation tool. In the virtual benchmarking session, teachers presented their courses mirroring these against the
elements of authentic learning, received peer feedback from their own partner and from other benchmarking practitioners
and observers participating in the session. Each pair continued discussion in Ning as necessary, in which observers could
also participate. Finally, the benchmarking pairs collected learning outcomes of the benchmarking process in Ning.
Table 1: Courses involved in the virtual benchmarking project
UAS /UNI Course Description
1 Savonia
UAS,
and counselling
3ECTS
Students develop client education and counselling skills in individual and group counselling and
acquire skills to operationalise the empowerment paradigm with individual patients. The module
is in Moodle and includes active (functional) studying online (e.g. counselling virtual clients
based on authentic cases), and tasks and exercises implemented in the student`s workplace and
reected on in Moodle.
2 CVO
HIKem-
didactical goals
for own lessons,
LIO Teacher
The course is organised for rst-year secondary school teachers. Didactical models are
interpreted depending on real situations within school contexts and learners expertise. First-
year teachers are also coached on-the-job. The teacher training department follows learners in
three ways: 1. Contact hours: lecturer gives the new theory, explaining didactical processes and
components of a powerful learning environment. 2. Learners integrate the theory to individual
exercises in the learning environment. 3. The lecturer attends lessons delivered by the learner
and comments on application of theory. Platform used is Toledo, a Blackboard adaptation.
3 Kajaani
UAS /
The objective of this course is to study a companys planning targets by means of the business
plan. Students complete a Business Plan of a real or imaginary company on a formulated
sheet. Course content: entrepreneurship, mission, vision, values, business idea, competitor
analysis, calculations, SWOT analysis, strategy and risks. E-studies in Moodle according to
provided instructions. Students write a business plan following instructions, links and hints of the
mentoring business plan in web form. The instructor evaluates and comments on the completed
4 Hanyang
Cyber Uni
World - Conver-
IV Learning
to Talk about
The course focuses on the conversation skills for business with cultural aspect in EFL class by
providing a framework for analyzing culture and social value systems. Beginning with uncovering
the complexities of the term culture, students will then gure out the methodology of comparing
cultural differences and situate their ideas in the local contexts. Cultural studies in relation to
examples in literature and visual media are explored. English language learners will negotiate the
idea of culture in the business environment.
5 Haaga-
Modelling and
Students are able to model an Internet service and understand the possibilities of implementa-
tion, understand present and future trends of ICT and the possibilities of ICT in e-commerce and
communication. Students work with a real organisation, analysing its requirements for an Internet
service, then designing and implementing a prototype along the requirements. Student groups
write a nal report of their project work. Also, they comment on each others essays and evalua-
tions. Customers would evaluate the project work, both the process and the products.
6 Mount
Current and
Pedagogical
The course introduces theoretical and practical components of computers in education with
particular reference to their academic, social and cultural implications. The practical component
exposes students to different computing environments and several software packages. Through
the application of course content, participation in learning activities, and the related assessment
techniques, students should be able to emphasize computer literacy skill, communication skills,
group effectiveness skills. Method of instruction is blended learning.
7 HAMK
Protecting Public
Health in Disas-
ters & Promoting
Public Health in
This module focuses on increasing and deepening knowledge of approaches to public health
in disasters and developing understanding of the relationship between environmental, mental
health and psychosocial wellbeing, epidemiology and the nutritional health of individuals, families
and communities affected by disasters. Learning and teaching methods: 1) Summer school:
Face-to-face lessons, exercises-with group and teachers, conference, eld day-learning from eld
and research experts. 2) Blackboard: Self directed-material/tasks, peer-sharing discussions and
individual assignment, individual support and feedback, 3) Field practice: applied knowledge.
8 Uni of
Ulster &
Uni of
Glamor-
gan &
HAMK
practice in
10 ECTS = 20
UK Credits
The focus of this module is research related to disaster healthcare and the identication of
evidence-based best practice for disaster relief delivery and management. Topics: The research
process, critical appraisal of literature, primary and secondary research approaches, research
questions and hypotheses testing, research aims and objectives, outcome measurements, ethical
issues in relation to disaster relief research, brief overview of statistical analysis and scientic
and proposal writing. Teaching, learning and assessment is designed to allow students exibility
to partly structure their own learning and explore aspects of disaster relief healthcare of particular
interest to individuals or their sponsoring organization.
IMPLEMENTATION OF THE STUDY
Authentic elements of online education were examined and modelled in the IVBM project in a multicultural envi-
ronment. This examination promoted the understanding of authentic online education as a phenomenon and its imple-
mentation in teaching. The research questions of the study were: How did teachers evaluate implementation of authentic-
ity in the examined courses as mirrored against the authentic learning criteria? What cultural differences emerged in the
implementation of authenticity between the various countries?
Research data comprised:
1) the initial survey 2009 (N=17, Webropol) in March-April 2009
2) the nal survey (N=9, Webropol) April 2010
3) Ning documents: interaction and discussion between benchmarking pairs, prior preparation, questions on own
course for pair, questions for pair regarding her/his course, and summary discussion, summary and benchmarking
process reection, learning outcome summaries, self-evaluations using the authentic learning evaluation criteria
form (n=6, two cases lacked self-evaluations), and pair evaluations (n=4)
4) recordings of 10 ACP virtual meetings, which also contain benchmarking session chats (analysed from perspective
of research task.)
5) coordinator observations, notes and discussions
The research methodology was qualitative content analysis. Implementation of authenticity is described and com-
pared applying Herrington and Olivers (2000) elements, which form the research analytical framework and thematic
basis. Discourse was carried out with earlier authentic e-learning studies.
IMPLEMENTATION OF AUTHENTICITY IN THE IVBM PROJECT COURSES AND DEVELOPMENT IDEAS
The data were analysed according to how the learning environments instantiated principles of authentic learning, and
the results are described below.
In designing authentic courses, the context needed to be all-embracing, to provide the purpose and motivation for
learning, and to provide a sustained and complex learning environment that can be explored at length (Herrington, et
al., 2010, p. 19). Often the subject of online material is divided into suitable components for each taskit is believed a
simple form facilitates learning, but simplied data does not meet a complex and multi-voice reality (Engestrm, 1991).
Herrington, et al. (2010) warn against the tendency to oversimplify in learning environments and recommend preserving
the complexity of the real-life setting with rich situational affordances. Realistic levels of complexity in a learning envi-
ronment can even help to make learning easier.
The examined courses were closely linked to specic occupational areas. Teachers saw that the context and content
of the online courses largely represented and mirrored real-life: An authentic context has been created in [this course],
and it includes cases that create a feeling of a genuine learning environment (c4 coding relates to data from Case 4
described in Table 1). One teacher conrmed that her/his course supported authentic learning as: problem solving was
tied to their own business or work setting (c3). In Case 5, the students project for a real customer was the courses major
focus, resulting in authentic implementation.
Development ideas were also identied in creating authentic context: I would like to develop live video-conferencing
discussions with groups of students at a time so to create authentic environments (c4). Learning pathways were generally
seen to be fairly exible, but further ideas were also identied: Pathways that students take could be even more exible
(c1). In many cases, learning was tightly linked to professional development in the learners eld, enhancing meaning
and motivation in study and learning (c1, c2, c6), such as: Course attempts to model a K to 12 teaching/learning environ-
ment for pre-service teacher candidates (c6) (cf. Herrington & Herrington 2006; Herrington, et al., 2010). The environ-
ment represents the kind of setting where the knowledge and skills will be applied: The viewpoint in this course is to
analyse the situation in working life at the moment and to nd means to develop patient education and counselling to be
evidence based (c1).
Teachers considered the creation of a comprehensive learning process as a target for development, in which multiple
contexts form a whole that reects the issues multiple voices and complexity. Course content is often text-oriented, and
multiple methods to produce context were seen as required for the future. Teachers also recognised that interaction be-
tween learners and experts forms contentnot only teacher-produced content. Further development ideas include enrich-
ing content produced through interaction.
The e-learning courses needed to provide ill-dened activities that have real-world relevance, and which present a
single complex task to be completed over a sustained period of time, rather than a series of shorter disconnected exam-
ples (Herrington, et al., 2010). Teachers felt authenticity was realised in tasks, as they were more pragmatic than academ-
ic. Task scope raised much discussion in the evaluation: they could be made more demanding by constructing overarch-
ing problems requiring students to dene sub-tasks. Greater complexity would require a sustained period of time, content
would be employed more extensively, new content created and knowledge deepened. The activities were organised so
that students were working with the same problem throughout the course (c1). Herrington et al., (2010) recommend that
authentic learning tasks provide a sustained period of time for investigation. Activity in two cases was designed around
a complex task (c 3, c5). Herrington, Reeves, Oliver & Woo (2004) observe activity does not necessarily supplement the
course, it can be the course.
In several cases tasks had a clear real world transfer: The new knowledge is used in real-life at three levels: rst of
all the learners appropriate the theory via concrete skills in online exercises, then they receive comment from their pairs
and nally they are coached while integrating the skills in their teaching environment (c2). Identied challenges included
a better use of previous learners work, building on what goes before to serve as more authentic examples (c8). Her-
rington et al., (2006) also identied the opportunity for the detection of relevant versus irrelevant information as a factor
that increases authenticity. Only a rich and diverse pool of source material promotes a critical assessment of knowledge
relevance. In many benchmarking cases students are able to choose information from a variety of inputs (e.g., web links,
materials designed by teachers, fellow students experiences). Quantity and adequacy of material caused discussion in
the IVBM process: When doing this self-evaluation we discussed and came to the conclusion that it (using irrelevant
sources) would be a good idea. A peer evaluator raised the possibility of utilising cases or podcasts made by learners in
the detection of relevant and irrelevant knowledge. Future activities would require students to analyse relevant versus ir-
relevant information in order to be able to support their method of choosing relevant knowledge related to their subject.
In order to provide expert performances, the e-learning course needed to provide access to expert thinking and the
modelling of processes, access to learners at various levels of expertise, and access to the social periphery or the observa-
tion of real-life episodes as they occur (Herrington et al., 2010). Too often the course remains teacher-centric; the teacher
denes content and tasks, preventing collaborative doing which would model expert performance. It is important for stu-
dents to be able to compare their performance with others at various levels of expertise. Herrington et al (2010, 25) also
remind educators that the lecturer is also an expert who can share and model expert performance. The evaluated courses
offered a number of opportunities to move among different levels of expertise. Methods were not teacher-centric, but
border-crossing and collaborative, facilitating the sharing of learning experiences and construction of a learning commu-
nity: Access to expert performances and the modelling of processes is facilitated e.g. in discussions and interviews with
the customer business experts. (c5). An industrys operational guidelines also represent expert knowledge and modeling.
Also teachers participate, for example, in discussions providing one point of view of expertise (c1). Learners are at vari-
ous levels of expertise and can enrich the learning of others: Each learner can reect on the proposed items, taking into
account the approach of the (place) where they work (c2). Students are from different backgrounds and some of them are
experts in (this eld) having years of experience. The students really share different kinds of stories about professional
practice (c1).
While expert-like work was evident, there is room for improvement. There was discussion on how the expertise of
previous students could be harnessed in virtual learning communities: Each year I have new learners and they dont have
access to previous communities. possibility is to develop a database of good practice examples... This database gath-
ers all tips and tricks from previous learners so new learners can search for help or advice within the platform. Links to
expert performance on some courses was constructed through social media (e.g. blogs, Facebook, webinars, Twitter):
Sharing content in social media provides good discussions with experts (c5). The facility of the Web to create learning
communities who can interact readily via participatory technologies also enables opportunities for the sharing of narra-
tives and stories, professional examples (e.g., c6). Linking experts to teaching through social media is an issue for further
development, also justiable from a multicultural information literacy perspective: I will try to build in more web 2.0
content where students can contribute to a collective intelligence via wikis or other websharing tools. I think that it will
encourage students to think more globally and to recognize possible global knowledge as well as perhaps, areas where
culture codes and such do not allow for complete universal truths... Certain wikis are geared towards more western audi-
ences and so, certain biases prevail. Exposure to information is key to understanding and developing what is known as
the collective conscience. (c4).
Essential to learning in an information society is the crossing of traditional borders and multiple, discipline integrat-
ing perspectives, as life is not one subject (Leppisaari, Silander & Vainio, 2006). For students to be able to investigate a
problem or task from more than a single perspective, it is important to enable and encourage students to explore different
perspectives on the topics from various points of view, and to criss cross the learning environment repeatedly. As Her-
rington et al. (2010) observe many e-learning courses and resources are designed in a linear instructional format, assum-
ing that learners begin at the beginning and work through to the conclusion. Such courses provide inadequate experiences
for students to deal with complex issues. How can students be given opportunities in online studies and virtual groups to
work with students representing other professions/disciplines as they may need to, at least partially, in the workplace?
Teachers felt the courses provided very different perspectives on the topics from various points of view. Opportuni-
ties to criss-cross the learning environment by multiple pathways was rated very highly. Multiple perspectives were pro-
moted by versatile material (scientic knowledge, more practice-based knowledge) and range of experiences represented
by students (most of them being professionals and having a lot of practice experience) (c1). In one case, role-play in the
dialogue section provided multiple perspectives (c4) and in another, student roles were developed through teamwork /
team-players (c8). In project-based implementation, this element is prominent: In project work there are leader, tech-
nology expert etc. and in e-exam technical consult to some customer (c5). In Case 6, inquiry-based project activities
provided students with opportunities to examine the problem from a variety of theoretical and practical perspectives.
Students could also use social media for multiple purposes from different points of view. Exploring issues from multiple
perspectives also caused critical reection. In particular, cultural differences emerged within ill-structured learning envi-
ronments, a feature of this fourth authentic learning principle. Too many perspectives or unlimited material do not nec-
essarily support learning (cf. Kinshuk, 2010). Limited perspectives were defended as follows: In this context, we focus
on one of the competences of (profession). The peer assisted methodgives a variety of examples, applications of the
theory, which is enriching for each learner, focussing on the domain (acquired) (c2).
The opportunity to collaboratively construct knowledge is seen as important element of an authentic learning mod-
el (Herrington et al., 2010). Especially in e-learning, tasks need to be addressed to a group rather than an individual,
and appropriate means of communication (discussion forums, social networking, wikis, etc.) need to established. In the
IVBM project, teachers felt their courses offered fairly good opportunities for pair or group work. However, their self-
evaluations indicated course structure supported a groups purposeful construction of knowledge weakly. e-Learning
communication was seen to be crucial, but was not sufciently effective: How to encourage group interaction around the
aims of the module? At the moment the communication is greatly facilitator-led (c8). The evaluations showed that not
all cases employed group tasks, while in some they were used substantially: together they (learners) construct, improve,
give feedback (c1). Collaboration was realised mainly through tasks (pairs, small group) and discussion. In one case,
the depth of collaboration was perceived to vary, depending on assignment and learner motivation: Real collaboration
depends on students willingness to work with each other (c3). Purposeful tasks and online discourse roles of peer learn-
ers also promoted collaborative construction of knowledge: All learners will read the outcomes of the others and some
of them are directly involved in peer coaching (c2). Colleague-learners may bring problems and issues to the collective
discourse environment, receive advice and comments from their peers and work through issues together. Cultural differ-
ences were also evident in use of group work. For example, in Asian cultures learners usually prefer to work indepen-
dently and the teacher felt it slightly challenging to integrate student collaboration into the course.
Reection on individual or group grades for products was closely tied to group work and the collaborative construc-
tion of knowledge: The grades will be given more in individual/pair effort this is something that we should maybe
think about once again (c1). Often participation in discussion forums is not rewarded or graded, a practice teachers want
changed. In the two project-based cases evaluated here, learners came from different levels, degree programs, countries
and cultures, bringing added value to collaborative construction of knowledge. Large group projects denitely provided
opportunities for collaboration. Either students received a group rather than individual mark for all group tasks, or the
course contained an appropriate mix of group and individual assessment (portfolio) and feedback.
In order to provide opportunities for students to reect on their learning, the e-learning course needed to provide an
authentic context and task to enable meaningful reection. It also needed to provide non-linear organisation to enable
students to readily return to any element in the site if desired, and the opportunity for learners to compare themselves
with experts and other learners in varying stages of accomplishment. (Herrington et al., 2010).
Teachers in their self-evaluation of course content gave a low rating for authentic material and tasks that required
learners to make decisions on reaching learning objectives. However, learners were able to return to any element to re-
ect on material and resources and they had fairly good opportunities to compare themselves with other learners in vary-
ing stages of accomplishment. Learning in the examined courses did not, however, rigorously support reciprocal reec-
tion among each pair or the groups collective reection. One teachers insight during the benchmarking process was:
The course promotes reection but students in most cases dont use this feature. I should get students to compare their
thoughts and ideas to experts, teachers and other students (c3). Reection is seen as a central authentic learning element
in working life-centric education: The discussion forums and a written assignment are planned so that they promote
reection we think that it is not possible to developeducation to be evidence based without reection. Thus, it is
necessary to nd means how to support students reection: they have to reect on their own actions and values, as well
as actions and values in their work settings (c1). Experiential learning and theoretical knowledge are integrated through
reection (Kahne & Westheimer, 2000).
Often purposeful reection was realised individually: self-reection on the course takes place as written assignments
but again, there is very little interactive reection amongst students (c4). Exceptions were the project-learning cases, in
which group decisions were denitely required to complete tasks, and in which feedback and discussion between groups
was realised. Reection was supported by learners at varying stages of expertise comparing thoughts and ideas in learn-
ing communities with their peers, teachers and working life experts. One benchmarking pairs insight was that dividing
a big student group (about 20 students) into smaller groups promoted more in-depth reective discourse. Reection can
be supported through diverse educational technologies and social media tools (Jonassen, Marra & Crismond, 2008), for
example, in discussion and chat forums, blogs, and wiki spaces that promote reection. Reection was deepened in the
cases by compiling a portfolio of reection tasks, which explicated the learning journey (c2, c6). Reection was seen to
be supported if the course offered a self-assessment component for all assignments (the assignment rubrics) or student
blogs (students comment on what they learned and areas for future improvement). Cultural differences in the use of self-
reection were evident, which could partly be explained by a learners age: I nd that asking for students self-reection
about a topic is challenging especially because in a traditional sense, they are not accustomed to reecting on the process
of their learning (c6).
In order to produce an e-learning course capable of providing opportunities for articulation, tasks need to incorporate
inherent, as opposed to constructed, opportunities to articulate, collaborative groups to enable articulation, and the public
presentation of argument to enable defence of a position (Herrington et al., 2010). Teachers self-evaluations indicate that
course tasks required little discussion and articulation of beliefs and growing understanding. Teachers gave a slightly
better rating to case tasks providing collaborative groups and forums to enable articulation of ideas. Public presenta-
tion of arguments was seen to be realised well. In several cases (e.g., c1, c2) articulation was concretised in discussion
forums: The discussion forums support students to discuss also beliefs and growing understanding. the students have
also formed smaller collaborative groups according to their interest and working eld. The role of articulation has also
been recognised in the value of peer tutoring. Understanding through cognitive conict occurs when students are re-
quired to develop arguments and achieve consensus (Herrington et al., 2010). Discussion forums also enabled defence of
arguments. However it was noted that this is also time consumingarticulation and group coaching can be much bet-
ter integrated in the online learning with group sessions where they can reect, ask for advice, discuss with peers and
nd and defend their own arguments. (c2)
According to the evaluations, portfolios and use of wikis were also factors that promote articulation (e.g., c6). One
teacher observed during the examination of the benchmarking pairs course: Perhaps I could also encourage my student
groups to use cooperative wikis instead of Word in their reports and essays. Articulation, according to teachers, was
strengthened by using videoconferencing as an interactive e-learning tool, so students could control lecture speeds and
have unlimited access to contents to review and practice articulation (c4). In project work or drawing up a business plan,
the entire learning task constituted articulation and defence of arguments. The group project denitely provided students
with opportunities to discuss and demonstrate their growing understanding (c5). Articulation was also promoted by all
major assignmentsposted to the Web for external presentation and feedback (c5). Articulation could also be developed
by opening up learning contexts and products to a wider public in ones institution or on the internet globally: Potential
to video tape the group teaching workshops and post them to YouTube (cf. Makino 2007).
In order to accommodate a coaching and scaffolding role principally by the teacher (but also by other students),
the e-learning course needed to provide the opportunity for more able partners to assist with scaffolding and coaching,
as well as the means for the teacher to support learning via appropriate communication technologies (Herrington et al.,
2010). The teacher as coach is a fundamental and integral part of an e-learning course that provides a substantial scaf-
folding and coaching support for students. However authentic learning principles also underline collaborative learning,
where teachers and more able partners can assist with scaffolding and coaching.
Scaffolding support in multiple forms was seen to be easily available, although teachers had identied situations in
their work when learners sometimes experienced a lack of coaching. Peer guidance was in some cases strongly linked to
the learning process: all assignments have a formal peer review component. Groups are intentionally created to provide
peer tutoring opportunities (c6). Teachers felt the structure of their courses provided strongly for collaborative learning,
in which more expert (learners) offered guidance and support (c1, c2): More knowledgeable students are able to assist
othersthe students are sharing ideas, working tools/methods - it seems that they are also teaching each others when
need arises (c1). However, one teacher felt reciprocal peer guidance should be more purposefully planned and integrated
into learning tasks. Peer guidance should not increase an adult learners workload. Coaching and scaffolding conducted
by students could be planned in advance (e.g., group and pair assessment in discussions) (c1).
Teacher guidance was, according to self and peer evaluations, available as needed and its utilisation was dependent
on the learners themselves. (c3). Technology tools used for guidance included: message forums (c5), email or a Q&A
forum, and also a site where the student and professor discussed on a one-to-one basis(c6). In one country, quality of
teacher guidance was assured from an employers perspective: all professors are evaluated by students and percentages
are given based on an evaluative test. External, working life expert guidance was also utilised: These online exercises
are combined with coaching on the work oor (c2). Stronger integration of working life experts to guidance is wanted
in future (cf. Leppisaari & Helenius, 2005): There is a good possibility for experts in the companies to coach and advise
students (c3, c4), although its practical arrangement is considered somewhat challenging.
In order to provide integrated and authentic assessment of student learning, the e-learning course needed to provide
the opportunity for students to be effective performers with acquired knowledge, and to craft polished performances or
products in collaboration with others. It also requires the assessment to be seamlessly integrated with the activity, and to
provide appropriate criteria for scoring varied products (Herrington et al., 2010).
According to teacher self-evaluations, opportunities for learners to create polished performances or products was re-
alised well in their courses. Continuous assessment structuring in modules and adequacy of assessment measures caused
considerable discussion. In the benchmarking process, it was observed that there is a lot of weight at the end of the
course: the written assignment will be graded. At the moment the learning process is not assessed (c1). One teacher ex-
pressed the view that: In future, we will assess the whole learning process the evaluation will be continuous during the
course including self-, peer-, and group evaluation. It is important to assess participation and contribution to discussions
because they are so important a part of the course (c1). In another case, all major course assignments had self, peer, and
instructor assessment components (c5). Multiple measures of assessment were deployed (e.g., group work, individual es-
says, and nal exam). Students also created individual essays based on their group work experiences. Assessment could
be developed to include external assessment opportunities for the student projects and use of wiki summaries and student
portfolios. Additionally, participation in each others learning process can be more intense.
In some cases, assessment was teacher-centric and based on the quality implementation of a required working life-
oriented product according to prior criteria. After the benchmarking process, one teacher felt that the entire learning pro-
cess should be developed towards assessment: The instructor could assess the plan step by step and give feedback after
which a student can go on tooling the plan. (c3). The teacher saw a need to diversify assessment by including working
life experts in the process which, however, s/he felt presented practical obstacles, both from the course implementation
and external experts time and commitment perspectives. Learners were aware of evaluation criteria, but teachers felt the
criteria needed clarication. In one case, students co-created assessment rubrics for all assignments (c5). In Canada and
Korea, assessment rubrics were used in each assignment, outlining all assessment measures and points of different levels.
In Finland, an exact scoring criteria (rubric) on a scale of 1 to 5 was not used in all contexts; teachers can subjectively
assess a product according to certain requested expectations. However, Finnish teachers challenged their international
colleagues to consider innovative assessment methods: What about some innovative approaches which dont t into any
of the assessment rubric levels? (c6).
The IVBM project offered teachers new ideas for a pragmatic development of assessment in teaching: Evaluation
takes place through attendance, homework, discussion board participation and the two exams An oral test component
as in a live online discussion would give me a better sense of a students level (c4). Although authentic assessment was
considered a difcult learning element to implement, in vocationally oriented higher education authentic assessment is
an evident strength: Since the tasks are always based on authentic situation from the working environment of the learner,
also the assessment is based on the reference world of the learners, linked to their direct reality (c2).
Overall, authentic learning principles were implemented quite consistently and adequately in the e-learning cases
evaluated in the IVBM project. The average of all elements in six self-evaluations was 3.9/5. From an authentic learn-
ing perspective, collaborative construction of knowledge was the most weakly implemented element. Collaboration as
a group was according to teachers self-reection and peer evaluation the most challenging component of this element.
Collaboration was also not supported very effectively in relation to group assessment. Other areas implemented below
average were authentic contexts exible learning pathways that reect real-world settings, and opportunities to identify
irrelevant and relevant knowledge. Implementing reection was also challenging. Multiple roles and perspectives and au-
thentic coaching were, according to teachers evaluations, the most successfully implemented elements.
Teachers felt they had succeeded quite well in planning their courses to meet authentic learning criteria. Learning ac-
tivities reected real-world relevance. In addition, the reective nature of the self-evaluation process enabled teachers to
gain many more ideas for developing authenticity in their course. Teachers self-evaluation was conrmed and supported
by the feedback from their benchmarking pair. These ideas were related to the organisation and structuring of the course,
learning activities, and especially to evaluation. It was also felt that pathways that students take could be even more ex-
ible, and that more collaboration between students and between university and working life/real life is necessary. Assess-
ment also needs attention: it should be continuous throughout the course and include self-, peer-, and group evaluation.
The IVBM project offered teachers opportunities to become aware of cultural differences in teaching and learning
online. Cultural background greatly affects views of online learning. The culture code of online learning such as exists in
Finland, Korea, Canada, Belgium and Wales/UK is worth considering when constructing global content. An examination
of cultural differences in online education, in implementation and expressions of authenticity in this limited data, pre-
vents generalisation but the observations made provide certain perspectives for later broad comparative studies.
Access to technical support was considered a cultural difference factor in the IVBM project (cf. Lee, Leppisaari &
Im, 2009). Korea emerged as a good example of faculty support, an experienced supporting team to help professors de-
velop virtual courses. A Finnish teacher described how he/she had to learn the learning environment and do all technical
things alone: If I had resources and technical support, I would design the course again, especially...create the plan tem-
plate much more impressive, layout, colours, scaling planning target boxes etc. (c3). Course layout and an extensive use
of visual material, multimedia and video streaming in the Korean implementation interested the Finnish teachers, who
saw the potential for Finnish online education to be more diverse and visually rich. Korean online learning culture code
expresses that: the average Korean student is quite tech-savvy and prefers much graphic detail when learning. Korea has
the advanced technology to readily provide this type of learning. However, consistent with, for example, a Finnish cul-
ture code, the learners age affects online study: Older students are less inclined to engage content whether it is because
of their basic knowledge of computers or because they are used to teacher-directed/textbook learning (c4). Consistent
with previous studies, the project examined here indicated that a western method of processing knowledge is traditionally
more text-based, while an eastern approach relies more on knowledge visualisation (cf. Munro, 2009). It should, how-
ever, be noted that Koreans enjoy strong technical infra-structure for fast communication connections, facilitating the use
of video streaming and multimedia.
Authentic learning elements in which eastern and western approaches diverged included, for example, structure of
the learning environment, self-evaluation and group work. Authentic assessment especially caused discussion on cultural
factors. Finnish teachers may have experienced Canadian course grading policy as complicated. One teacher analysed
this as a cultural concept too: Some prefer to think about fullling objectives while others prefer to see quantitative scor-
ing. Combining the two could benet the studentCanadian students are familiar with rubrics because our curriculum
requires teachers to compose them (c4).
This multicultural examination of the implementation of authentic elements in eight e-learning cases across ve
countries indicates that online education could have a more signicant role in the development of multicultural global
education. For this reason, it is important teachers gain experience of multicultural peer development of authentic educa-
tion. The IVBM model provides one such method of implementation.
Engestrm, Y. (1991). Developmental work research: Reconstructing expertise through expansive learning. In M. I. Nurminen &
G.R.S. Weir (Eds.) Human jobs and computer interfaces. Amsterdam: Elsevier.
Hakkarainen, K., Palonen, T., Paavola, S. & Lehtinen, E. (2004). Communities of networked expertise. Professional and educa-
tional perspectives. Earli. Amsterdam: Elsevier.
Herrington, J. (2006). Authentic e-learning in higher education: Design principles for authentic learning environments and tasks.
In T.C. Reeves & S. Yamashita (Eds.), Proceedings of E-Learn 2006 (pp. 31643173). Chesapeake, VA: AACE.
Herrington, A., & Herrington, J. (2006). What is an authentic learning environment? In A. Herrington & J. Herrington (Eds.),
Authentic learning environments in higher education (pp. 113). Hershey: Information Science Publishing.
Herrington, J., & Oliver, R. (2000). An instructional design framework for authentic learning environments. Educational Tech-
nology Research and Development 48, 2348.
Herrington, J., & Oliver, R. (2002). Designing for reection in online courses. In A. Goody, J. Herrington & M. Northcote
(Eds.), Quality conversations: Research and Development in Higher Education (pp. 313319). Jamison, ACT: HERDSA
Herrington, J., Reeves, T.C., & Oliver, R. (2006). Authentic tasks online: A synergy among learner, task, and technology. Dis-
tance Education 27 (2), 233 247.
Herrington, J., Reeves, T.C., Oliver, R., & Woo, Y. (2004). Designing authentic activities in web-based courses. Journal of Com-
puting in Higher Education, 16(1), 3-29.
Jackson, N. (2000). Benchmarking educational processes and outcomes. In N. Jackson & H. Lund (Eds.), Benchmarking for
Higher Education. Society for Research into Higher Education & Open University Press.
Jackson, D., & Temperley, J. (2007). From professional learning community to networked learning community. In L. Stoll & K.
S. Louis (Eds.), Professional learning communities (pp. 4562). Berkshire: Open University Press.
Jonassen, D., Howland, J., Marra, R. M., & Crismond, D. (2008). Meaningful learning with technology. Pearson.
Kahne, J. & Westheimer, J. (2000). A pedagogy of collective action and collective reection. Journal of Teacher Education
51(5), 372383.
Karjalainen, A., Kuortti, K., & Niinikoski, S. (2002). Creative benchmarking. Designing sustainable international cooperation
in higher education. University of Oulu, Finnish Higher Education Evaluation Council.
Kinshuk (2010). e-Learning principles for exploratory learning. Lecture in Chungbuk National University, 14.9.2010.
Lee, O., Leppisaari, I., & Im, Y. (2009). Guidelines for national e-learning evaluation International comparative study between
Korea and Finland. Asian Pacic Collaboration Education Journal. 5(1), 2948.
Leppisaari, I. & Helenius, R. (2005). Online mentoring toward a new teachership. In Proceedings of RWL4, 4th International
Conference on Researching Work and Learning, December 11 14, University of Technology, Sydney.
Leppisaari, I., Ihanainen, P., Nevgi, A., Taskila, V-M., Tuominen, T. & Saari, S. (2008). Hyvss kasvussa Yhdess kehitten
kohti ammattikorkeakoulujen laadukasta verkko-opetusta (Growing well Developing together towards quality university
of applied sciences online education). FINHEEC 4: 2008. Helsinki.
Leppisaari, I, Silander, P., & Vainio, L. (2006). Autenttisuus ammattikorkeakoulun virtuaaliopetuksen haastena (Authentic learn-
ing: The challenge for universities of applied sciences e-learning). In M. Ylikarjula (Ed.) Wonder and learning admidst re-
search. Teachers as researchers of their own work- Symposium III articles (pp. 1736). COU, Kokkola.
Lewis, D., & Allan, B. (2005). Virtual learning communities. Berkshire: Open University Press.
Munro, J. (2009). Learning internationally on a future context. In M. Hayden, J. Levy & J. Thompson (Eds.) The SAGE Hand-
book of Research in International Education (pp. 113127). London: SAGE, .
Makino, Y. (2007). The third generation of e-learning: expansive learning mediated by a weblog. In Leppisaari et al. (Eds.)
Kolme siett kasvuun. (Three threads to growth: online pedagogy, educational technology and working-life connections)
(pp. 96-113). COUAS, Kokkola.
Oliver, R., Herrington, J., & Reeves. T.C. (2006). Creating authentic learning environments through blended learning approach-
es. In C. Bonk & C. Graham (Eds.), The handbook of blended learning (pp. 502515). San Francisco: Pfeiffer.
Resnick, L. (1987). Learning in school and out. Educational Researcher, 16, 1320.
Saari, S., & Leppisaari, I. (2008). Web-based learning services experienced by the students in the Finnish UASs. Paper presented
in EAIR, The European Higher Education Conference, the 30th Annual Forum Polishing the silver: Are we really improv-
ing higher education? 2427.8.2008, Copenhagen, Denmark.
Woo, Y., Herrington, J., Agostinho, S., & Reeves, T. C. (2007). Implementing authentic tasks in web-based learning environ-
ment. EDUCAUSE Quarterly 3, 3643.
Student Perceptions of Immersive Virtual Environments for the Meaningful Assessment of Learning 185
Student Perceptions of Immersive Virtual Environments for the Meaningful Assessment
of Learning
JILLIANNE CODE
, Jody Clarke-Midura, NiCk ZaP, aNd ChriS dede
Integrating technology in the teaching and learning process goes beyond mere usability. Traditional usability relies
heavily on an end-of-use judgment made by the user on the utility of the technology to satisfy the purpose for which it is
used. Numerous reports and recommendations exist for evaluating the usability of technical content (i.e. Chalmers, 2003;
Nielsen, 1993; Preece, Rogers, & Sharp, 2002) Less frequently studied is the evaluation of technology and its utility for
teaching and learning (Nokelainen, 2006). For a technology to be useful in education it must be both usable and hold
some kind of utility for teaching and learning. Ultimately, the effectiveness of an educational technology depends upon
teachers and learners choice, perception, and functional utility. We propose that both the teacher and the learner should
judge the effectiveness of a technology to play a facilitative role in education along at least three dimensions: teaching,
learning, and assessment. An important part of the iterative design of any teaching and learning environment involves us-
ers process-use judgments between the beginning-of-use and end-of use cycle that reect the ability of the virtual learn-
ing environment to perform its intended functions. The focus of this research is specically on a learners judgment of
the assessment utility (JOAu) of virtual performance assessments.
A constructivist, learner-centered approach to technology design requires a constructivist, learner-centered approach
to the evaluation of that design (Mayer, 2009). Advances in information technology enable innovative ways for using
performance-based assessments to measure learning (Pellegrino, Chudowski, & Glaser, 2001). One such technology is
Immersive Virtual Environments (IVEs). IVEs are three dimensional (3D) simulated contexts that provide rich, authentic
contexts in which participants interact with digital objects and tools. With funding from the Institute of Education Sci-
ences (IES), the Virtual Performance Assessment project at Harvard University is developing and studying the feasibility
of using immersive virtual performance assessments to assess scientic inquiry of middle school students as a standard-
ized component of an accountability program. The goal of our Virtual Performance Assessment (VPA) research project
is to develop and study the feasibility of using IVEs as a platform for assessing middle school students science inquiry
skills in ways not possible with item-based tests (http://vpa.gse.harvard.edu). IVEs allow for performances and observa-
tions of these skills that are not possible via traditional testing formats (Clarke, 2009). The purpose of this study is to ex-
amine student perceptions of the assessment utility of IVEs for the evaluation of their science inquiry knowledge, skills,
and abilities (KSAs). We present criteria for evaluating the assessment utility of virtual performance assessments and
how this feedback is used to make experiential changes to the design and implementation of the IVE while keeping the
same student assessment model.
Traditional assessments often focus on individual test items and rely on student afrmation as a response that indi-
cates knowledge. In our VPAs, we base the evaluation of student performance on measurements captured as in-world in-
teractions. These interactions allow us to assess what students know and do not know about science inquiry and problem
solving. As a part of the inquiry progression embedded within the VPAs, students are required to make a series of choices
as a part of an ongoing narrative. Similar to the Choose Your Own Adventure books (Chooseco, 1977-2011) where read-
ers construct their own narrative through a series of choices, students in our VPAs experience the interactive outcomes of
their choices. The focus of the VPAs is not the attainment of a single right answer, but rather on the result of a series of
choices that students make. The series of interactions result in rich observations that enable us to make a ne distinction
of students understanding of the various facets of inquiry (Figure 1).
Authentic assessment in science requires students to apply scientic knowledge and reasoning to situations similar
15 The work presented in this paper was performed while Jillianne Code was a Postdoctoral Research Fellow at Har-
vard University.
186 Code, Clarke-Midura, Zap, and Dede
to those they will encounter in the world outside the classroom, as well as to situations that approximate how scientists
do their work (National Research Council (NRC), 1996). By nature, then, authentic assessment in science examines stu-
dents understanding and appreciation of a scientic community of practice. The epistemic frame hypothesis (Shaffer,
2006) states that every community of practice has a culture whose basic value structure is composed of skills, knowl-
edge, identity, values, and epistemology. The epistemic frame of scientic inquiry engaged by the Virtual Performance
Assessment project follows a generally accepted denition of science inquiry formally outlined by White, Frederiksen,
and colleagues (White, Collins, & Frederiksen, in press; White & Frederiksen, 1998). Broadly speaking, science inquiry
involves: 1) theorizing, 2) identifying questions and hypothesizing, 3) accessing data and investigating, and 4) analyzing
and synthesizing (Figure 1). The assessments in the VPA project are more specically based around the National Science
Education Standards (NSES; NRC, 1996) and College Board Standards for College Success (CBSCS; College Board,
Figure 1. A semiotic frame of science inquiry learning.
Inquiry learning, like problem solving, involves more than one best way of arriving at a solution especially in sci-
ence. Forcing students to abide by a single model of thinking inhibits their natural ability to establish a plausible explana-
tion for existing phenomena. Thus, as a part of our design we intend to explore how handing over control of the assess-
ment experience to the student affects assessment achievement outcomes. (see Clarke-Midura, Dede, & Mayrath, 2010a;
Clarke-Midura, Dede, & Mayrath, 2010b for an overview of the VPA design process). A one-size-ts-all approach has
been criticized for the fact that the students are required to adjust their learning to t the teachers conception of the best
way of learning certain material (Jonassen, Myers, & McKillop, 1996). A similar argument can be made for the way
learners are assessed. Since teaching inquiry learning often takes a less structured approach, the assessments of inquiry
learning should do the same. In our assessments, students are given unstructured source material for a problem and they
are expected to hypothesize and test their theories for what is causing the problem.
Assessment 1: Save the Kelp!
To frame the discussion and illustrate the implementation of aspects of the design for Assessment 1, the following is
an excerpt of a vignette of Assessment 1: Save the Kelp! highlighting a few aspects of the learner experience.
Thomas sits at his computer and logs into the student portal. He watches an introductory video telling him that they kelp
in Glacier Bay are dying and he needs to help fgure out why. He is given a brief description Bull Kelp and why they are
important to the ecosystem. Thomas opens the assessment and is immediately told how to navigate the world and that
he will be given quests to complete and he will know that there is a quest by a yellow ! over a persons head. He is
transported to Glacier Bay where Ranger Tina gives him his frst quest.
Figure 2. A character, Ranger Tina, in Assessment 1 that has a quest to give.
Ranger Tina asks Thomas to travel under water by clicking on the scuba tank on the beach to see where the kelp live and
learn more about kelp from a diver named Marc. Thomas sees the scuba tank on the beach, walks over to it, and clicks
on the tank. He is then teleported to the kelp bed in the harbor.
Figure 3. Scuba Marc teaches about kelp and each of the data collection tools in the toolbar.
While under the water Thomas talks with Marc, where he learns more about kelp, and how he can use each of his tools
in his toolbar. Thomas discovers that he has four tools he can use to take measurements around Glacier Bay. He tries
to take a population sample of the Bull Kelp in the harbor by clicking on the tool and then clicking on one of the kelp in
the bay. He fnds out that there are 20 Kelp in Glacier Bay. Thomas decides that this information is important and that
he may need it later so he saves it to his notebook. Marc tells Thomas to return to the shore and talk to Ranger Tina to
receive his next quest. Thomas clicks on the tank on his back and is teleported back to the shore.
Figure 4. Two scientists on the beach have competing hypotheses and quests to give in Assessment 1.
Ranger Tina tells Thomas there are two scientists on the shore have competing hypotheses about why the Kelp in Glacier
Bay is dying. Thomas goes over to the scientists to talk to them about what they think is going on. Thomas talks to both of
the scientists and they ask him to help them collect additional data to try and fgure out if the Power Plant in Glacier Bay
is causing the kelp to die. The scientists also tell Thomas to travel to a healthy bay, Green Cove, to collect comparison
The assessment utility of any IVE is guided by design assumptions of how the interactions in the IVE facilitate the
demonstration of students knowledge and skills (see Clarke-Midura, et al., 2010a; Clarke-Midura, et al., 2010b for an
overview of the VPA design process). To examine our constructivist, situated design assumptions about how the VPA
facilitates the immersive assessment of science inquiry, we conducted an empirical investigation of student perceptions of
the assessment utility of the VPA for this purpose. Evaluating the assessment utility in this context focuses the research
on the types of skills the VPA enables students to demonstrate; providing additional evidence that the VPA is a valid as-
sessment of science inquiry. Building on the work of Nokelainen (2006) on pedagogical usability we have developed
the Meaningful Assessment of Learning Questionnaire for Virtual Environments (MALQ-VE). Items on this scale are
loosely based on items from the Pedagogically Meaningful Learning Questionnaire (PMLQ; Nokelainen, 2006) and help
to establish how well each of our VPAs enable learner control, engagement in activity, added value for learning, exibil-
ity, feedback, and valuation of previous knowledge.
METHODS & DATA SOURCES
Using a sample of middle school science students (N = 260, 125 Female) we adapted 20 items from the Pedagogi-
cally Meaningful Learning Questionnaire (PMLQ; Nokelainen, 2006). This included student perceptions of the follow-
ing components of VPAs: learner control, learner activity, added value, exibility, feedback, and valuation of previous
knowledge. Each of these items is in Appendix A. Following a 90 minute exposure to Assessment 1, students were asked
to state their agreement with a series of items using a 5 point Likert scale from 1 = strongly disagree to 5 = strongly
agree. A second study using Assessment 2 and validating the revised MALQ-VE is planned for Spring 2011.
A classical item analysis, including an exploratory factor analysis, was conducted to assess the unidimensionality of
this scale. A classical approach was chosen for this analysis because of the small relative sample size (N = 260). The re-
sults of a classical analysis for the MALQ-VE are presented in Table 1. The distribution of item correlations (CITC) was
from -.13 to .58. Since the CITC for items 1, 8, 10, 12, and 16 are low (< .25) they are poorly discriminating. These items
were removed from subsequent analyses.
Table 1. Classical Item Analysis of the MALQ-VE (N = 260, = .80, CI
= .76, .83)
Response Category
Item M
SD CITC
1 3.06 1.09 -0.13 0.82 16 70 84 62 28
2 3.48 0.98 0.48 0.78 10 31 74 115 30
3 3.54 0.92 0.48 0.78 9 21 80 120 30
5 3.26 1.07 0.53 0.78 10 63 67 89 31
10 3.05 1.08 0.15 0.80 21 63 76 82 18
11 3.75 1.11 0.33 0.79 13 28 37 114 68
12 3.62 0.92 0.20 0.79 4 29 68 121 38
Note: CITC = Corrected Item Total Correlation; Bolded items have a CITC < .25 and are poorly discriminating;
Item mean is a classical test theory (CTT) indicator of difculty.
Indicates item discrimination.
if item is deleted;
1 = strongly agree; 2 = disagree; 3 = neutral; 4 = agree; 5 = strongly agree.
EXPLORATORY FACTOR ANALYSIS
An exploratory factor analysis (EFA) was used to assess latent dimensionality of the MALQ-VE since the original
validation of the PMLQ was conducted with elementary students and was designed for evaluating Learning Manage-
ment Systems. The factors were extracted using Varimax rotation with Kaiser Normalization. The EFA on this data set
revealed a three-factor structure as reported in Table 2: learner exibility and feedback ( = 0.77, CI
= .72, .81), learner
control ( = 0.69, CI
= .62, .74), and learner activity ( = 0.59, CI
= .50, .67). The calculated internal consistency of
the entire scale is = 0.83, CI
= .80, .86, above the acceptable level of
> .70 (Tabachnick & Fidell, 2006) however,
learner control and learner activity scales could be improved. Items on each of these scales will be revised for the larger
study.
Table 2. Factor Pattern and Structure Matrices for the MALQ-VE ( = 0.83, CI
Item Factor 1 Factor 2 Factor 3 h
Learner Flexibility and Feedback ( = 0.77, CI
13 0.60 0.30 0.16 0.38
3 0.56 0.16 0.21 0.48
15 0.52 0.29 0.38
Learner Control ( = 0.69, CI
Learner Activity ( = 0.59, CI
Based on each newly dened factor, a summary analysis reveals (Table 3) that students strongly agreed or agreed
that the IVE enabled increased learner exibility and feedback (52.2%), control (46.5%) and activity (60.6%). However,
these results reveal that there is still room for improvement of the overall student experience in each of these areas.
Table 3. Summary of Student Responses by Factor
Learner Flexibility & Feedback 5.5% 13.7% 28.6% 38.9% 13.3%
Learner Control 8.3% 20.0% 25.2% 36.4% 10.1%
Learner Activity 2.4% 11.2% 25.8% 47.7% 12.9%
In addition to the items in the MALQ-IVE, students were given a series of open-ended questions to give them an op-
portunity to provide additional feedback on specic elements of the VPA. The following is a list of the top three elements
that the students liked and disliked about Assessment 1.
Students liked:
Taking samples
Making their own decisions about the problem
Moving around, choosing where to go and what
to do, game-ness
Students didnt like:
Didnt know if they were right at the end
Aside from specic elements of the VPA environment, the students had a lot to say about how they viewed this
method of assessment. The following are some excerpts from student responses (note that these responses have been ed-
ited for grammar and spelling).
I think instead of telling us if we were right or wrong, dont tell us at all because that makes it seem like a test and you
get nervous so you tense up and forget what you are supposed to answer.
This was very different and was a little bit more fun than pen and paper tests. It kept you thinking the whole time and it
was fun playing a game at the same time as taking a test.
Tests I take in school are stressful and more fact-based. This was more like learning.
ASSESSMENT 2: THERES A NEW FROG IN TOWN
To frame the discussion and illustrate the implementation of the design changes made from Assessment 1 to Assess-
ment 2, the following is a vignette of Assessment 2: Theres A New Frog In Town highlighting some of the changes in the
user experience while maintaining the same assessment outcomes. In May and June 2011, we intend to implement As-
sessment 2 with middle school students. Currently we have a potential population of N = 15,000. As a follow up to this
analysis we intend on using Item Response Theory to provide additional validation evidence of the MALQ-IVE.
Arielle sits at her computer and logs into the student portal. She opens the assessment and is immediately allowed to
choose what her avatar looks like. She selects and avatar and enters the world.
Figure 5. An example of a VPA avatar selection screen.
The ability for students to choose their own avatar (Figure 5) is a design decision that we hope will help highlight for
the students that they have a sense of autonomy and control for their assessment experience. Further, an aerial overview
of the farms helps provide situational awareness for students so that they are less likely to get confused and/or lost in the
The camera slowly provides an aerial view of the world to orient Arielle to the problem space. Arielle sees that there is a
village and what appear to be farms with ponds. The camera then focuses in on a multi-colored frog with 6 legs. Arielle
wonders, What could be causing this frog to have 6 legs? The assessment begins. A scientist and farmers who have
just discovered this mutated frog greet Arielle. The farmers all offer competing hypotheses for why the frog is mutated.
The scientist turns to Arielles avatar and tells her that she must conduct and investigation and come up with her own
theory and back it up with evidence. He asks her if she thinks any of the hypotheses are plausible.
Figure 6. An example of characters presenting competing hypotheses in a VPA.
Having the characters present competing hypotheses sets up the context of the assessment (Figure 6). This question
also allows us to identify in the assessment any misconceptions or prior knowledge that students may bring to the prob-
lem. Research has shown that although this is an important part of science inquiry that is often overlooked in assessments
(Sadler, 1998).
The scientist shows Arielle a science lab and tells her to come fnd him when she is ready. Arielle inspects the 6-legged
frog and puts in her backpack to investigate in the lab. She then walks around the village and sets out to explore the
At this point in the assessment, Arielle has a choice. She could have gone to the Internet kiosk and access information
there such as research articles. However, she has chosen to go explore. This choice is recorded on the back-end. We are
recording students choices and that are then be compiled into patterns. These patterns are then built and compared to
profles of students inquiry knowledge established during our cognitive task analyses. Because of this, the assessment
has a built-in framework that enables us to examine students intent and interpret their actions.
Figure 7. The Internet kiosk for conducting literature searches.
Before entering the frst farm, she is asked by the farmer what she is planning to do there. At the frst farm, Arielle says
she plans to collect a water sample. She enters the farm and collects a sample of the water. She also picks up a frog
and a tadpole to bring back to the lab and run some tests. She fnds a research article and starts reading it. It contains
information on tadpoles and viruses so she puts it in her backpack and decides to visit another farm. At this point, Arielle
has collected 5 pieces of data. Her backpack will only allow her to hold 8 pieces of data at a time.
Figure 8. An example of the VPA backpack containing a limited number of items.
Arielle will be forced to make a choice about what data she thinks is the most important or that she wants to investigate
frst. If students were allowed to pick up every piece of data in the world then it would be diffcult to make inferences
about their knowledge of what data is important evidence in the investigation. If students were asked to evaluate a piece
of data every time they collected it then the task would become boring. Thus, the design is requiring students to make a
choice through actions. She can go to the lab at any times to run tests on the data (e.g. water tests, blood test, genetic
test). Any piece of discarded data from the backpack will go back into the world and can be picked back up at any time
(given there is space in the backpack).
Arielle has collected 8 pieces of data from two farms. She does not want to discard any data and decides to go to the lab
to run some tests. She arrives at the lab and examines the water samples. Her tests show that the lab water and water
from one of the farms contains pesticides. However, one of the farms has clean water. She runs genetic tests on the 2
frogs she collects and sees that they are the same. She notes that both of the frogs have high counts of white blood cells.
She decides that she needs more evidence and goes to collect water samples from the other two ponds. At this point,
Arielle has spent her time collecting data and running tests.
Figure 9. An example of VPA lab results.
At the computer on her left, Maria has been tackling the assessment differently. As soon as Maria spoke with the scientist
she decided to go to the science lab. She examined the research that was available on frogs and tadpoles. She reads
about viruses and genetic mutations in frogs and decides to go gather data to determine which is the cause. She goes to
each of the four farms and collects a tadpole and a frog to run tests on. She gets back to lab and fnds that all of the frogs
have similar genetic make up. However, two of the tadpoles have small tails. She notes that a frog from the same farm
also has a virus in its blood. She looks up the virus in the research documents and believes she has found evidence. She
speaks to the scientist and builds a claim for why the frog is mutated, including evidence and then reasoning.
After class, Ms. Jones reviews the reporting tool to see the diagnoses the assessment provides about what each student
knows and does not know about the various sub-skills involved in science inquiry. The tool presents data at both the
individual and class level, and Ms. Jones fnds that, while the majority of students are strong in providing evidence
they are weak in reasoning from evidence. However, only a few of the students took Marias approach and went to the
research frst to seek information on the problem. The rest of the class spent time exploring and gathering data and
then analyzing it frst. Some of the students did not even use the research. Some only collected one or two pieces of data
before they attempted to make a claim.
The goal of both VPAs is for students to make choices based on sound science inquiry skills that advance the theory
that they are attempting to build. In both VPAs presented here, a students measure of science inquiry performance is
based on their in-world actions. Their actions and choices are given a range of scores and weightings that contribute to
an ongoing student model of science inquiry. They are temporally evaluated based on past, present, and future actions. In
other words, a choice is evaluated in terms of the previous actions, their actual choice within the context of the available
choices, and the outcome of their choice that sets the stage for the next set of actions. For example, if a character asks a
student what they think the problem is and the student responds that they think the mutant frog is a result of pollution,
the character will ask the student to provide evidence for their claim. The evidence that a student gives will be weighted
and evaluated based on their prior actions (data that they have previously collected) and by what they choose to present
as evidence. Although both Assessment 1 and Assessment 2 evaluate the same learning outcomes and that are tracked,
measured, and scored in the same way, the user experience in qualitatively and markedly different which we hypothesize
will provide a more valid, reliable, and accurate representation of the actual state of a students science inquiry abilities.
Chalmers, P. (2003). The role of cognitive theory in human -computer interface. Computers in Human Behavior, 19, 593-607.
Chooseco. (1977-2011). Choose Your Own Adventure. Retrieved January 30, 2011, from http://www.cyoa.com/public/index.
Clarke, J. (2009). Exploring the compexity of inquiry learning in an open-ended problem space. Unpublished Doctoral Disserta-
tion, Harvard University, Cambridge, MA.
Clarke-Midura, J., Dede, C., & Mayrath, M. (2010a). Designing immersive virtual environments for assessing inquiry. Paper
presented at the Annual Meeting of the American Educational Research Association.
Clarke-Midura, J., Dede, C., & Mayrath, M. (2010b). Ensuring the integrity of data in virtual immersive assessments. Paper pre-
sented at the Annual Meeting of the American Educational Research Association.
College Board. (2009). Science College Board Standards for College Success. New York, NY: The College Board.
Jonassen, D. H., Myers, J., & McKillop, A. (1996). From Constructivism to Constructionism: Learning with Hypermedia/Multi-
media Rather Than from It. In B. Wilson (Ed.), Constructivist Learning Environments (pp. 93-106). Englewood Cliffs, NJ,
USA: Educational Technology Publishers.
Mayer, R. E. (2009). Multimedia learning (2nd ed.). New York, NY: Cambridge University Press.
National Research Council (NRC). (1996). National Science Education Standards. Washington, DC: National Academie Press.
Nielsen, J. (1993). Usability Engineering. Boston, USA: Academic Press.
Nokelainen, P. (2006). An empirical assessment of pedagogical usability criteria for digital learning material with elementary
school students. Educational Technology & Society, 9(2), 179-197.
Pellegrino, J. W., Chudowski, N., & Glaser, R. (2001). Knowing what students know: The science and design of educational as-
sessment. Washington, DC: National Academies Press.
Preece, J., Rogers, Y., & Sharp, H. (2002). Interaction Design: Beyond Human-Computer Interaction. New York, NY, USA:
John Wiley & Sons.
Sadler, P. M. (1998). Psychometric models of student conceptions in science: Reconciling qualitative studies and distracter-
driven assessment instruments. Journal of Research in Science Teaching, 35(3), 265-296.
Shaffer, D. W. (2006). Epistemic frames for epistemic games. Computers and Education, 46(3), 223-234.
White, B., Collins, A., & Frederiksen, J. (in press). The nature of scientic meta-knowledge. In M. S. Khine & I. Saleh (Eds.),
Dynamic modeling: Cognitive tool for scientic enquiry. London: Spinger.
White, B., & Frederiksen, J. (1998). Inquiry, modeling, and metacognition: Making science accessible to all students. Cognition
and Instruction, 16(1), 3-118.
This research was supported by a grant from the US Department of Education, Institute of Educational Sciences
awarded to Chris Dede (IES# R305A080141).
Visualizing Summaries of Performance for Instructors Assessing Physical-motion Skills 195
Visualizing Summaries of Performance for Instructors Assessing Physical-motion Skills
NEIL C. ROWE, JEFF P. HOUDE, REY M. OSOTEO, RIQUI SCHWAMM, CORY R. KIRK, AHREN A. REED,
SAAD KHAN, CHRIS BROADDUS, AND CHRIS MENG
Automated surveillance technology has made important progress in recent years due to concerns about terrorism.
This technology can be used to track people and note suspicious behavior without human assistance. Traditionally the
technology has emphasized cameras and video analysis, but recent work has shown that much can be done with simpler
and cheaper sensors such as motion detectors.
Many important skills have a physical-motion component. Physical education (Hay, 2006) and theater arts (Smith-
Autard, 2004) prominently feature physical-motion skills. Many vocational skills such as operating machinery require
performers to go to different places to accomplish different things. Collaborative group work in a classroom often
involves discussions that require participants to move around. Motions in other activities such as studying can be meta-
phors for the mental activity of the student, and can be deliberately encouraged to enable better tracking, as by asking
students to visualize a conict using their hands and a map.
Instructors can be helpful in teaching physical-motion skills with a range of pedagogical strategies (Coker, 2004).
Such skills have long been difcult for instructors to assess, but technology is changing that. The cost of sensor technol-
ogy has greatly decreased in recent years, making it possible for routine education and training in an area 10 meters by
10 meters for around $1000 US.
We have been exploring surveillance technology for assessment of training of U.S. Marines (a kind of soldier). We
report on the visualization methods we have used in two prototype systems we have built, one involving nonimaging sen-
sors for the task of searching a building, and one involving multi-camera tracking for the task of patrolling an urban area.
Both are important but challenging because of the variety of simultaneous concerns that Marines must handle. The task
for instructors is difcult as well because of the constantly changing locations and angles of view, occlusions by walls,
and need of trainees to improvise in response to changing circumstances. Thus automated assessment (grading) could
be provide useful feedback to trainees. In military training, feedback is most naturally done in the standard post-exer-
cise assessment or after-action review (Hixson, 1995); current assessment predominantly uses checklists for instructors
(Hone et al, 2008) which poorly map to their visual experience.
VISUALIZATION METHODS FOR INSTRUCTORS
The rst thought of instructors in using technology to assess physical-motion skills is to look at video. But video
poses many difculties. Cameras must be placed carefully to obtain a view unobstructed by the students bodies, walls,
and other obstacles. Unobstructed views at times may be impossible when subjects move around considerably, as with
team sports. This suggests using multiple cameras, but that is expensive and it is easy for people to become confused
in correlating one video stream with another. The main challenge of video, however, is that it provides a considerable
amount of data and this is difcult to search. An instructor who wants to locate errors when they are rare (as with stu-
dents who have progressed beyond the basics) must watch large amounts of uninteresting video. Even when interesting
events are found, a view may be insufciently clear to show to students, who do not have the analytical skills of an in-
structor and need pictures to be unambiguous.
A better approach is to summarize automatically what happened. Summarizing and indexing of video is difcult in
general because of the many kinds of human activity and the many perspectives from which we may view it. However, it
can be simplied with two restrictions: We can extract only the overall location and orientation of subjects, and look only
for unusual events as predened. Locating and orienting subjects in video is considerably easier than classifying what
they are doing as in (Minnen et al, 2007), and if cost is a factor, can be done with relatively inexpensive non-video sensor
hardware. Detection of unusual events can be done by computing statistics of normal activity and comparing observa-
196 Rowe, Houde, Osoteo, Schwamm, Kirk, Reed, Khan, Broaddus and Meng
tions to it; often it is easier to detect that something unusual has happened than to classify it, but often instructors can
gure out what happened once an interesting time is identied to them.
In assessing what happened in a time period for a group of students, the four dimensions are time, space, student,
and type of event. Event can mean mistakes (like getting too close to someone else) or key transitions (like different
periods of a choreography). Displaying all four dimensions together is generally too much information for instructors.
(Using three-dimensional displays to encode more information is expensive and hard to make work consistently, and
using display time as an additional dimension provides something too similar to video.) Instead we should summarize
for instructors and let them drill down to nd further details they need. A number of similar such systems have been
developed for analysis of multidimensional sensor data, e.g. (Morreale et al, 2010) for environmental monitoring. Ulti-
mately, instructors should be able to drill down to the video.
Several kinds of summarization displays are helpful for instructors. The most direct approach is to plot two dimen-
sions chosen from time, space, student, and type of event while encoding a third in a different way such as by color.
There are four basic such displays:
We can plot events over time for different students on parallel timelines to indicate how events tend to
bunch up, color-coding by type of event.
We can plot events over time for different types of event on parallel timelines to indicate how each event
tends to occur, color-coding by student.
We can plot events over space for each student to show the places where events occur, color-coding for type
of event
We can plot events of a particular type for all students over space to show where particular types tend to
occur, color-coding by student.
Instructors will need to refer to each of these four kinds at different times, but often one kind will be primary for a
particular application. For instance for Marine training, the rst kind of display is primary for after-action review, since
performance of individuals is usually considered in sequence.
It is also often useful to supplement these displays with plots of direct parameter measurements of students over
time. For instance, in choreography it is useful to plot visibility to lines of sight over time, and in team sports it is useful
to track speed of the student.
When students work as a team, another useful display focuses on the group of students as a whole. Plotting the cen-
troid (location of their center of mass) and dispersion (deviation of locations from the centroid) can indicate coordination
problems and inconsistencies. For instance in team sports, it is useful to measure how well the students cover the eld of
play. Since the centroid averages the effects of all students, its pattern in time will be less noisy than that of individuals,
and more able to indicate fundamental problems such as a group veering too far from their target position.
It is often useful to display comparisons of student performance at corresponding times in different rehearsals or
run-throughs, as with theatrical performances or crisis-procedure rehearsal with training of power-plant operators. Com-
parisons of locations over time or time progression through the exercise can make the problems more obvious than when
considered in isolation.
Finally, it is useful to display averages over all students on a particular exercise, all students engaged in a particular
event or making a particular mistake, or particular students over all exercises. This allows us to see respectively things
like the difculty of an exercise, the types of students making a mistake, and the relative accomplishments of students.
ASSESSMENT OF INDOORS TRAINING USING NONIMAGING SENSORS
One system we have built is for automated assessment of Marine performance in searching a building for people and
contraband, an urban-warfare skill. It is especially hard for cameras to see what occurs indoors because of occlusion by
walls and furniture, and there are additional difculties with the low levels of light and the lens distortions at short dis-
tances. These pose challenges for automated image processing. Unfortunately we cannot attach accelerometers to Ma-
rines bodies during training as in (Chen and Hung, 2009) because that interferes with their ability to move naturally and
requires unavailable extra time to set up and remove for every training exercise.
We used instead nonimaging sensors for tracking. High accuracy of localization is not essential, as it is more im-
portant to tell whether Marines searched an area at all than to determine their exact paths. Our experience with infrared
motion sensors and inexpensive sonars (Rowe et al, 2011) has shown them to be good at detecting people up to 5 meters,
which provides good coverage for much indoor terrain. Sensors can also be placed on weapons to detect their orienta-
tions, a key concern of Marine instructors. So we built a prototype system using infrared, sonar, light, and audio sensors.
(U.S. Marine Corps, 1998) summarizes Marine doctrine on the searching or clearing of a building. Since this is
often done in dangerous areas and involves interaction with the public, Marines must be well trained. There are rules for
preferred methods of entry, where to point weapons, and how to coordinate with other Marines. Most of these skills are
employed in teams of 2-4 Marines. Some of the most important considerations are the following constraints which we
attempted to address:
c1. Marines should aim their weapons to jointly cover as much as possible of a dangerous area.
c2. Marines should maintain a stacked orientation when entering a door with one Marine behind the other.
c3. Marines should thoroughly survey an area they are asked to search.
c4. Marines should identify suspicious objects and leave them for trained personnel.
c5. Marines should communicate important information to the other Marines and civilians.
c6. Marines should avoid pointing their weapons at one another (agging).
EXPERIMENTAL SETUP
We used much of the same sensor hardware and software as in our research on monitoring of public places (Rowe et
al, 2011). Sensors other than microphones were commercial off-the-shelf products from Phidgets (www.phidgets.com)
and used a simple hardware interface and software. We polled them around 40 times per second and downloaded values
to a le. Audio was collected using Audacity software running cardioid microphones with preampliers attached to USB
ports on computers.
Our experiments used eleven laptop computers to run the sensors. Four computers ran just microphones. Two of the
other seven were loaded into backpacks and carried by students simulating Marines in a scenario. Four of the remaining
ve were stationed at table height in the rooms searched, and the sensors other than light sensors were oriented parallel
to the oor to avoid gait effects associated with legs and heads. The remaining set of sensors were stationed at ground
height and monitored two doors.
The ve sets of stationary sensors all had a microphone, an infrared narrow-range sensor, a motion sensor, a light
sensor, and a simple sonar sensor. Two sets also had force sensors which were placed under suspicious objects to detect
if the objects were picked up. Sensors were oriented in a variety of directions parallel to the oor to provide broad cov-
erage. The sensors on the Marines were orientation sensors for the weapons, sonar sensors for the weapons, and vibra-
tion sensors. The orientation sensors had three-axis accelerometers, rotation sensors, and magnetometers. We summed
the rotation values over time to estimate orientation angles in azimuth (yaw), inclination (pitch), and roll; our primary
concern is azimuth (where the weapon is aimed horizontally), secondarily inclination (the degree to which the weapon is
horizontal). Use of Runge-Kutta integration improved the accuracy of these estimates. Experiments showed that yaws
were accurate within ten degrees over a ve-minute period. For further accuracy, the magnetometer values can correct
for the accumulated error over time; although indoors we found signicant deviations from compass north, these could
be learned from experiments.
We devised 14 building clearing scenarios in which the two Marines cleared two rooms and a corridor. Each sce-
nario was done as a separate experiment. Each involved searching a small room and a large room, taking a civilian they
discovered into custody, possibly throwing a fake grenade, and possibly discovering suspicious objects. The 14 exercises
were in pairs of correct execution followed by incorrect execution. Incorrect methods entailed violating of constraints
listed above. One pair of exercises was the basic procedure; one involved a grenade; one involved with noting booby
traps; one involved moving counterclockwise instead of clockwise; one involved more detailed actions with booby traps;
one involved moving unusually slowly; and one involved repeated actions when the rst action was done incorrectly.
Detailed scripts for each exercise were provided to subjects who simulated Marines
PERFORMANCE ASSESSMENT
We are only interested in sensor values that signal human activity. For most of our sensors, these are abnormal val-
ues. For the infrared and force sensors, this was two standard deviations above the mean value; for the light and sonar
sensors, this was two standard deviations below; and for the motion sensors, this was two standard deviations either
above or below. Means and standard deviations were calculated separately for each sensor because they did uctuate
with position; the sonar, in particular, gave a mean value in the absence of people that corresponded to the distance of the
nearest large surface. This thresholding of signals gave in our experiments a reduction of the data to 14% of its initial
volume for the non-microphone sensors, resulting in an average of about 6 points per second per sensor. The audio of
20,000 samples per second was averaged to get a compressed signal of 200 hertz, and peaks of width roughly 0.1 sec-
onds were sought (to aid in nding footsteps while not ignoring other loud sounds). To further reduce spurious audio
peaks, we excluded peaks whose heights were less than two standard deviations above the mean peak height, giving an
average of one audio peak per second.
Figure 1 shows an example plot of such exceptional events during exercises 11 through 14 for a set of sensors run-
ning off a single computer. Time is measured in seconds. Exercises 11 (time 150 to 300) and 13 (time 600 to 750) were
done correctly, and exercises 12 (time 400 to 550) and 14 (time 750 to 900) were done deliberately incorrectly. Row 2 is
the microphone sound-energy peaks, row 9 is the estimated periods of human speech, row 12 is the narrow-range infra-
red sensor, row 14 is the motion sensor, row 15 is the light sensor, row 17 is the sonar sensor, and row 18 is the force sen-
sor. The poorly performed exercises clearly have a different pattern. Analogous plots are being used in applications such
as healthcare monitoring, e.g. (Wang and Skubic, 2008) where changes in activity patterns of patients are clearly visible
in comparisons even if they are hard to see in examining a single plot.
Assessment of the Marines can be done for each constraint except c6. Since we try to emplace sensors to cover the
training area well, constraint c3 is assessed by checking whether a sufciently signicant record of activity is found at
each set of sensors. For this we used the rate of occurrence of any sensor reading over its threshold at a particular loca-
tion, and compared against the rate for correct performance. Rates that are signicantly different trigger a ag on con-
straint c3. We can see an indication of loitering in the density of the indicators at time periods 400-450 and 820-860 for
sensors 14 and 15.
Constraint c4 is assessed by noting whether the force sensor indicates that a suspicious object was moved or picked
up (bad), and whether other sensor readings indicated that the Marine was inspecting the object (good). Trainees picked
up suspicious objects at times 415 and 820 in Figure 1.
Figure 13: Example plot of exceptional events at one sensor for indoor training.
Constraint c5 can be addressed by noting speech that occurred during the exercise. We estimated this from noting
when the amount of sound energy in the 100-400 hertz frequency range was at least 25% of the total sound energy in
the range of 1-2000 hertz. Figure 1 plots times exceeding this threshold on the sensor 9 line. The experimenters were
talking for time periods 0-100 and 900-1000.
Constraint c2 can be measured by the amount of time the Marines take to traverse a door or entryway, which we
measure with sonar sensors across the door. We compare this time to that of normal traversal of a door, and signicantly
shorter times trigger a ag on constraint c2. Figure 1 does not show this, but saw this for microphones closer to the door.
Constraint c1 concerns weapons handling. Chiey we are interested in the coverage of the area by weapons (or
how wide a range of locations are pointed at by the weapons in the horizontal plane). A way to approximate this for two
Marine weapons is the average difference in the yaw angle over the exercise weighted by the degree to which the
weapons are horizontal:
0.5(1 (1/ ) min(| |, 2 | |))(| cos | | cos |) D = + + where is the
yaw and is the pitch angle in radians (where zero is horizontal). This ranges from 0 to 1 and represents the fraction of
azimuth range that is adequately covered by the two weapons, discounted by the angle of inclination of the weapons (so
a weapon pointing at the ground does not contribute any coverage). Values of D that are 50% or less of those of normal
runs trigger a ag on constraint c1. Figure 2 plots this D metric for the two weapons in the experiments in Figure 1. It
can be seen that coverage was poor for the second (400 to 550) and fourth (750 to 900) experiments, the bad runs, ex-
cellent for the third experiment (600 to 750), and marginal for the rst experiment (150 to 300).
0 100 200 300 400 500 600 700 800 900 1000
Coverage by pair of weapons
Time in seconds
Figure 14: Coverage metric values in the experiments of Figure 2.
A disadvantage of our noninterventionist approach to tracking is that we could not localize trainees precisely, only
note when they were nearby to sensors. We tried double integration of the accelerometers to estimate positions, but this
was too error-prone; and location by satellites using GPS requires a signicant antenna which would have added weight
to the backpacks. This means we could not monitor constraint c6 (accidental pointing of weapons at fellow Marines) as
we could on the other system to be described. Nonetheless, we judged that we did well on automated assessment of the
other constraints.
Assessment of outdoors training through multi-camera tracking
We also conducted research on automated assessment of outdoor Marine training as part of the BASE-IT project
(Rowe et al, 2010). We did experiments with real Marines who followed a set of scenarios. Multi-camera fusion, GPS
(Global Position System) devices, and motion-smoothing lters were applied to obtain relatively smooth estimates of
paths followed by the Marines, using technology from Sarnoff Laboratories in Princeton, New Jersey, US (Cheng et al,
2009); all three were necessary to obtain sufcient accuracy. (Similar tracking technology for training is in the Ubisense
system of General Dynamics, www.ubisense.net.) Orientation sensors on weapons and helmets were used in response to
difculties in obtaining adequate orientation estimates from video, though our goal is to eventually rely exclusively on
the latter.
BASE-IT automatically evaluates the performance of a group of 4-20 Marines (a re team or a squad) in a realistic
setting. It goes well beyond previous work in virtual environments such as (Lampton et al, 2005) since it handles real-
world data. Again we used (U.S. Marines Corps, 1998) as our reference. The primary concerns of instructors relate to
safety, including how well the Marines are looking in all directions (360-security), degree of exposure to potential
snipers (danger), and how far apart the group of Marines are (dispersion). Other concerns are collinearity (they
should not form a single line), coverage with their weapons (they should cover the major threats), speed (they should not
be too fast or too slow), mobility (they should be able to take cover), centrality of the leader, and interactions with role
players simulating local residents (more interactions during training are better). Again, assessment is based on positions
and orientations of the Marines. Also used is a preanalysis of a graphics model of the terrain to nd potential sniper po-
sitions, including windows, doors, and corners of buildings (depending on the angle of view); larger areas of unobstruct-
ed terrain are also assigned a small threat level.
Data was obtained from the tracking database using C++ code, and visualized with Python routines. Figure 3 shows
an example of a direct parameter visualization, for the count of the number of clusters formed by seven Marines (with a
threshold of 10 meters) during an exercise. Low values represent times when Marines came unnecessarily close to one
another, which is undesirable.
Figure 15: Example plot of number of clusters of Marines during an exercise.
We also plot exceptional values of important parameters, occurrences of which we call issues rather than mistakes
since they may have legitimate excuses. Issues for individual Marines include a Marine being too close to another, a Ma-
rine failing to cover a nearby window or door with their weapon, and a Marine accidentally pointing a weapon at another
Marine. Issues for the group of Marines include forming too few clusters, moving too fast, failing to cover all directions
adequately, and returning to a location visited previously. Figure 4 shows a time plot of individual issues for one train-
ing experiment where colors indicate Marines. Personnel labeled 0 were two roleplayers, and three Marines did not
have any issues during this exercise.
Spatial plots provide additional perspectives on the data. Figure 5 shows locations of occurrences of Marines com-
ing too close to one another (issue type 1). Black rectangles indicate tents, and colored dots represent Marines with the
same encoding as in Figure 4. Although this was agged, Marines do have legitimate excuses to be near the tents be-
cause the tents were small and they need to take cover behind them. Figure 6 plots all the issues (the larger circles) for
one Marine during one exercise; small green circles represent normal behavior. The Marine had problems with pointing
his weapon, mostly in aiming too closely to other Marines, and some in regard to covering the window of the northern-
most building.
Figure 16: Example plot of issues for individuals over time during an exercise.
Figure 17: Example plot of agging (pointing weapons too much towards one another).
Comparison plots for different runs were useful for this task because the Marines were being rehearsed. Figure 7
shows an example comparison plot of the centroids of the Marines in exercises 490 and 496. Clearly the Marines slowed
down and responded to suspicious activity in the latter part of 496 that did not occur in 490. Figure 8 shows a different
kind of comparison, of the number of issues observed in three different runs. Here redness (on a continuum of blue to
purple to red) indicates the degree of seriousness of the rate for each kind of issue.
Figure 18: Example plot of all issues for Marine 3911.
Figure 19: Comparison of distance traveled and locations in exercises 490 and 496.
Figure 20: Comparison of group issue counts for three exercises.
Danger and its coverage by the trainees are important but complicated, so we have special visualizations for them.
Marines are expected to pay attention to possible sniper positions at visible windows, doors, corners, and centers of large
areas, and weight them by distance. We precompute threat levels at evenly spaced sample points in the terrain for each
possible sniper position. Then for a particular Marine location, we look up the threats for the nearest sample point and
their seriousnesses, and check how well the Marine is viewing them and covering them with their weapon, based on their
body orientation and weapon orientation yaws. We total these values for a group of Marines to measure how well they
are doing. A coupled pair of diagrams is helpful. Figure 9 plots threat awareness over space at a particular time, and
Figure 10 plots the product of the danger and unawareness (one minus awareness). In both diagrams, the degree of red-
ness on a scale red-purple-blue indicates the degree of danger. Note that Figure 9 shows the Marines are not covering
the east side of the terrain very well, but Figure 10 shows that those regions are not critical to their safety. But they could
better cover the north end of the left road (the purple-colored circles stacked vertically) as well as the south side of the
terrain as indicated by the purple dots.
Figure 21: Example plot of threat awareness at one instant of time.
Figure 22: Danger weighted by awareness for the situation in Figure 6.
Our approach can be used to enhance a wide range of training experiences involving physical motion. Much of our
hardware is inexpensive and could be reused many times for different kinds of training. Large numbers of sensors are
not required when the action of students is conned to a relatively small area of 10-100 feet square. If cameras are used,
there need not be many, and cameras may be unnecessary for tracking in many cases. We have shown that a set of visu-
alizations are useful on the sensor data and straightforward to provide.
This work was sponsored by the U.S. Navy Modeling and Simulation Ofce, the Ofce of Naval Research as part
of the BASE-IT project, and the National Science Foundation under grant 0729696. Opinions expressed are those of the
authors and not necessarily those of the U.S. Government.
Chen, Y.-J., Hung, Y.-C., 2009. Using real-time acceleration data for exercise movement training with a decision tree approach.
Proc. 8th Intl. Conf. on Machine Learning and Cybernetics, Baoding, CN, July 2009.
Cheng, H., Kumar, R., Basu, C., Han, F., Khan, S., Sawhney, H., Broaddus, C., Meng, C., Su, A., Germano, T., Kolsch, M.,
and Wachs, J., 2009. An instrumentation and computational framework of automated behavior analysis and performance
evaluation for infantry training. Proc. I/ITSEC, Orlando, Florida, December.
Coker, C., 2004. Motor learning and control for practitioners. McGraw-Hill, New York.
Hay, P., 2006. Assessment for learning in physical education. In Kirk. D., MacDonald, D., OSullivan, M., The handbook of
physical education, Sage, London.
Hixson, J., 1995. Battle command AAR methodology: a paradigm for effective training. Proc. 27th Winter Simulation Conf.,
Arlington, Virginia, USA, pp. 1274-1279.
Hone, G., Swift, D., Whitworth, I., and Farmilo, A., 2008. The case for coarse-grained after action review in computer aided
exercises. Intl. Command and Control Research and Technology Symposium, Bellevue, WA, June.
Lampton, D., Cohn, J., Endsley, M., Freeman, J., Gately, M., and Martin, G., 2005. Measuring situation awareness for dis-
mounted infantry squads. Interservice/Industry Training, Simulation, and Education Conference.
Minnen, D., Westeyn, T., Ashbrook, D., Presti, P., Starner, T., 2007. Recognizing soldier activities in the eld. Proc. Conf. on
Body Sensor Networks, Aachen, Germany, March.
Morreale, P, Qi, F., Croft, P., Suleski, R., Sinnicke, B., and Kendall, F. 2010. Real-time environmental monitoring and notica-
tion for public safety. IEEE Multimedia, 17 (2), pp. 4-11, April-June 2010.
Rowe, N., Houde, J., Kolsch, M., Darken, C., Heine, E., Sadagic, A., Basu, C., and Han, F., Automated assessment of physical-
motion tasks for military integrative training. Proc. Second International Conference on Computer Supported Education,
Valencia, Spain, April 2010.
Rowe, N., Reed, A. Schwamm, R., Cho, J., and Das, A., 2011. Networks of simple sensors for detecting emplacement of impro-
vised explosive devices. Chapter 16 in Critical Infrastructure Protection, ed. F. Flammini, WIT Press.
Smith-Autard, J., 2004. Dance composition, 5th ed. A&C Black, London.
U.S. Marine Corps, 1998. Military Operations on Urbanized Terrain, Publication MCWP 3-35.3.
Wang, S., and Skubic, M., 2008. Density map visualization from motion sensors for monitoring activity level. Proc. IET 4th
International Conference on Intelligent Environments, Seattle, WA, US, pp. 1-8, June 2008.
Learning Presence in the Community of Inquiry Model: Towards a Theory of Online Learner Self- and Co-regulation 207
Learning Presence in the Community of Inquiry Framework: Towards a Theory of Online
Learner Self- and Co-regulation
PETER SHEA
Online learning in US higher education has exploded in recent years. Studies by the US Department of Educa-
tion (Parsad & Lewis, 2008) indicate that online students generated more than 12 million course enrollment in 2007-
2008 with more than one in four of all American college students enrolled in at least one online course (Allen & Sea-
man, 2010). For learners who were married with dependent children that percentage rises to one in three (Staklis, 2010).
Given that todays growth in distance higher education continues to be driven largely by developments in asynchronous
online learning (Allen & Seaman, 2008; Parsad & Lewis, 2008; U.S. Department of Education, National Center for Edu-
cation Statistics, 2008) it is necessary that we focus our attention on these rapidly growing environments
Recent large scale meta-analytic research indicates that success rates for online students are at least equivalent (Ber-
nard, Abrami, Lou, Borokhovski, Wade, Wozney, Wallet, et al., 2004; Allen, Bourhis, Burrell, & Mabry, 2002; Tallent-
Runnels, Thomas, Lan, Cooper, Ahern, Shaw, et. al., 2006; Zhao, Lei, Yan, Lai, & Tan, 2005) and may be better than
(Means, Toyama, Murphy, Bakia & Jones, 2009) those of classroom students. Means and her colleagues (2009) con-
ducted the most recent meta-analysis including an exhaustive search of 1,132 studies that compared online and face-to-
face conditions. Reviewing studies that investigated elements of online learner self-regulation (e.g., Bixler, 2008; Chang,
2007; Chung, Chung & Severance 1999; Cook, Dupras, Thompson &. Pankratz, 2005; Crippen & Earl, 2007; Nelson,
2007; Saito & Miwa 2007; Shen, Lee & Tsai 2007; Wang, Wang, Wang, & Huang, 2006) the authors found advantageous
outcomes for scaffolding meta-cognitive learning strategies including self-reection, self-explanation, and self-moni-
toring. These positive ndings for online learner self-regulation represent fertile ground for the development of a more
comprehensive explanatory model for understanding the potential benets of online instruction, a task to which we now
turn.
The present study is grounded in several complementary theoretical perspectives reective of research on the com-
plexity of interaction in online classrooms. Rex, Steadman, & Graciano, (2006) suggest that such inquiry falls within
seven traditions and this study contains elements of several of these including process-product, cognitive-constructiv-
ist, socio-cognitive, and situative perspectives. From a process-product perspective we are interested in how online dis-
course informs and shapes jointly created products developed by collaborative teams. This study also attempts to bridge
cognitive and socio-cognitive/situative perspectives by introducing concepts in learner self regulation within the Commu-
nity of Inquiry framework (Garrison et al. (2000) - see below).
The CoI framework is based on a model of critical thinking and practical inquiry. The authors posit that learning oc-
curs through the interaction of students and their instructor and is manifest as three integrated elements that contribute
to a successful online learning community: social presence (SP), teaching presence (TP), and cognitive presence (CP).
The framework theorizes online knowledge building as the outcome of collaborative work among active participants in
learning communities reecting instructional orchestration appropriate to the online environments (teaching presence)
and an encouraging collegial online setting (social presence). The teaching presence construct delineates task sets such
as organization, design, discourse facilitation, and direct instruction (Anderson, Rourke, Garrison, & Archer, 2001) and
articulates the behaviors likely to result in a productive community of inquiry (e.g. Shea, Li, Swan & Pickett, 2005). So-
cial presence represents online discourse that promotes positive affect, interaction, and cohesion (Rourke, Anderson, Gar-
rison, & Archer, 1999) that support a functional collaborative environment. The model also includes cognitive presence,
a multivariate measure of critical and creative thinking that results from the cyclical process of practical inquiry within
such a community of learners. The specic form of interaction within the cognitive presence construct thus reects a
pragmatic view of learning (Dewey, 1933; Lipmann, 2003, Pierce, 1955)
Given that learner self regulation is a well researched construct compatible with multiple theories of learning (see
e.g. Zimmerman, 2001) its introduction has the potential to enhance the CoI framework, which foregrounds the social
construction of knowledge with less focus on the self and co-regulatory strategies that successful individual learners may
employ. We propose that the introduction of learner self regulation can better enhance the scope of the CoI framework.
Recent work on the CoI model (Shea, Vickers, &Hayes, 2010) suggested that previous research methods may have
resulted in a systematic underrepresentation of the instructional effort involved in online education. Using quantitative
208 Shea
content analysis these authors examined course documents within and external to threaded discussion areas and con-
cluded that the majority of teaching presence in two undergraduate online business courses occurred outside of threaded
discussion (emails, assessments, private folders, etc), areas that are generally not included in past investigations of the
teaching presence construct (e.g,. Akyol & Garrison, 2008; Anderson, Rourke, Garrison & Archer, 2001; Bliss & Law-
rence, 2009; Coll, Engel & Bustos, 2009; Pawan, Paulus, Yalcin & Chang, 2003). This ongoing project to document all
instances of teaching, social, and cognitive presence in complete online courses also resulted in identication of learner
discourse that did not t within the model, i.e. could not be reliably coded as indicators of teaching, social, or cognitive
presence (Shea, 2010; Shea, Vickers & Uzuner, 2010). These exceptions represent interesting data for rening and en-
hancing the model as they suggest that learners are attempting to accomplish goals that are not accounted for within the
CoI framework. This research concluded that the learners under investigation engaged in discourse on course logistics
including collaborative attempts to understand instructions provided to them by the course professor. Learner discus-
sions also included strategic efforts to divide up tasks, manage time, and set goals in order to successfully complete
group projects. As such they appeared to be indicators of online learner self and co-regulation, which can be viewed as
the degree to which students in collaborative online educational environments are metacognitively, motivationally, and
behaviorally active participants in the learning process (Zimmerman, 1986).
Self Regulated Learning- Learning Presence
Research on self-regulated learning indicates that, it is viewed as especially important during personally directed
forms of learning, such as discovery learning, self-selected reading, or seeking information from electronic sources, (but
is) also deemed important in social forms of learning. (Zimmerman, 2008). Given the electronic, social, and self-
directed nature of online learning, it seems imperative that we examine learner self- and co-regulation in online environ-
ments especially as they relate to desired outcomes such as higher levels of critical and creative thinking as described in
the CoI framework. Accomplishing this goal requires that we examine a wide variety of issues including meta-cognitive,
motivational, and behavioral traits and activities that are under the control of successful online learners and which past
research indicates may be fostered in online environments (Means, et. al, 2009). We suggest that this constellation of
behaviors and traits may be seen as elements of a larger construct learning presence (Shea, 2010). We suggest that the
name learning presence integrates with the other forms of presence in the CoI framework and reects the proactive
stance adopted by students who marshal thoughts, emotions, motivations, behaviors and strategies in the service of suc-
cessful online learning. Learning presence thus indicates the exercise of agency and control rather than compliance and
passivity and more fully articulates popular beliefs about the importance of self direction in online environments.
In previous research (Shea, Vickers & Uzuner, 2010) student discourse that occurred in certain collaborative activi-
ties could not be reliably coded as teaching, social, or cognitive presence. Given that these activities are core to learner-
centered approaches to online education we felt it essential to integrate them into the CoI framework. This required that
we look for additional theoretical grounding to attempt to understand and categorize the discourse. We used concepts
from a variety of sources (Azevedo, Crompley, & Seibert, 2004; Azevedo, Guthrie, & Seibert, 2004; Curtis & Lawson,
2001; Zimmerman, 1989) that examined self regulated learning and collaboration to accomplish this goal. Using concep-
tual element from these sources we then searched for patterns of self and co-regulation within areas of the course where
previous attempts to code student discourse using standard CoI indicators proved unreliable. This discourse included
student debate preparation areas, ask a question areas as well as within two of the discussions in two courses. As a
result of this exploratory analysis, we developed a coding scheme that represents what we call learning presence (see Ap-
pendix A).
Once the coding scheme was established, two researchers analyzed all of the communicative processes in the two
courses, including all of the discussions, debate preparation areas and debates, as well as ask a question sections. The
goal of this analysis was to determine if the theory derived codes indicating self and co-regulatory behaviors would be
evident in online learner discourse. The researchers met to compare results of coding and established an inter-rater reli-
ability metric using Cohens kappa and Holstis coefcient of reliability. Cohens kappa measures the achieved beyond-
chance agreement as a proportion of the possible beyond-chance agreement (Sim & Wright, 2005, p.258). A limitation
with Cohens kappa is that symmetrical imbalance in the marginal distribution of agreements can result in a signicantly
lower kappa (Feinstein & Cicchetti, 1990). In order to compensate for this, we employed Holstis coefcient of reli-
ability (Holstis CR), which reports the simple agreement of raters.Lombard, Snyder-Duch, and Bracken (2002) suggest
utilizing multiple reliability indices in order to when interpreting interrater reliability, and past CoI research have utilized
this approach (see Garrison, Anderson, & Archer, 2001; Rourke, Anderson, Garrison,& Archer, 1999; Shea, Hayes, &
Vickers, 2010; Shea, Hayes, Vickers, Gozza-Coheh, et. al., 2010).Initial and negotiated interrater agreement was estab-
lish for each coding session. The benets of this approach is that coders can locate transcription errors as well as rene
the coding scheme.
When the patterns had been identied, we collated the data by categories. From these data we made charts indicating
the number of occurrences for our major categories: forethought/planning, monitoring, and strategy use.
Table 1 (see Appendix B) reects the initial and negotiated inter-rater reliability for both Instructor A and Instructor
B. Initial kappa for Instructor A was xx-xx, and initial CR was xx-xx. Initial kappa for Instructor B ranged from 0.2875-
0.8140, and initial CR ranged 0.9025-1.0000. Discussion IRR can be seen in Table 2 (see Appendix B). Initial kappa for
Instructor A was xx-xx, and initial CR was xx-xx. Initial kappa for Instructor B ranged from 0.3086-0.7460, and initial
CR ranged 0.8867-0.9759. Variation in kappas may have been a result of symmetric imbalance of no codes for LP indi-
cators. As reected by Holstis CR, however, inter-rater agreement is well above the recommended 0.70 IRR for explor-
atory research of this nature (see Lombard et al., 2002).
Learning Presence has different components (forethought/planning, monitoring, and strategy use ) that are contextu-
ally dependent. All three are evident in preparation area, only strategy use appears in actual discussions. It appears that
student self and co-regulation is triggered by the kinds of activities that learners are asked to complete. Chart 1 provides
a visual representation of the levels of learning presence that are evident in two illustrative learning activities.
Chart 1: Forethought (FP), Monitoring (MO) and Strategy Use (SU) in different online activities
As can be seen there is forethought/planning, monitoring and strategy use in the areas where students were instruct-
ed to prepare for a class debate.
Charts 2 and 3 also indicate that learning presence varies by type of activity students are asked to complete. Chart
2 shows that while there is a relative paucity of learner self and co-regulation in traditional activities such as standard
threaded discussions, learning presence is more frequent in online preparatory areas where students must collaborate
actively to be successful. Cognitive presence levels also increase in such activities. Chart 3 documents that when the
activity turns back to the standard threaded discussion format, we see a relative decline in indicators of learner self and
co-regulation as well as a decline in cognitive presence, though a slight increase in social presence.
Chart 2: Instances ofTP: Teaching Presence, SP: Social Presence, CP: Cognitive Presence, and LP: Learning Presence in
course B
Chart 3: Forms of presence in debate preparation areas and debate discussion in course B
As can be seen in charts 4 and 5 there is a correlation between learning presence and social presence. Chart 4 indi-
cates that learning presence density per small group is correlated with social presence density within collaborative activi-
ties. Additional analysis show a similar pattern when the message rather than the group is used as the unit of analysis.
The average frequency of learning presence per message is correlated with the average frequency of social presence per
message in collaborative activities.
Chart 4: Learning Presence density per small collaborative group
DISCUSSION AND IMPLICATIONS
Decades of research indicate that learner self regulation is an important predictor of learning, yet self regulated
learning in online environments in a topic that is not well understood. This gap is important to address given the near
universal recognition that online learning requires signicant self direction and regulation.
The CoI model represents a powerful framework for understanding online learning in collaborative pedagogical en-
vironments. While it represents an ideal in which teachers and learners perform the same roles (expressed as teach-
ing presence), it ignores some of the real world dynamics that shape and constrain much of online learning in practice.
Learners and instructors do not perform identical roles and thus must engage in different behaviors to succeed. For ex-
ample, in this study, forethought and planning, monitoring and strategy use exhibited by students is quite distinctive from
instructional design, facilitation of discourse, and direct instruction that characterize teaching presence. Because learn-
ers are accountable for their learning they exhibit distinctive behaviors, motivations, and strategies that are more or less
adaptive and effective we suggest that these are indicative of a distinct learning presence in online environments.
Specifying component of online learner self- and co-regulation is necessary. We accomplished this through and
examination of discourse that occurred in two fully online courses. This student-student discourse had previously been
coded (unsuccessfully) using coding schemes that prior researchers have developed for the Community of Inquiry frame-
work. We identied concepts, examples, and patterns of learning presence within hundreds of instances of online dis-
course. We believe that the categories and examples presented through this new coding scheme represent a benecial
extension to the CoI model.
Learning presence is evident where learners are asked to actively collaborate. Far less forethought/planning, moni-
toring of learning and strategy use appear in whole class discussion than in collaborative activities Asking students to
collaborate more deeply fosters learning presence. Teaching presence is thus the route to better learning presence but
relationship between forms of presence is probably reciprocal given the correlation identied here.
Kuhn (1977) argued that determining the superiority of one theory over another was a matter of weighing compet-
ing values including accuracy, consistency, scope, and fruitfulness among others. In this paper, we examine the degree to
which the CoI model represents an advantageous theoretical framework for understanding online learning based on some
of these criteria. We suggest that the addition of learning presence to the CoI model represents progress by these stan-
dards. Again, the roles of teachers and learners may be similar in collaborative environments, but they are not identical.
Further explicating the roles of online learners by drawing on theories of learner self regulation provides added scope,
accuracy and we believe will make the CoI model increasingly fruitful in describing and explain online learning.
APPENDIX A: LEARNING PRESENCE CODING SHEET (INCOMPLETE LIST SAMPLE CODES ONLY)
Forethought and Planning
Original Codes & Source Description Example
FP1 - Goal setting
(Zimmerman, 1989)
Online learner discourse establishing
desired tangible/intangible outcomes
At the end of next week, as a team, we
have to submit a summary of our discussion
points.
FP2- Planning
Online learners considering approaches,
procedures, or tasks to be used to attain
goals.
Why dont we list (all of us) what we perceive
to be the cons of outsourcing.
FP3 - Coordinating & assigning tasks to
self and others
(Emergent)
Online learners distributing, sequencing
tasks and sub-tasks to others/self for future
Are you picking this [task] up next?
Original Codes & Source
Description Example
M1- Checking for understanding
Online learners seeking verication of un-
derstanding of tasks, events or concepts
from other online learners.
Are we sure that everything has been cited
M2 - Identifying problems or issues
Online learners drawing attention of other
online learners to difculties that may
interfere with completion of tasks or other
I am unable to open the quiz. Does
anyone else have this problem?
M3 - Noting completion of tasks
((Azevedo. Et al., 2004)
Comments between online learners
that indicate tasks/activities have been
nished to support attaining a goal.
I did some research and then typed up the
employer section.
M4 - Evaluating the quality of an end
product, its content or its constituent
parts (Azevedo. Et al., 2004)
Statements between online learners that
judge the accuracy, comprehensiveness,
relevance or other aspects of an end
product or its components
I fully agree with this concept. This is
denitely an area we should build upon.
Strategy Use
Codes & Source Description Example
S1 - Seeking, offering, providing help or
information (Curtis & Lawson, 2001)
Online learners requesting, offering, or
providing assistance or information related
to learning materials, activities, tasks or
If you need any assistance, please let me
know what I can do to help you out.
S2 - Seeking, offering, providing
clarication (Emergent)
Seeking, offering, providing clarication
between online learners
Just as a point of clarication, are you
seeking a critique of the specic information
contained in the readings or
S3 - Advocating effort (Curtis & Lawson,
Encouraging/urging online learners to
contribute to the online group
Has everyone contributed their pieces?
Inter-rater Reliability
Table 1. Course Discussions Inter-rater Reliability
Instructor A Instructor B
Kappa CR Kappa CR
Pre Post Pre Post Pre Post Pre Post
M1 0.6479 0.9283 0.9574 0.9894 0.7460 1.0000 0.9203 1.0000
M3 0.2255 1.0000 0.9468 0.5405 0.9309 0.8923 0.9848
M5 0.3767 1.0000 0.7270 1.0000
Allen, M., Bourhis, J., Burrell, N., & Mabry, E. (2002). Comparing student satisfaction with distance education to traditional
classrooms in higher education: A meta-analysis. American Journal of Distance Education, 16(2), 8397.
Allen, E. & Seaman, J. (2010). Learning on Demand: Online Education in the United States 2009. Needham, MA: Sloan Con-
sortium.
Anderson, T., Rourke, L., Garrison, D. R., & Archer, W. (2001). Assessing teaching presence in a computer conferencing envi-
ronment. Journal of Asynchronous Learning Networks ,5 (2).
Bernard, R.M., Abrami, P.C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., Wallet, P. A., Fiset, M., & Huang, B. (2004). How
does distance education compare to classroom instruction? A meta-analysis of the empirical literature. Review of Educa-
tional Research, 74(3), 379-439.
Bixler, B. A. (2008). The effects of scaffolding students problem-solving process via question prompts on problem solving and
intrinsic motivation in an online learning environment.PhD diss., The Pennsylvania State University, State College, Penn.
Chang, M. M. (2007). Enhancing Web-based language learning through self-monitoring. Journal of Computer Assisted Learn-
ing 23 (3):187196.
Chung, S., M.-J. Chung, and C. Severance (1999). Design of support tools and knowledge building in a virtual universi-
ty course: Effect of reection and self-explanation prompts. Paper presented at the WebNet 99 World Conference on the
WWW and Internet Proceedings, Honolulu, Hawaii. (ERIC Document Reproduction Service No. ED448706).
Cook, D. A., D. M. Dupras, W. G. Thompson, and V. S. Pankratz (2005). Web-based learning in residents continuity clinics: A
randomized, controlled trial. Academic Medicine 80 (1):9097.
Crippen, K. J., and Earl, B.L (2007). The impact of Web-based worked examples and self-explanation on performance, problem
solving, and self-efcacy. Computers & Education 49 (3):809821.
Dewey, J. (1933). How we think. A restatement of the relation of reective thinking to the educative process. Boston, MA: D. C.
Heath.
Feinstein, A. R., &Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical
Epidemiology., 43(6), 543-549.
Greeno, J. (2006). Theoretical and practical advances through research on learning. In J. Green, G. Camilli, P. Elmore (Eds.),
Handbook of complementary methods on education research (pp. 795-822). Mahwah, NJ: Lawrence Erlbaum Associates.
Kuhn, T. (1977). The essential tension: Selected studies in scientic tradition and change. Chicago: University of Chicago Press.
Lipmann, M. (2003). Thinking in Education. New York: Cambridge University Press.
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002). Content Analysis in Mass Communication: Assessment and Reporting
of Intercoder Reliability. Human Communication Research, 28(4), 587-604. doi: 10.1111/j.1468-2958.2002.tb00826.x
Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2009). Evaluation of evidence-based practices in online learning: A
meta-analysis and review of online learning studies. Washington, D.C.: U.S. Department of Education, Ofce of Planning,
Evaluation, and Policy Development.
Nelson, B. C. (2007). Exploring the use of individualized, reective guidance in an educational multi-user virtual environment.
Journal of Science Education and Technology 16 (1):8397.
Parsad, B., & Lewis, L. (2008). Distance education at degree-granting postsecondary institutions: 200607 (NCES 2009044).
National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Washington, DC.
Pierce, C. S. (1955). The xation of belief. In J. Buchler (Ed.), Philosophical writings of Pierce. New York: Dover, pp-5-22.
Rex, Steadman, & Graciano (2006). Researching the complexity of classroom interaction, in Green, J, Camili, G, & Elmore, P.
(Eds), Handbook of Complementary Methods in Educational Research. Lawrence Erlbaum: Mahwah, NJ.
Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (1999). Assessing social presence in asynchronous text-based computer
conferencing. Journal of Distance Education, 14, 5170.
Saito, H., and K. Miwa (2007). Construction of a learning environment supporting learners reection: A case of information
seeking on the Web, Computers & Education 49 (2), 214229.
Shea, P., Li, C., Swan, K., & Pickett, A. (2005). Developing learning community in online asynchronous college courses: The
role of teaching presence. Journal of Asynchronous Learning Networks, 9(4), 59-82.
Shen, P. D., Lee, T.H., and Tsai, C. W. (2007). Applying Web-enabled problem-based learning and self-regulated learning to
enhance computing skills of Taiwans vocational students: A quasi-experimental study of a short-term module. Electronic
Journal of e-Learning,5 (2), 147156.
Sim, J., & Wright, C. C. (2005). The kappa statistic in reliability studies: use, interpretation, and sample size requirements.
Physical Therapy, 85(3), 257-268.
Staklis, S. (2010). Prole of undergraduate students 2007-2008. US Department of Education, National Center for Education
Statistics, Institute of Education Sciences, Washington, DC. Downloaded from http://nces.ed.gov/pubs2010/2010205.pdf
Tallent-Runnels, M. K.; Thomas, J. A.; Lan, W. Y.; Cooper, S.; Ahern, T. C.; Shaw, S. M, et. al. (2006). Teaching courses online:
A review of the research. Review of Educational Research, 76(1), 93-135.
U. S. Department of Education. National Center for Education Statistics. (2008). Distance Education at Degree-Granting Post-
secondary Institutions: 200607 (NCES2009044). National Center for Education Statistics, Institute of Education Sci-
ences, Washington, DC. Retrieved July 2, from http://nces.ed.gov/pubs2009/2009044.pdf.
Wang, K. H., Wang, T.H., Wang, W. L., & Huang, S. C. (2006). Learning styles and formative assessment strategy: Enhancing
student achievement in Web-based learning. Journal of Computer Assisted Learning 22 (3), 207217.
Zhao, Y., Lei, J., Yan, B., Lai, C., & Tan, H.S., (2005). What makes the difference? A practical analysis of research on the effec-
tiveness of distance education. Teachers College Record, 107(8), 18361884.
Zimmerman, B. J. (1986). Development of self-regulated learning: What are the key sub-processes? Contemporary Educational
Psychology, 16, 307-313.
Acknowledgement: Support for this study was provided in part by the SUNY Learning Network, Ofce of the Provost, State
University of New York
Literacy in Multimedia Environments: Preliminary Findings 215
Literacy in Multimedia Environments: Preliminary Findings
BARBARA OBYRNE, DIANA BAILEY, AND STACEY MURRELL
Web-authoring tools that have proliferated in the last ten years provide a unique window on literacy processes, mak-
ing visible the multiple ways in which students construct meaning in electronic environments. Multiplying and morphing
web-authoring tools such as CD-ROM, DVD, podcasting, wikis, blogs, and video conferencing tools, are functionally
multimodal, relying upon the signs and codes of audio, visual, and interactive tools to engage participants in the orches-
tration of meaning (Brusilovsky, 2001). Multimedia tools encourage users to shift between and among different interac-
tive tools embedded with the codes or grammars of sound, voice, movement, images, and movie, and promote and
extend a broader, richer denition of literacy (Kalantis, Cope & Cloonan, 2010).
Seventy years ago, Suzanne Langer (1942), building on the work of Ernest Cassirer (1955) sketched out the symbol-
ic systems of music, art, and literature and wrote convincingly of the languages of music, art, sculpture, literature, and
dance. The advent of multimedia, interactive technologies has shifted and expanded exponentially the terrain of com-
posing and lent credence to the premise of the New London Group (1996) that pedagogy of multiliteracy needs to replace
a narrow, traditional language-based framework. These new forms of expression and communication are not unlike the
symbolic languages Langer and Cassirer explored. Within the multimodal electronic world, the sign systems of sound,
image, and text blend in complex ways to create meaning. The new literacies, immediately make sense in a digital,
multimodal environment. They create exciting opportunities for teachers and students to actively construct knowledge.
For their full potential to be actualized, however, they need to be contextualized in a solid theoretical framework and ex-
amined by careful research.
This research examines the meaning of multimedia tools for literacy learning in the P-12 classroom. Reading, writ-
ing speaking and listening, the traditional literacies, dominate the reading and language arts curriculum of public schools.
Current models of literacy theory endorse active knowledge construction in these four domains. The presence and ease
of access to multimedia web-authoring tools support a broader conceptualization of literacy and provide a timely oppor-
tunity to examine the impact of use of these tools on knowledge-construction. The focus of this investigation is to identi-
fy and evaluate the impact of the active, composing qualities associated with multimodal web-authoring tools on literacy
in grade two and a grade eleven classrooms.
THEORETICAL CONTEXT
Certain denitions and suppositions underpin this investigation. By literacy, is meant all of the processes associ-
ated with using sign systems to construct meaning. These include, but are not limited to the traditional forms of reading,
speaking, writing, listening, and viewing. In addition, viewing, a fth component, has established a place in contempo-
rary literacy curricula. Multimodal refers to the different media through which meaning is conveyed and constructed.
This concept has its own literature in learning theory and preceded the appearance of multimedia tools (Biggs &Collis,
1991; Kress& Van Leewen, 2001; OByrne 2009); Multimedia refers to digital artifacts embedded into electronic sites
that include sound, image, hyperlinks, and video. It also refers to the avalanche of web-authoring tools that have prolifer-
ated in the last decade, empowering users to construct their own products. Multimedia tools are functionally multimodal,
incorporating the sign systems of image, sound, and print to convey meaning. These tools are ubiquitous in schools and
in the general society, available to anyone with access to a computer and the Internet. Computers that enabled an indi-
vidual to self-publish a book fteen years ago now provide the tools to make movies, create sound recordings, design
clothes and construct identities in virtual realities.
216 O'Byrne, Bailey, and Murrell
Two epistemological premises underlie this investigation. The rst is an acceptance of constructivism as a valid ex-
planation of knowledge construction. The premise of constructivism is that individuals construct theories and knowledge
from interactions with the environment while searching for solutions to problems. Conceptualizing knowledge as a verb,
the action of problem-solving, contrasted with the epistemology of the positivist, rational view that situated knowledge as
an objective reality outside the individual that could be validated by deductive reasoning and empirical theory. The para-
digm shift associated with constructivist epistemology had consequences for theories of thinking, learning, and instruc-
tion. Applications to instructional practice in public schools took much longer, and to date, do not inform the intentional
pedagogy of even a majority of educators.
Two main branches emerged in constructivist learning theories in the early decades of the twentieth-century. The
rst placed the locus of knowledge-making in the individual consciousness. Piaget (1929) supported this cognitive theory
that emphasized knowledge-construction as a mental activity that took place within the mind of an individual. The sec-
ond model of constructivism situated knowledge-construction within a broader social context. Vygotsky (1997) took the
position that all knowledge is mediated through a social or cultural context, and awarded a special place to language in
knowledge-construction. The gap between the learner as discoverer of knowledge (constructivism) and the learner as
learning from and with others (social constructivists) is a theoretical issue that cannot be ignored
On the surface, composing with digital tools lends weight to a constructivist, or discovery model with individual
learner manipulating composing tools to construct meaning. However, Crook (1994, 2001) described the more subtle
sense in which Vygotsky dened social as interactions encoded into semiotics, into sign systems that are present even
when no one is in the room (Crook, 1994, p. 27). Computers make use of these sign systems as intrapersonal tools that
can transform mental functions and extend the social dimension of computer-assisted instruction far beyond interpersonal
interactions (Ravencroft, 2001). This more rened sense of language or literacies as sign systems embedded into the in-
trapersonal tools of the computer suggests a way to narrow the gap between Vygotsky and Piaget and the social construc-
tivist position informs this investigation.
The second premise of this investigation is that the traditional denition of literacy has been supplanted by a broader,
more inclusive notion that acknowledges the distinct sign systems of still and moving images, sound, speech, and ges-
ture. It further assumes that electronic media have their own sign systems and that these change rapidly (The New Lon-
don Group, 1996; Kalantzis, Cope, & Cloonan, 2010). These electronic sign systems, synonymous with the new litera-
cies are embedded into multimodal web-authoring tools. Yet, most classrooms continue to operate with the boundaries
of traditional denitions of literacy. Research is needed that explains how the composing tools, processes, and outcomes
associated with digital literacies articulate with the tools, processes and outcomes of traditional literacies. Literacy edu-
cators will want to understand the gains in knowledge-making associated with the digital literacies, and will need evi-
dence that the gains warrant the investment of precious curriculum resources, especially instructional time.
This research starts with questions that emerge from the observations of many educators who cite the enthusiasm
of students for digital projects. What is it about technology that captivates learners? How does engagement with the
technology change the process of learning in the literacy curriculum? How does it inuence learning outcomes? These
large questions are often pushed to the side by teachers and administrators who prefer to see technology as an additive,
the newest tool, rather than the handmaiden of fundamental changes in the nature, goals, and context of literacy learning.
Literacy educators cannot easily sidestep these questions because the nature of the tools raises fundamental questions
about what counts as literacy. There is a need for systematic analysis of the impact of technology on literacy and learn-
ing and for evaluation of related theories of knowledge-making. Empirical studies are scant. The continually morphing
of digital tools and the difculty of conducting experimental studies in the social communities of schools are two reasons
for this dearth of scientic studies. The deictic nature of the technology argues for case study as a valid methodology
from which to begin research investigations. This investigation adds to a growing body of research that investigates the
role of the new technologies in teaching and learning in P-12 literacy and language arts classrooms. It represents a pre-
liminary effort to map out the impact of multimedia web-authoring tools on composing processes of learners P-12 and to
formulate ideas on the qualities and features of these electronic forms
RESEARCH CONTEXT
Graduate students from two literacy courses received training in the use of web-authoring tools to facilitate and ex-
tend literacy learning in the classroom. The teachers created and eld-tested literacy projects using one or more of the
web authoring tools in their classrooms in spring/fall 2010. Each investigator constructed and delivered a project using
one or more of the web authoring tools within a class of students. The central web-authoring tools used in this project
include VoiceThread, Blogs, and Digital Story Telling. Students process and products were gathered and analyzed to de-
termine the impact of the web-authoring tools on composing processes. Where possible and relevant, comparisons were
made with projects accomplished with traditional tools and classroom resources. .
PARTICIPANTS/SETTINGS
Second grade students from one urban grade school and grade eleven students one high school in Southern West
Virginia participated in the project. The second grade teacher operated as a teacher/investigator in this study, as did the
instructor of the Advanced Placement English and Composition class.
The urban grade school in Southern West Virginia had 220 students of whom 84% qualied for free and reduced
lunch. The student population of the school is one-third African American and two-thirds white. Eighteen grade two stu-
dents participated in this study.
The urban high school had a student population of 1600 of whom 48% were eligible for free and reduced lunch. The
fty-four participants came from two grade eleven Advanced Placement Language and Composition classes. The ethnic
diversity of the participants, one Asian, seven African Americans, and forty-six white students reected the diversity of
the school population. This sample consisted of thirty-ve females and nineteen males. The population consisted of stu-
dents preparing to enter university.
The literacy and technology projects were directly integrated into the language arts block. All participants completed
the technology activities as part of the literacy curricula. Each intervention took place within a three week period.
Students in second grade wrote a narrative story using the process writing model suggested in their basal. They se-
lected a topic, brainstormed, worked with a graphic organizer and composed their story. During the next cycle of writing,
students constructed images on a sheet of paper divided into at least three compartments that helped tell their story. The
teacher scanned and loaded the storyboards into VoiceThread. Students were encouraged to narrate the story into Voice-
Thread, using their draft and the pictures to guide and expand upon their ideas. Students were free to work on the project
at home or in school. The objective was to examine the impact on switching between and among the different medium
of text, image, and sound on the composing processes and products associated with the creation of student narratives.
Langer (1942) and Cassirer (1955) identied the features of different symbolic forms of language, music and art, multi-
modal learning suggests the hypothesis that composing with the multimodal sign systems embedded into web-authoring
tools enriches knowledge construction. The objective was to evaluate how the composing processes associated with mul-
timedia tools lend themselves to this blending and bending of ideas to create new meanings.
The grade eleven Advanced Placement English and Composition students were challenged to create a movie on a
socially relevant theme using MOVIE MAKER. The different components of the assignment included, description of
topics, peer responses, reections, a literature review, a short movie and an expository paper. Students posted each of
these elements on a blog. The movies were presented for peer and instructor review in a live class session. Students had
the option of uploading their movie to the blog. The goal was to determine the ways in which the multimedia tools in-
uenced the process and products associated with the complex learning expected in a multigenre project (Herrington &
Hodgson, 2009).
Because schools are more like communities than they are like clinical research sites, the preferred method of inves-
tigation leans heavily towards naturalistic methodologies. In addition, the newness of the technologies supports the need
for qualitative studies to locate key features and themes that can be followed up with more generalized investigations.
Case studies and ethnographic studies are suited to the population and the intervention under investigation. The current
investigation used a case study methodology to make sense of how literacy learning in grades two and eleven changes in
multimodal, multimedia environments.
The second grade teacher/researcher assembled writing samples before and after the intervention. She gathered and
analyzed artifacts from composing done with on the VoiceThread site. She analyzed the samples with a coding analysis
of student work. The grade eleven teacher/researcher gathered data from several points in a multigenre project, all of
which were posted on a blog. These data were analyzed using a combination of descriptive statistics and coding analysis
of data from student responses, essays, and products for emergent themes. The data illustrated the composing processes
of learners as they interacted and constructed meaning with the multimedia tools associated with the project.
PRELIMINARY DATA ANALYSIS
It is clear that many learners, P-2, struggle with the physical acts associated with writing and the need to focus at-
tention on the shapes of letters and words. A central challenge for P-2 learners is the lack of renement of ne motor
skills necessary for either efcient manual scribing or keyboard stroking. The physical acts associated with constructing
texts interfere with composing processes associated with story-making. It is obvious to any kindergarten teacher that stu-
dents at this age are better at telling a story than they are at writing one. Emergent literacy educators address this issue
by such practices as sharing the pen (Clay, 2001).
Web-authoring tools such as VoiceThread provide emergent learners with ways of composing texts that circumvent
the limitations imposed by more slowly developing motor skills by providing tools that draw upon the more sophisticated
oral language skills of emergent learners. Preliminary ndings from the spring 2010 pilot of this research demonstrated
convincingly that the stories of kindergarten students composed with Voice Threads were richer in content, voice, and
depth than those composed using traditional tools and method used for writing projects. The fall 2010 iteration of this
project situated this investigation in a grade two class to see if students approaching the end of the emergent literacy
stage demonstrated similar trends in their stories told on VoiceThread. Early indications, based on initial research analy-
sis, are that the multi-modal dimensions inherent on the VoiceThread site enabled students to leapfrog over scripting ob-
stacles. More signicantly, the projects drew out learning gains traced to the multimodal methods and multimedia tools.
As students moved from idea to pictures to composing stories about the pictures, the quality and range of their compos-
ing processes became obviously richer. Additional analysis of data will identify and elaborate on these qualities.
Data from two classes of grade eleven Advanced Placement English came from ve sources: blogs features, mov-
ies, student peer responses on the blog, student essays and student reections. Analysis of blog features convincingly
demonstrated that students operated with a richer, more interactive concept of blogs. The assignment required that stu-
dents set up a blog on a specic site and make postings related to their movie project. The blogs became one of the most
interesting revelations of the projects and the current analysis of data focuses on what was learned from analysis of blog
postings. Students developed and personalized the blogs in a myriad of ways. Table 1. provides data on the instances of
additional features incorporated into the blogs.
Table 1. Blog Add-Ons
Ask me information and photos 33%
Images 7%
Audio 10%
Video 22%
Product ads 27%
Opinion surveys (general) 4%
Opinion surveys( topic-related) 6%
Interactive games 9%
These numbers from the pilot study, by themselves, provide a clear window into student understanding and expecta-
tions about blogs. Ongoing analysis of these data research will further rene the analysis of blog features.
Analysis of the number and times of blog postings reected similar engagement of students with this form of inter-
action (Table 2.). It is evident that students had specic windows during which they posted entries and went beyond the
stated requirements of the project.
Table 2. Blog Postings
More than required responses to peer topics 27%
More than required responses to peer papers 11%
More than required words in essay 44%
Optional posting of video 45%
Postings between 9 p.m. and 1 a.m. 75%
These data demonstrate a signicant trend to go beyond the minimum expected by the assignment and together,
are indicative of high learner engagement. The two last statistics are especially intriguing. The videos were presented in
class, the expectation of the assignment but nearly half the students posted the video to the blog and began discussions
about these videos that extended beyond the classroom project. It is a safe conclusion that students posted the videos
with the understanding that they would be seen, suggesting they had self-identied not only an authentic purpose for
the video but an authentic electronic place. The timing of the postings makes sense for those familiar with the habits of
adolescents. They make a compelling case that electronic tools provide a time and place more in sync with the rhythms
of adolescents and also demonstrate that even when students have class time to complete these types of projects, much
of their best work is done at other times. It is anticipated that greater renement of the analysis of data will demonstrate
specic ways in which participants use blogs as tools to make meaning and to communicate.
Preliminary data analysis of student movies composed with Movie Maker supported and extended the ndings from
the analysis of blog features and content. The same methods used to analyze the pilot group will be used to analyze the
movies from the second group. Much like the ndings from the analysis of blog add-ons, it is clear that signicant per-
centages of students added features to the movie to enhance the meaning. Students showed an ease and condence in
composing electronic products and sophisticated understandings of features that enhanced the product and features that
detracted from the presentation.
CONCLUSIONS AND IMPLICATIONS
This preliminary study on the impact of multimodal, web-authoring tools on the composing processes associated
with reading and writing provided some tentative conclusions. In the grade two class, the use of this web-authoring tool,
in conjunction with uploaded images, enabled them to focus on their composing processes instead of the skills associated
with constructing letters and words and moving from composing with images to composing with texts showed positive
results from expression translating from one medium to another. High school students used Digital Story Telling tools
to construct original understanding of the features of genre and to create powerful documentaries on contemporary and
historical themes. An unexpected discovery of this project was that participant use of web-authoring tools demonstrated
a richer, more expansive understanding of literacy that extended beyond the narrow boundaries of printed text. The data
from this intervention will be more rigorously analyzed to identify specic forms of communication and knowledge-
expression associated with the blogs.
In the multimedia environment, students are bending and blending forms of literacy to express unique meanings.
They are composing in new ways, drawing from a wide range of media, and fusing these into products that express
meaning. While multimodal learning is, in and of itself, not new, electronic technologies, especially multimedia web-
authoring tools make this mode of composing available to anyone with a reasonably current computer. Literacy in a mul-
timedia environment is multifaceted, as the New London Group (1996) theorized. The multimodal nature of web-author-
ing tools both expressed the new literacies and provided a unique set of tools to enable educators and learners to enhance
and expand meaning construction. Digital composing shifts the discussion from technology as tool to technology as a set
of literacy forms used by learners to construct meaning.
The current investigation creates a platform for a more rened analysis of case study data that will investigate the
impact of the web authoring projects on different groups of students such as those identied in this study. Specic quali-
tative and empirical studies are needed examine the ways in which e-technologies are changing the fabric of literacy and
knowledge construction for specic groups of learners. As a knowledge base expands, hypotheses and questions will
emerge that explore not only the how of technology but also the impact of technology on pedagogy and learning and in
particular, of the effect on knowledge-composing processes encouraged by the forms of multimedia tools.
Biggs, J.B., and Collis, K.F. (1991). Multimodal learning and the quality of intelligent behavior. In Helga A.H. Rowe (Ed.), In-
telligence -- reconceptualization and measurement. Hillsdale, New Jersey: Lawrence Erlbaum Associates.
Brusilovsky, P. (2001) Adaptive hypermedia. User Modeling and User Adapted Interaction, Ten Year Anniversary Issue, 11
(1/2), 87-110 http://www.sis.pitt.edu/~peterb/papers/brusilovsky-umuai-2001.pdf
Cassirer, E. (1955) The philosophy of symbolic forms Vol. 1. Language. Trans. R. Manheim. New Haven: Yale University Press.
Clay, M. (2001). Change over time in childrens literacy development. Heinemann Press.
Crook, C. (2001). The social character of knowing and learning: implications of cultural psychology for educational technology.
Journal of Information Technology for Teacher Education, 10(1&2), 19-36.
Crook, C. (1994). Computers and the collaborative experience of learning. London: Routledge.
Hasselbring, T. S. (2010). Reading prociency, the struggling reader, and the role of technology. In E. A. Baker, The new litera-
cies: multiple perspectives on research and practice (pp. 23-40). New York: The Guilford Press.
Herrington, A., Hodgson, K., & Moran, C. (2009.). Teaching the new writing: Technology change and assessment in the 21st-
century classroom. New York: Columbia University.
Kalantzis, M., Cope, B., & Cloonan, A. (2010). A multiliteracies perspective on the new literacies. In E. A. Baker (Ed.) The new
literacies multiple perspectives on research and practice (pp.61-87). New York: The Guilford Press.
Kress, G., & Van Leewen, T. (2001). Multi modal discourse: The modes and media of contemporary communication. London:
Oxford University.
Langer. S. (1942). Philosophy in a new key: A study of the symbolism of reason, rite, and art. Boston: Harvard University Press.
OByrne, B. (2009). Knowing more than words can say: Using multimodal assessment tools to excavate and construct know-
ledge about wolves. International Journal of Science Education. 31(4), 523-539.
New London Group (1996). A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review 66(1), 60-92.
Piaget, J. (1929). The childs conception of the world. London: Routledge & Kegan Paul.
Ravenscroft, A. (2001). Designing E-learning interactions in the 21
century: revisiting and rethinking the role of theory. Euro-
pean Journal of Education, 36:2, 133-156.
Vygotsky, L. (1997). Language and thought. Trans. Ed. A. Kozulin. Cambridge, MA: The MIT Press.
Student-Centered Library Instruction: An Assessment of Online Graduate Students Information Literacy Skills and Needs 221
Student-Centered Library Instruction: An Assessment of Online Graduate Students
Information Literacy Skills and Needs
SWAPNA KUMAR AND MARILYN N. OCHOA
Enrollment in online courses at U.S. institutions of higher education has steadily increased in recent years. Over 5.6
million students took an online course in the fall 2009 semester, compared to 4.5 million in the fall 2008 semester (Al-
len & Seaman, 2009; 2010). The increase in online programs in higher education has been accompanied by initiatives
and discussion about the quality of distance education and support for online students (Sloan Consortium, 2004; Meyer,
2002). The Sloan Consortium identies access, learning effectiveness, faculty satisfaction, student satisfaction, and scale
as the ve pillars of effective practice in online education (http://sloanconsortium.org/effective). Students perception of
connectedness to an institution can inuence student completion of online courses and student satisfaction in an online
program (Cain & Lockee, 2002; Tait & Mills, 2003). It is thus important to provide online students with various other
forms of support at the institutional, program, and course level if they are to have a quality online learning experience
and feel connected to an institution and to their program (Distance Education & Training Council, 2010).
Students who take courses online often do so for reasons of convenience and access, and because they are working
professionals who want to study part-time, live in remote locations, or cannot commute to campuses for several reasons.
In addition to technical and administrative support, they need support in accessing library materials and databases that
will help them complete assignments and activities in their program. Lack of access or inability to access library re-
sources can lead to frustration and ultimately to higher drop-out rates in courses (Lee, 2000). Recognizing this need, the
Association of College and Research Libraries (ACRL) has worked to create standards that ensure all members of a uni-
versity community, whether online or on-campus, have equal access to library resources and services, and has suggested
creating a program of library user instruction designed to instill independent and effective information literacy skills
while specically meeting the learner-support needs of the distance learning community (ACRL, 2004, p. 5).
Library support for off-campus learners is not a new phenomenon. Librarians have provided instruction to remote
and off-campus learners in the past by traveling to remote sites and more recently, using new technologies for synchro-
nous and asynchronous (e.g. web pages, online pathnders, CMS links and web tutorials) instruction. Library instruction
in higher education has often been provided to students either as a stand-alone class about a specic topic or database
that is not connected to a course or particular audience, or as a one shot session e.g. an overview or orientation within
a course which is sometimes geared toward a specic assignment. Notwithstanding the value of stand-alone and one-
shot library instruction, library instruction within a course and tied to assignments within the course has been found to be
effective (Adams, 1998; Allegri, 1985; Beile, 2003; Bordonaro & Richardson, 2004; Hall, 2008; Dugan, 2008; Kohl &
Wilson, 1986).
LIBRARY SUPPORT IN ONLINE PROGRAMS AT THE COLLEGE OF EDUCATION
Online graduate offerings at the University of Floridas College of Education have grown rapidly in the past few
years, with several online programs that lead to degrees in Master of Education, Educational Specialist, and Doctorate
in Education being offered. Many assignments and activities in these programs require students to search, nd, and re-
view scholarly materials and peer-reviewed publications; to write annotated bibliographies, book reviews and literature
reviews; to support their online opinions on blogs or discussion posts with research and seminal theories in their eld;
and to conduct research that connects theory to practice and builds on existing research in the eld. Students in these
programs are encouraged to use library resources and services and to attend library orientations if they are on-campus,
but few students are able to do so due to busy schedules and perceived barriers of distance. Nevertheless, information
literacy and library skills are essential to students success in the online education programs.
222 Kumar and Ochoa
The university system of nine libraries on campus has vast holdings and hundreds of databases available for
the University community. Librarians consistently work to provide easy access and discoverability for new users to the
system. The Education librarians in particular redesigned the Education Library website to improve access for online
students. With the increase of online programs and reliance on using online resources, this website overhaul included
development of tutorials and pages describing services for campus and remote users. Dropdown menus enable users
to reach education related databases and other web sources. Subject oriented and program related guides were also
carefully constructed to help new students get started with navigating information resources if they could not speak
with a librarian directly. The Education Library website, a portal aimed to simplify nding relevant content from the
vast resources available, made the existing resources and services of the Education Library more visible to a user. Work
with the College of Educations Distance Learning Ofce to place modules within the course management system used
(Moodle) also raised awareness of the resources and services available; this collaborative effort extends the ways for
students to learn about and use library content.
Analyzing the Needs of Online Graduate Students
In an attempt to provide support to students within online education courses, academic librarians at the Col-
lege of Education conducted course-specic projects intended to provide library instruction to online graduate students
in Educational Technology. Student reports of increased condence using library resources during these projects led to
collaboration between the education librarian and a faculty member in an online doctoral program to create structured,
systematic instruction for online Ed.D. students in 2010. Two new doctoral cohorts in Curriculum and Teaching, begin-
ning in the summer of 2010, enrolled students who were employed full-time in educational institutions, did not live near
the university campus, and were returning to graduate school after a gap of 5+ years. Given the advances in information
technology and library services since their last university experiences, it was considered important to provide them with
information literacy instruction to help them succeed in their graduate studies. The librarian and faculty member focused
on the content and form of the online instruction to be provided to entering online doctoral students about accessing,
using, and citing resources. Adult learners in a study that explored K-12 educators perception of their ability to access
library resources had been reported to often suffer from library anxiety (Collins &Veal, 2004, p.12), therefore the goal
of library instruction was to reduce such anxiety in the incoming students, all K-12 educators, by tailoring instruction
to their needs. Based on Nicholas and Tomeos (2005) call for including a self-assessment for information or research
skills in online learning, it was decided that students would be sent a survey intended to measure their current abilities to
access information online, search and retrieve different types of scholarly publications, manage library resources and use
them in their writing, and cite appropriately. The results of the survey could then be used to design instruction that met
the needs of that specic group of students, instead of offering them general or one-shot library instruction that might be
useful to them.
A needs assessment or a needs analysis is an integral component of the instructional design process that helps
to determine the needs of the learners and context and identify gaps between the current state of knowledge or skills and
the goals of instruction (Dick & Carey, 2005). In adopting a learner-centered approach to designing online instruction in
the online graduate program, it was important to not only estimate the prior knowledge and skills of the online graduate
students but also the ways in which they would prefer to learn online. In any library instruction context, an analysis
of students prior knowledge and current skills with respect to accessing library resources can provide librarians with
valuable insight into student needs and they can structure library instruction accordingly. Instructors and administrators
in the program can use the information to provide students with adequate support that will help them succeed in course
activities and assignments, thereby leading to increased student satisfaction and learning.
A survey was designed to evaluate the library and research needs of incoming Curriculum and Teaching docto-
ral students enrolled in the new online programs. The 22-question survey was constructed to give the librarians informa-
tion on the students experience and confdence with using, evaluating and citing resources. This needs assessment was
hosted in Survey Monkey software, distributed by email and also linked within the learning management system before
students participated in the orientation to the doctoral program. The orientation included a library instruction session
held during their required weeklong visit to campus. Some questions were used to identify specifc resources that the
librarian would incorporate into that face to face session. Of the 26 incoming doctoral students in one online Ed.D. pro-
gram in Curriculum and Teaching that were administered this survey, 57% (n=15) responded to the survey in June 2010.
Initial items in the survey were structured to gauge students familiarity with the university and university
resources. Fifty-four percent of students who responded had taken courses at the University of Florida, and 31% of those
students had taken those courses online. Seventy-seven percent of students who responded reported that they had not
received any guidance for conducting library research at institutions of higher education, 15% had previously used online
library instruction tutorials, and 75% of them were most likely to consult a student peer if they had questions about re-
search. Only 8% of them had previously consulted librarians when they needed to do research.
As part of the self-assessment, students were asked to rate their experience and their condence with using
various library resources. When asked about their prior experience with using library resources, 15% of students rated
themselves as experienced, 39% as somewhat experienced, and 46% of students rated themselves as not experienced (Ta-
ble 1). In response to a more specic question about library catalogs and databases, 25% of students rated themselves as
experienced and somewhat experienced while 50% stated that they were not experienced (Table 2). None of the respon-
dents to the survey rated themselves as very experienced on either question.
Table 1: Experience using library resources
Your prior experience using library resources: Response
Count (n=13)
Very experienced 0.0% 0
Experienced 15.4% 2
Somewhat Experienced 38.5% 5
Not Experienced 46.2% 6
Table 2: Experience using catalogs and databases
Your prior experience using the library catalog and article
databases:
Students responses to their awareness and use of databases that are useful to graduate students in education are
detailed in Tables 3 & 4. The databases listed in these questions are those that librarians consider essential knowledge
for doctoral students to nd peer-reviewed and other types of scholarly publications. The incoming doctoral students
were most aware of ERIC (92%), Education Full Text (85%), Google (85%), Google Scholar (54%) and JSTOR (46%) in
that order. They most frequently used Education Full Text (75%) and ERIC (58%) in their library searches.
Table 3: Awareness of databases
Are you aware of the following databases? Response
Percent (Yes)
Response Count
(n=13)
ERIC 92.3% 12
Education Full Text 84.6% 11
Education Index Retro 0.0% 0
Library Catalog 30.8% 4
Social Science Citation Index 0.0% 0
Academic Search Premier 15.4% 2
Dissertations and Theses 0.0% 0
JSTOR 46.2% 6
PsycInfo 7.7% 1
Web of Science 0.0% 0
WorldCat 7.7% 1
Google 84.6% 11
GoogleScholar 53.8% 7
Other (please specify) 0.0% 0
Table 4: Database use
Which of these databases do you use most frequently? Response
ERIC 58.3% 7
Education Full Text 75.0% 9
Library Catalog 8.3% 1
Google 33.3% 4
In addition to assessing incoming students knowledge and use of databases in the eld of Education, it was con-
sidered important to assess their affective perceptions (e.g. condence, anxiety) that inuence their success in nding
resources in an online course (Colin & Veal, 2004). Responding to questions in this section, only 8% of students agreed
that they were very condent or condent using library catalog and article databases (Table 5), and 31% and 50% rated
their anxiety regarding the literature search process as high or moderate. Seventeen percent rated themselves as success-
ful and 58% as somewhat successful in nding relevant literature during a search (Table 6).
Table 5: Condence using library databases
Rate your condence with using the library catalog and
article databases:
Very condent 7.7% 1
Condent 7.7% 1
Somewhat condent 38.5% 5
Not condent 46.2% 6
Table 6: Success with nding relevant literature
Rate your success with nding literature that is relevant Response
Very Successful 8.3% 1
Successful 16.7% 2
Somewhat Successful 58.3% 7
Not Successful 16.7% 2
Following searches, the ability to evaluate and use relevant literature in writing assignments in graduate online
courses was considered important in the online Ed.D. program. Questions about students ability to evaluate and cite on-
line resources appropriately indicated that students rated their ability to evaluate the quality of resources pretty high 8%
as excellent, 23% as good, and 69% as fair (Table 8). Fifty percent of students rated their ability to cite resources appro-
priately as excellent or good (Table 9).
Table 8: Ability to evaluate quality of resources
Rate your ability to evaluate the quality of resources discovered Response
Excellent 7.7% 1
Good 23.1% 3
Fair 69.2% 9
Poor 0.0% 0
Table 9: Ability to cite resources appropriately
Rate your ability to cite your resources appropriately Response
Poor 16.7% 2
The process of needs assessment described in this paper enabled the education librarians to identify areas in which
students needed assistance and instruction as they began their graduate studies. Instruction was designed in the following
ways:
Since few students reported prior contact with librarians and with searching the library catalog and article data-
bases, the library orientation session at the beginning of the program stressed the importance of contacting the
education librarians if students had questions.
Time during the introductory library session was devoted to accessing resources from off-campus locations.
Despite students reported familiarity accessing resources, this was done to ensure that any changes in modes
of access were familiar to students. Signifcant time was spent on showing students how to search the library
catalog and article databases, the different types of databases available, and the citing of resources.
Additional asynchronous training resources were developed and posted on the Education Library website - these
resources included tutorials in the form of screen-casts, videos recordings, and detailed step-by-step Adobe
Acrobat fles.
Students will be administered another survey at the end of their frst year in the program to assess whether the li-
brary instruction provided was valuable to them in their search, retrieval, and citation process, and also whether it raised
their confdence and reduced their anxiety with library searches.
Adult and professional online students who turn to institutions of higher education for online graduate degrees after
several years of employment are often confronted with new information systems or digital resources that did not exist
during their prior educational experiences. While they may be expert at searching, retrieving, and evaluating online in-
formation in a search engine, they might not be aware of available library catalogs, databases, and digital resources that
contain scholarly publications. Online students use of such resources can be crucial to their success, to student retention
in online programs, and to student satisfaction. Notwithstanding the value of providing one-shot or general instruction
that could benet all graduate students, library instruction and support that is specic to a particular group and targeted at
their specic eld of study could be benecial and more effective with professional graduate students. An assessment of
graduate students online library skills, condence and anxiety can help librarians design appropriate instruction for those
students and to impart information literacy skills that align with skills that students need to be successful in an online
Adams, B. K. (1988). Library lecture seminars and workshops in course integrated instruction. The Southeastern Librarian, 38,
Allegri, F. (1985). Course integrated instruction: Metamorphosis for the twenty-rst century. Medical Reference Services Quar-
terly, 4, 47-66.
Allen, I. & Seaman, J. (2009). Learning on Demand: Online Education in the United States. Babson Park MA:
Babson College Survey Research Group. Retrieved February 9, 2010, from www.sloanconsortium.org/publications/survey/pdf/
learningondemand.pdf
Allen, I. E., & Seaman, J. (2010). Class Differences: Online Education in the United States, 2010. Retrieved December 15, 2010
from http://sloanconsortium.org/publications/survey/class_differences and http://sloanconsortium.org/sites/default/les/
class_differences.pdf
Association of College and Research Libraries. (2004). Guidelines for distance learning library services. http://www.ala.org/ala/
mgrps/divs/acrl/standards/informationliteracycompetency.cfm
Beile, P. M. (2003). Effectiveness of course-integrated and repeated library instruction on library skills of education students.
Journal of Educational Media & Library Sciences, 40(3), 271-277.
Bordonaro, K., & Richardson, G. (2004). Scaffolding and reection in course-integrated library instruction. The Journal of Aca-
demic Librarianship, 30(5), 391-401.
Badke, W. (2009). Ramping up the one-shot. Online (Weston, Conn.), 33(2), 47-49.
Brown, J. S., Collins, A. & Duguid, S. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32-
Cain, D. L., & Lockee, B. (2002). Student support services at a distance: Are institutions meeting the needs of distance learners?
EDRS ED468729.
Collins, A., Brown, J. S. and Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts ofreading, writing, and math-
ematics. In L.B. Resnick (Ed.), Knowing, learning and instruction: Essaysin honour of Robert Glaser, Hillsdale, NJ: LEA,
Collins, K. M. T., & Veal, R. E. (2004). Off-campus adult learners levels of library anxiety as apredictor of attitudes toward the
internet. Library & Information Science Research, 26, 5-14.
Dick, W., & Carey, L. (2005). The Systematic Design of Instruction. Allyn & Bacon.
Dugan, M. (2008). Embedded librarians in an agecon class: Transcending the traditional. Journal of Agricultural & Food Infor-
mation, 9(4), 301-309. doi:10.1080/10496500802480342
Edwards, M., Kumar, S. & Ochoa, M. (2010). Assessing the value of embedded librarians in an online graduate educational
technology course. Public Services Quarterly, 6(2/3), 1-21. Doi: 10.1080/15228959.2010.497447
Hall, R. A. (2008). The embedded librarian in a freshman speech class. College & Research Libraries News, 69(1), 28-30. Re-
trieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=28719340&site=ehost-live
Kohl, D. F., & Wilson, L. A. (1986). Effectiveness of course-integrated bibliographic instruction in improving coursework. RQ,
26, 206-211.
Lee, C-Y. (2000). Student motivation in the online environment. Journal of Educational Media & Library Sciences, 37(1), 367-
Lillard, L. L., Norwood, S., Wise, K., Brooks, J., & Kitts, R. (2009). Embedded librarians: MLS students as apprentice librar-
ians in online courses. Journal of Library Administration, 49(1), 11-22. doi:10.1080/01930820802310544
Liu, B. (2007). Web data mining: Exploring hyperlinks, contents, and usage data. Berlin: Springer.
Meyer, K. A. (2002). Quality in Distance Education: Focus on On-line Learning. In A.J. Kezar (Ed.), ASHE-ERIC Higher Edu-
cation Report (Vol. 29, pp. 1-134). Jossey Bass.
Nicholas, M., & Tomeo, M. (Summer 2005). Can you hear me now? Communicating library services to distance education stu-
dents and faculty. Online Journal of Distance Learning Administration, 8(2).
Sloan Consortium (2004). Entering the mainstream: The quality and extent of online education in the United States, 2003 and
2004. Needham and Wellesley, MA: The Sloan Consortium.
Stein, L. L., & Lamb, J. M. (1998). Not just another BI: Faculty-librarian collaboration to guide students through the research
process. Research Strategies, 16(1), 29-39.
Tait, A., & Mills, R. (Eds.). (2003). Rethinking learner support in distance education: Change and continuityin an international
context. London: Routledge Falmer.
Paths to Exemplary Online Teaching: A Look at Teacher Roles, Competencies and Exemplary Online Teaching 229
Paths to Exemplary Online Teaching: A Look at Teacher Roles, Competencies and
Exemplary Online Teaching
EVRIM BARAN, ANA-PAULA CORREIA, AND ANN THOMPSON
Having its roots in distance education, online learning showed a substantial growth in the last decade particularly
with the emergence of Information and Communication Technologies (ICT) (Allen & Seaman, 2007). The increasing
availability, accessibility and popularity of online technologies created educational opportunities with rich resources, dif-
ferent media, asynchronous and synchronous communication modes, and social and networked technologies. We are now
living in an era in which social, mobile and Internet technologies have diffused into many areas of personal, social and
professional environments creating a demand for anytime anywhere learning. Hence, higher education institutions have
embraced online education as an opportunity for meeting the needs of diverse group of students. The Chronicle of Higher
Education (2009) research report predicts that students will demand more online courses in the near future (Van Der Werf
& Sabatier, 2009).
The growing interest in online education has particularly challenged higher education institutions to rethink their
cultural, academic, organizational and pedagogical structures to adopt to the new culture of teaching and learning in an
ever changing environment (Howell, Saba, Lindsay, & Williams, 2004). Along the same line, the drastic increase on the
number of online programs and course offerings have caused changes on the role of the faculty and the nature of teaching
with an increasing number of faculty and support staff required for online teaching (Bennett & Lockyer, 2004). Perhaps
most importantly, faculty, being at the center of this increasing demand and pressure to teach online, have been chal-
lenged to rethink their underlying assumptions about teaching and learning, the roles they have taken as educators (Wi-
esenberg & Stacey, 2008) and teaching competencies they develop.
The purpose of this paper is to review the literature on online teaching with a focus on the roles and competencies
exposed by successful online teachers and exemplary online teaching. This paper later continues with a critique on the
research on online teacher competencies and roles. Following questions guided the literature review:
1. How is the changing role of the teacher portrayed in the literature?
2. How is exemplary online teaching defned in the literature?
3. What is lacking in the current teaching approaches to support and cultivate exemplary online teaching?
It should be noted that for the purpose of this review, online teacher is dened as the faculty who teaches online
in higher education intuitions. Moreover, online teaching is dened as teaching that is conducted completely online.
Face-to-face teaching is dened as teaching that is conducted completely in a physical classroom.
This review began as a broad search for research on online teaching using the various combinations of these key-
words: exemplary teaching, exemplary online teaching, successful online teaching, best practices, online pedagogies,
and online teacher roles. Overall the articles included in this review consist of both qualitative and quantitative research
that were located through online databases (ERIC, EBSCO), online journals, Social Science Citation Index, and Internet
search engines such as Google Scholar. The research conducted in the online journals such as British Journal of Educa-
tional Technology, Journal of Distance Education, Journal of Asynchronous Learning Networks, The Internet and Higher
Education, Computers and Education, and the American Journal of Distance Education. These journals were selected
230 Baran, Correia & Thompson
because of their interest in online teaching and best practices. To locate the review studies, Review of Educational Re-
search journal was examined focusing on the reviews on online teaching published in the last 10 years. In addition to
the search in online databases and journals, three other essential search methods were used extensively: search on the
printed books, search on the references of the key articles, and search for the articles of key researchers in the eld. Due
to the lack of consistency and agreement on the terminology used in the online teaching literature, the references of the
related publications were extensively used. The following sections summarize the major results of the literature review
conducted.
Changing Teaching Practices and Changing Online Teaching Roles
Online environment changes the fundamental interaction between the teacher-student, student-content, and teacher-
content in the learning environment. For instance, the hierarchy in the online environment is attened with a more dis-
tributed power and control, creating a need for providing learner- centered environments (Schrum & Hong, 2002) where
teachers are expected to adopt more facilitative approaches (Salmon, 2004; Smith, 2005). The research focusing on the
change on the student-teacher relationship in online environments suggests a new role for teacher: guide on the side.
The teacher, rather than becoming the center of the interaction or the source of the information, is now expected to de-
sign and facilitate a student-oriented approach to learning (Laat, Lally, Lipponen, & Simons, 2007). Taking the role of
facilitator, coach, or mentor, teacher is expected to design, organize and schedule the activities where learners take the
responsibility of their own learning by coordinating and regulating their learning activities (Anderson, Rourke, Garrison,
& Archer, 2001; Berge, 2009).
In an online environment teachers are not the sole performers on the stage, but share the roles and responsibilities
with the other actors. The roles required for online teaching are distributed with number of specialized professionals and
teams i.e. instructional support personnel, instructional designers, teaching assistants, technology experts, media artists,
online program coordinators, and even other faculty (Howell, et al., 2004; Miller, 2001; Paulson, 2002). Teachers col-
laborate with others to a much greater extend in order to receive support and help during the planning, design and the
delivery of online courses (Bennett & Lockyer, 2004). Teachers take different roles and responsibilities at different levels
such as facilitating, teaching, organizing, mentoring, and providing the content etc. This requires to the adoption of a
more industrialized educational fashion (Howell, et al., 2004).
Over the years, different online teacher roles have been described in the literature. Researchers have created taxono-
mies and models specifying the roles that online teachers need to perform while teaching online. Although the studies
addressing these roles show variety in terms of context and their denition of online teacher, commonalities can be found
in the roles that teachers assume as they teach online. One of the early models that describes teachers role in a virtual
environment is the Instructors Roles Model (Berge, 1995) that identies teachers functions under four different catego-
ries: Pedagogical, social, managerial, and technical. Berge (1995) denes these roles within the online discussion context
where pedagogical role is taken to facilitate the learning in the discussions, social role to encourage and promote working
together, managerial role to organize and design the logistics of the discussions, and technical role to provide a transpar-
ent technology environment to the learners (Berge, 2009; Berge & Collins, 2000). These roles were suggested at the time
when teachers were moving to online environments, where the main activities were designed around online discussions.
However, due the rise of virtual worlds and other learning environments, Berge (2009) calls for a change on the roles that
focus more on informal, collaborative, reective learning, with user-generated content (p. 412).
The role of the teacher in online environments changes, keeping the core principles of traditional teaching environ-
ments such as having clear expectations, critical discourse and diagnosing misconceptions (Garrison & Anderson, 2003).
The contextual dynamics make the teaching process complex and multifaceted, however online learning, by nature,
change the way the responsibilities are performed. Building on the previous research that suggests roles for online teach-
ers, Anderson, et al. (2001) suggest three categories for online teachers roles: Instructional design and Organization, fa-
cilitating discourse, and direct instruction. Accordingly, Anderson, et al. (2001) dene teaching presence as the design,
facilitation, and direct instruction of cognitive and social processes for the purpose of realizing personally meaningful
and educationally worthwhile learning outcomes (p. 5). Teachers roles here are dened within the community of inquiry
framework, one of the most cited models in the computer medicated communication (CMC) research (Garrison, Ander-
son, & Archer, 2003). The model consists of three elements: cognitive presence, social presence and teaching presence.
Research found that teaching presence is a signicant predictor of students perceived learning, satisfaction, and sense of
community (Gorsky & Blau, 2009; LaPointe & Gunawardena, 2004; Russo & Benson, 2005). Although teaching pres-
ence is considered as what the teacher performs to create a community of inquiry with social and cognitive presence, all
participants within the online learning environment can contribute to teaching presence by sharing the responsibilities.
As a result of an effort to dene online teaching roles and competencies, a group of international researchers and
practitioners describe main roles that online teachers take: Process facilitator, advisor-counselor, assessor, researcher,
content facilitator, technologist, designer and manager-administrator (Goodyear, Salmon, Spector, Steeples, & Tickner,
2001). By building on these roles, Aydin (2005) provided additional roles such as content expert, instructional designer
and material producer in the context of online mentors in a large open university in Turkey. Other roles are also identi-
ed in the literature (Aydin, 2005; Egan & Akdere, 2005; Varvel, 2007; Williams, 2003). Bawane and Spector (2009)
synthesized the research on online instructor roles and competences and derived eight comprehensive roles: Professional,
pedagogical, social, evaluator, administrator, technologist, advisor/counselor, and researcher. The investigation included
asking experts about the priority and criticality of these instructor roles. The results of their survey research indicate
pedagogical role as the highest ranked role followed by professional, evaluator, social facilitator, technologist, advisor,
administrator, and researcher roles (Bawane & Spector, 2009). However, in another context, Aydin (2005) nds out that
the assessor role received the highest ranking due to the importance given to the assessment within that online teaching
context. The variety in terms of the roles that teachers take in an online environment, demonstrates the context-dependent
nature of teaching, which have a direct inuence on the prioritization of the roles within an online environment.
What Makes Online Teaching Exemplary?
Researchers used various methods to identify exemplary teachers. For instance Schrum and Hong (2002) described
the dimensions that play signicant role in students success in online learning through a document analysis and later
presented these dimensions to expert online instructors. The participants in this study were selected from online teachers
who were known to have designed and taught online courses for considerable amounts of time and rate themselves as
successful online educators (Schrum & Hong, 2002, p. 60). A snowball method which included asking experienced in-
structors to nominate others around the world was used. Fourteen instructors participated in this study.
Carrying this notion of selecting exemplary teachers thinking and decision-making processes, Lewis and Abdul-Ha-
mid (2006) explored the effective online teaching practices with 30 online teachers identied as exemplars. The research-
ers focused on the examples and actual strategies that teachers described while teaching online. The results indicated
the examples of strategies under four broad categories: (1) fostering interaction, (2) providing feedback, (3) facilitat-
ing learning, and (4) maintaining enthusiasm and organization. Similarly, in order to capture what effective pedagogi-
cal practices that award winning experienced teachers perceive, Bailey and Card (2009) conducted a phenomenological
study. The analysis of the interviews indicated eight effective pedagogical practices: Fostering relationships, engagement,
timeliness, communication, organization, technology, exibility, and high expectations. Using a two phase study, Schrum
and Hong (2002) rst identied the dimensions of successful online learners and then asked experienced online instruc-
tors to review these dimensions in terms of their signicance and suggest strategies for student success in the online
learning environments. The results indicated seven dimensions for student success: access to tools, technology experi-
ence, learning preferences, study habits and skills; goals or purposes; lifestyle factors; and personal traits and charac-
teristics. In order to nurture these, the instructors suggested strategies such as establishing sense of community at the
beginning of the class, providing ongoing interaction and support to the students, creating collaborative learning activi-
ties, promoting participation and contribution in the discussions for knowledge building, providing up-to date resources,
responding to student questions, sharing course responsibilities with the students, and reducing technology difculties
with a technologically minimalist approach. The results indicate that these experienced instructors followed constructiv-
ist strategies with a focus on collaborative and active learning rather than traditional lecture models.
While the research analyzed looks at how exemplary teachers describe their own practices, another line of research
focuses on student evaluations as a way to collect information on the effective online teaching strategies. Bangert (2006)
conducted a study to develop and validate a student evaluation instrument on the teaching effectiveness by using Chick-
ering and Gamson (1991)s Seven Principles of Effective Teaching. The results of the exploratory factor analysis con-
ducted on the student responses on the survey indicated four factors: student-faculty interaction, active learning, time on
task, and cooperation among students. Similarly, in Young (2006)s study, students provided denitions of effective on-
line teaching: adopting to student needs providing meaningful examples, motivating students to do their best, facilitating
the course effectively, delivering a valuable course, communicating effectively, and showing concern for student learning.
Emerging Issues in Online Teaching Research
Existing literature has described a variety of roles that teachers take while teaching online, however the compli-
cated and context-dependent nature of online teaching makes generalizability difcult (Baran & Correia, 2009). Silcock
(1993)s critique on teaching effectiveness literature is a valid statement for the online teaching literature: Reading
about educational effectiveness can be dike opening a book and nding lengthy indices but very little text: there are
dozens of indicators about what good teaching and good schools might be, but no teacher strategies or lesson designs
deduces from these to enrich our existing assumptions (p. 13). The following paragraphs describe some of the issues
encountered while conducting this literarture review.
Diversity in the Interpretation of the Terms
The analysis of the research in online teacher competencies and roles shows that there is a diversity in the interpre-
tation of the terms teacher functions and competencies (Anderson, Rourke, Garrison, & Archer, 2001; Eraut, 1998;
Goodyear, Spector, Steeples, & Tickner, 2001, Salmon, 2001). Spector (2007) denes a competent individual as being
adequately qualied to perform a reasonably well-dened task (p. 5). The concept of competency is used in two differ-
ent ways: (1) competency as a personal skill linked to behavior efciency, (2) competency as a strategic behavior that is
adjusted and adopted to the demands from the context (Guasch, et al., 2010). According to Guasch, et al. (2010), the lat-
ter approach seems better suited to refer teachers competencies, whose exercise must unavoidably respond to the pecu-
liarities of their given educative context (p. 200). Therefore it is the teachers, and their expectations, that determine how
the competency would be shaped within their specic professional contexts.
Lack of focus on Authenticity and Contextual Dynamics Within Online Teaching Environments
Although the studies on effective teaching, teachers roles, competencies and functions present us a set of guidelines
to follow, they do not reect the authenticity of the online teaching contexts which are particularly affected by various
factors surrounding them. Furthermore, these studies were based on the systematization of teachers experience and
were clearly circumscribed to specic university environments (Guasch, et al., 2010, p. 199). The unique characteris-
tics that dene online education suggest that the outcomes of the online learning environment are dependent on different
conditions (Bangert, 2004). The lack of agreement on the denition of the roles and competencies as mentioned above
can also be explained by the fact that these roles diverge depending on the organizational, cultural and social dynamics
of each online teaching and learning context. Although the research on competencies contributes to the literature on the
clarication of teachers roles in the online environments, there is still limited evidence on how teachers actually perform
these roles within their peculiar teaching contexts and how these diverse teacher roles inform the development of teacher
support and professional development.
Competencies and Roles can be Outdated and Oversimplied
Changes in the nature of learning environments calls for an understanding of the new culture by adopting the exist-
ing roles or creating new ones according to the distinctiveness of each online context (Guasch, et al., 2010). Most of the
research on teacher role and competencies were built around the online environments such as online discussion platforms
within the classroom management systems. However, as new online environments emerge, teachers roles within these en-
vironments also change. Berge (2009) for instance, considers traditional teacher roles within the virtual worlds context as
different than traditional learning management systems. The denition of teachers functions and roles specic to virtual
teaching-learning environments is more recent. It results from observing and analyzing the experiences of those teachers
who in their daily practice respond to the challenge of teaching in virtual environments (Guasch, et al., 2010, p. 200).
Limited Focus on How to Attain Excellence in Online Teaching
The ndings from empirical research presents methods for prescriptions for effective online teaching, yet many on-
line teachers nd the intersection between the prescribed best online teaching principles and their practical applica-
tion is, at best, an elusive and confusing process (Lewis & Abdul-Hamid, 2006, p. 95). Moreover while the studies on
the competencies and roles add richness and depth to the online teaching literature, the question of how faculty give life
to the successful practices is still under researched. Teaching involves many complex and somewhat ill-structured ac-
tivities; as a consequence, establishing reliable and relevant performance measures for teaching competence is difcult
(Spector, 2007, p. 6). Similarly, teachers role in the online environment is dynamic and multi-dimensional. There is a
need for empirical research to create a framework, which is not solely based on the base lines, but more on the attainment
of excellence in online teaching.
Limited Connections Between Content, Online Pedagogies and Technologies
The literature on online teacher roles has limited emphasis on how content plays an important role in the creating of
successful teaching practices in connection with the pedagogical methods and the particular technologies. Teachers need
to go beyond the mere competence with the online technologies, but develop an understanding of the complex relation-
ships between technologies, pedagogies, content, learners and the context (Koehler & Mishra, 2005). While the roles are
dened in the literature for effective online teaching, many act as general guidelines and recipes. For instance facilitation
role is one of the most cited role in the literature. However how a teacher facilitates student learning in a particular con-
tent area, with the pedagogical methods and appropriate online technologies is a missing link between the research on
online teacher roles and the practical realities of teachers who teach online.
Another problem related to the existing literature is treating technology as a separate entity such as the role of tech-
nologist in Goodyear, et al. (2001), technological role in Berge (1995), and technical skills in using the features of the
software in Salmon (2004)s studies. However Koehler, Mishra, and Yahya (2007) argues that technology cannot be
treated as a knowledge base unrelated and separate from knowledge about teaching tasks and contexts it is not only
about what technology can do, but also, and perhaps more importantly, what technology can do for them as teachers (p.
742). Therefore, researchers, particularly in the area of technology integration argue for a more integrated and multidi-
mensional teacher knowledge, called Technological Pedagogical Content Knowledge (TPACK).
The purpose of this paper was to present the literature review on online teaching particularly focusing on online
teaching roles and competencies, and exemplary online teaching. The limitations of the existing research on online teach-
er roles, competencies, and exemplary online teaching were also identied. A review of the roles and teacher functions
in the online teaching literature indicates that there is a diversity in the interpretations of the terms teacher functions
and competencies and there is a lack of focus on authenticity and contextual dynamics within online teaching envi-
ronments. Although the research on the competencies and roles guide practitioners as they design and develop teacher
support and professional development programs, they become inadequate in terms of keeping up with the rapid changes
in the online education environments. For instance, teachers are now challenged to reconsider their current teaching ap-
proaches in the light of an increasing popularity of social and networked Internet technologies among highly active and
participative Internet users. New media technologies provided opportunities for users to create, share and express them-
selves in different media channels. Internet users can now appropriate and repurpose the online content as a mode of
creative and social expression. We are now witnessing a large technological shift in the history that is being driven by
the users who want to be engaged, active, social and networked than ever before. However, how much of this shift is re-
ected in our current research on teacher roles and support is still questionable.
Additionally, this review indicates that there is a limited focus on the methods how teachers attain excellence in their
online teaching. The empirical research on online teaching is limited in terms of guiding teachers to develop strategies
for self-improvement (Spector, 2007). Reection is a key factor for improving teachers practice. Schn (1983) asserts
that engaging in the process of continuous learning is the reection in action and it is an essential feature of professional
Both ordinary people and professional practitioners often think about what they are doing, sometimes even while
doing it. Simulated by surprise, they turn thought back on action and on the knowing which is implicit in action.
They may ask themselves, for example. What features do I notice when I recognize this thing? What are the criteria
by which I make this judgment? What procedures am I enacting when I perform this skill? How am I framing the
problem that I try to solve? (p. 50)
It is through this reective in action that practitioners can surface the tacit understandings that build on the special-
ized and repetitive practice and deal situations of uncertainty, instability, uniqueness, and value conict (p. 50). Schn
(1988) also talks about reection on action as a retrospective practice. Reection in action (during the experience) and
reection on action (after the experience) have become two essential elements of professional training and development
in different disciplines.
There are two ways that reective practice can be used in the online teaching context: reection in action and re-
ective in action (Schn, 1988). Analyzing the online teaching practices while teaching and reecting on teaching retro-
spectively when it is over are two essential processes that may offer online teaching faculty ways to rethink and improve
their online teaching practices. Al-Mahmood and McLoughlin (2004) indicates reective practice is about the process of
teaching rather than about a simple evaluation of teaching, questioning why we do something rather than how, and most
important of all, learning by this process (p. 38).
Although, reective practice is suggested to be an important factor for teachers development, the methods that sup-
port reective practice is not studied in the literature. Perhaps most importantly, the research literature on teacher roles,
support and development has had limited emphasis on the connections between how online technologies, pedagogies and
content interact in particular teaching environments and how teachers can design effective online learning environments
considering the peculiarities of their own teaching environments. Transition to online teaching requires more than grasp-
ing the knowledge of various technologies, but adopting new pedagogies while understanding the affordances and the
limitations of the technologies for teaching particular subjects. Therefore reective practice needs to be included in the
faculty professional development as an integral part of faculty members work. As universities offer more online courses
and require more faculty to teach online, faculty members need more assistance to reect back on themselves as teach-
ers? to be able to see a picture of who they are and what they do in the online teaching environment.
Allen, I., & Seaman, J. (2007). Online nation: Five years of growth in online learning. Needham, MA: Sloan-C Consortium. Re-
trieved from http://wiseplus.exp.sis.pitt.edu:8080/dspace/handle/123456789/47
Anderson, T., Rourke, L., Garrison, D., & Archer, W. (2001). Assessing teaching presence in a computer conferencing context.
Journal of Asynchronous Learning Networks, 5 (2), 1-17.
Aydin, C. (2005). Turkish mentors perception of roles, competencies and resources for online teaching. Turkish Online Journal
of Distance Education (TOJDE), 6 (3). Retrieved from http://tojde.anadolu.edu.tr/tojde19/articles/caydin.htm
Bailey, C., & Card, K. (2009). Effective pedagogical practices for online teaching: Perception of experienced instructors. The
Internet and Higher Education, 12 (3-4), 152-155.
Bangert, A. (2004). The seven principles of good practice: A framework for evaluating on-line teaching. The Internet and High-
er Education, 7 (3), 217-232.
Bangert, A. (2006). The development of an instrument for assessing online teaching effectiveness. Journal of Educational Com-
puting Research, 35 (3), 227-244.
Baran, E., & Correia, A.-P. (2009). Student-Led Facilitation Strategies in Online Discussions. Distance Education, 30 (3), 339-
Bawane, J., & Spector, J. (2009). Prioritization of online instructor roles: implications for competency-based teacher education
programs. Distance Education, 30 (3), 383-397.
Bennett, S., & Lockyer, L. (2004). Becoming an online teacher: Adapting to a changed environment for teaching and learning in
higher education. Educational Media International, 41 (3), 231-248.
Berge, Z. (1995). The role of the online instructor/facilitator. Educational Technology, 35 (1), 22-30.
Berge, Z. (2009). Changing Instructors Roles in Virtual Worlds. Quarterly Review of Distance Education, 9 (4), 407-415.
Berge, Z., & Collins, M. (2000). Perceptions of E-Moderators about their roles and functions in moderating electronic mailing
lists. Distance Education, 21 (1), 81-100.
Chickering, A., & Gamson, Z. (1991). Appendix A: Seven principles for good practice in undergraduate education. New direc-
tions for teaching and learning, 1991 (47), 63-69.
Egan, T., & Akdere, M. (2005). Clarifying distance education roles and competencies: Exploring similarities and differences be-
tween professional and student-practitioner perspectives. American Journal of Distance Education, 19 (2), 87-103.
Garrison, D., & Anderson, T. (2003). E-learning in the 21st century: A framework for research and practice. New York, NY:
Routledge Falmer.
Garrison, D. R., Anderson, T. & Archer, W. (2003). A theory of critical inquiry in online distance education. In M. Moore and G.
Anderson (Eds.), Handbook of distance education. (pp.113-127). New York: Erlbaum.
Goodyear, P., Salmon, G., Spector, J., Steeples, C., & Tickner, S. (2001). Competences for online teaching: A special report.
Educational Technology Research and Development, 49 (1), 65-72.
Gorsky, P., & Blau, I. (2009). Online teaching effectiveness: A tale of two instructors. The International Review of Research in
Open and Distance Learning, 10(3). Retrieved from http://www.irrodl.org/index.php/irrodl/article/viewArticle/712
Guasch, T., Alvarez, I., & Espasa, A. (2010). University teacher competencies in a virtual teaching/learning environment: Analy-
sis of a teacher training experience. Teaching and Teacher Education, 26 (2), 199-206.
Howell, S., Saba, F., Lindsay, N., & Williams, P. (2004). Seven strategies for enabling faculty success in distance education. The
Internet and Higher Education, 7 (1), 33-49.
Koehler, M. J., & Mishra, P. (2005). What Happens when Teachers Design Educational Technology? The Development of Tech-
nological Pedagogical Content Knowledge. Journal of Educational Computing Research, 32 (2), 131-152.
Koehler, M. J., Mishra, P., & Yahya, K. (2007). Tracing the Development of Teacher Knowledge in a Design Seminar: Integrat-
ing Content, Pedagogy and Technology. Computers & Education, 49 (3), 740-762.
Laat, M., Lally, V., Lipponen, L., & Simons, R. (2007). Online teaching in networked learning communities: A multi-method
approach to studying the role of the teacher. Instructional Science, 35 (3), 257-286.
LaPointe, D., & Gunawardena, C. (2004). Developing, testing and rening of a model to understand the relationship between
peer interaction and learning outcomes in computer-mediated conferencing. Distance Education, 25 (1), 83-106.
Lewis, C., & Abdul-Hamid, H. (2006). Implementing effective online teaching practices: Voices of exemplary faculty. Innova-
tive Higher Education, 31 (2), 83-98.
Miller, G. (2001). General education and distance education: Two channels in the new mainstream. The Journal of General Edu-
cation, 50 (4), 314-322.
Paulson, K. (2002). Reconguring faculty roles for virtual settings. Journal of Higher Education, 73 (1), 123-140.
Russo, T., & Benson, S. (2005). Learning with invisible others: Perceptions of online presence and their relationship to cogni-
tive and affective learning. Educational Technology & Society, 8 (1), 54-62.
Salmon, G. (2004). E-moderating: The key to teaching and learning online. London: RoutledgeFalmer.
Schn, D. (1983). The reective practitioner: How professionals think in action. London: Temple Smith.
Schrum, L., & Hong, S. (2002). Dimensions and strategies for online success: Voices from experienced educators. Journal of
Asynchronous Learning Networks, 6 (1), 57-67.
Silcock, P. (1993). Can we teach effective teaching? Educational Review, 45 (1), 13-19.
Smith, T. (2005). Fifty-one competencies for online instruction. The Journal of Educators Online, 2 (2), 1-18.
Spector, J.M. (2007). Competencies for online teachers. In J. M. Spector (Ed.), Finding your online voice: Stories told by expe-
rienced online educators (pp. 1-18). New Jersey, NJ: Lawrence Erlbaim Associates, Publishers.
Van Der Werf, M., & Sabatier, G. (2009). The College of 2020: Students: Chronicle Research Services.
Varvel, V. (2007). Master online teacher competencies. Online Journal of Distance Learning Administration, 10 (1). Retrieved
December 16, 2010, from http://www.westga.edu/~distance/ojdla/spring101/varvel101.htm
Wiesenberg, F., & Stacey, E. (2008). Teaching Philosophy: Moving from Face-to-Face to Online Classrooms. Canadian Journal
of University Continuing Education, 34(1), 63-69.
Williams, P. (2003). Roles and competencies for distance education programs in higher education institutions. American Journal
of Distance Education, 17 (1), 45-57.
Young, S. (2006). Student views of effective online teaching in higher education. American Journal of Distance Education, 20
(2), 65-77.
Virtual Gender Roles: Is Gender a Better Predictor of Internet Use than Sex? 237
Virtual Gender Roles: Is Gender a Better Predictor of Internet Use than Sex?
AMY TROMBLEY
The Internet has taken the world by storm since it emerged in 1982 (DiMaggio, Hargittai, Neuman & Robinson,
2001). Its hard to believe that just 30 years ago, the Internet was an abstract concept developed for scientic, and ulti-
mately Cold War inspired, military communication (Abbate, 1999). Although the Internet did not immediately take off
with the general public, the availability of graphical interfaces and participation of commercial interests initiated a boom
in both demand and access (Abbate). DiMaggio, Hargittai, Neuman & Robinson dene the Internet as the electronic
network of networks that links people and information through computers and other digital devices allowing person-to-
person communication and information retrieval (p.307). Although this is an effective operational denition, it fails to
encompass the complexities and implications of Internet use. Generation Y (also known as the Millennials) grew up im-
mersed in technology. This vast group of teenagers and young adults cant imagine a world without immediate satisfac-
tion of their every need with the click of a button (NAS, 2006). They hold the entitled perception that if they do not get
what they want from one source, they can immediately go to another (p. 3) and if they are not satised with the answers
they nd they know that there are other options out there (NAS, p. 3). Yet the Millennials arent the only generation
online. With 74.7% of the total US population online (Internet World Stats, 2009) the implications of such an Internet
saturation are undeniable. The question is no longer does the Internet have an effect, it is now, in what ways does the
Internet have an effect, and can these effects be measured? Researchers across disciplines seek to discover common
themes, extraordinary patterns, causal relationships, and ultimately any evidence that will give insight into how the Inter-
net is changing the world. Although the types of studies performed in regards to the Internet are as diverse as those who
perform the research, a recurrent theme across domains is that the Internet tends to complement rather than displace
existing media and patterns of behavior (DiMaggio et al., p. 307). Conversely, research has identied inconsistencies
in patterns of behavior in cyberspace that are so pervasive they have been labeled the disinhibition effect (Suler, 2004).
These incongruent ndings inspire the question; do Internet users stay true to their authentic personalities when commu-
nicating online? More specically are social roles redened or simply superuous in cyberspace?
SEX ROLES IN CYBERSPACE
Historically, the Internet was characterized as a male-dominated forum, yet a variety of more current studies paint
a cloudy picture of the current dynamics of cyberspace. By the end of 2000 women actually outnumbered men in the
United States online population (Gorski, 2001) and it has been hypothesized that social networking sites and online net-
work TV shows are leading this growth (eMarketer, 2008). This data seems to support the idea that the gender-gap is
closing rapidly (Weiser, 2000), yet others claim measures of frequency of use continue to point to a male-dominated
Internet (Hargittai & Shafer, 2006). Still others argue that equity in access does not equate to equity in opportunity (Gor-
ski; Rose & Jones, 2009). These scholars express that to simply assume that easy access to technology equals digital
equity is both shortsighted and simply wrong (Rose & Jones, 2009, p. 725). On the other end of the spectrum, there are
an equal number of scholars who have found that there is no gender difference in overall use of the Internet, rather that
men and women use the Internet equally but for different reasons (Jackson, Ervin, Gardner, & Schmitt, 2001; Fallows,
2005; Jackson, Kolenic III, Fitzgerald, Harold, & VonEye, 2008). Jackson, Ervin, Gardner, & Schmitt (2001) predicted
and found that females use email more than males and males use the Web more than females. Furthermore, Jackson et
al. (2008) concluded that women seek human connections on the Internet in the same way that they seek to maximize
connectedness in face-to-face interactions. Men on the other hand, value the breadth of research opportunities available
and prefer to take advantage of the tools offered on the Internet to add convenience to shopping, banking, and research
(Fallows, 2005). Additional ndings suggest that women focus on quality (i.e., depth) over quantity when using the In-
ternet for research.
238 Trombley
Sex Roles in Online Education
Similar ndings in regards to sex differences have been recorded when evaluating the domain of online education. A
great deal of literature regarding the dynamics of online classrooms suggest that collaborative learning activities tap into
a womans need for more connectional forms of learning (Cole, 2009, p. 314). For example, it was observed that fe-
male learners tend to more openly disclose personal information about themselves through casual dialogue and pictures,
yet women are mixed when it comes to preferences for learning through collaborative learning activities (Cole). Gender
differences in the culture of learning have also been observed.
Male Learning Culture Female Learning Culture
Tendency to dominant behavior in educational
Tendency to cooperative behavior and
More frequent take-over of monitoring discourse Willingness to be responsible for ongoing
Longer and more frequent contributions in
Shorter contributions in discourse
More often involved in the development of
enforcement strategies
Open for proposals of other people and for
cooperative work in general
Desire to impress others and competitive
Willingness to discuss topics, supportive of
Development and maintenance of competitive
Care for a just distribution of learning tasks;
preference for group work
Derichs-Kunstmann & Auszra (1999. P.184) cited in Kuhlen & Michels (2006)
In search of empirical support for these differences, Kuhlen & Michels (2006) found gender-specic differences in
learning styles, group behavior and group success and in degrees of (information and communication) activity during
which the differences depended on the social and cultural environment in through which the e-learning was delivered
(p. 2599).
Sex versus Gender
With the abundance of seemingly contradictory ndings about Internet use, what is the next step for those scholars
interested in discovering the truth about virtual behavior? An underlying glitch in the study of Internet use is the fact that
scholars use the terms sex and gender interchangeably. Scholars, including Jackson et al. (2001, 2008), focus on the in-
dependent variable of sex (i.e., male versus female), however it has been found that gender is a better determinant of be-
havior than sex (Bem, 1994; Spence, Hemlrich, & Stapp, as cited in Pearson & Davilla, 2001). Whether its because the
term sex seems to be taboo in the American culture or because it has become culturally acceptable to use the term gender
when referring to the study of sex-roles, the concepts of sex and gender are very different and should be studied concur-
rently rather than in substitution for one another.
During the 1970s psychologists inuenced a reconceptualization of gender (e.g., Bem as cited in Pearson & Davilla,
2001). In contrast to many gender theorists, Sandra Bem preferred to categorize individuals based on their internalized
societal standards for masculine and feminine behaviors rather than a product of their sex (as cited in Peason & Davilla).
She looked at masculinity and femininity as 2 independent dimensions rather than as opposite ends of a spectrum. Gen-
der is what one does rather than what a person is (e.g., Canary and Dindia, 1998; Pearson & Davilla). Individuals can
exhibit a high or low degree of both the masculine and feminine dimensions leading to that persons demonstration of
either a masculine, feminine or androgynous personality.
High Masculine + Low Feminine Masculine Personality
High Feminine + Low Masculine Feminine Personality
High Masculine + High Feminine Androgynous Personality
Low Masculine + Low Feminine Undifferentiated Personality
Bem Sex Role Inventory
The Bem Sex Role Inventory (BSRI) was generated to provide a measure of the degrees of masculinity, feminin-
ity, and androgyny according to Sandra Bems gender schema theory (Bem, 1981). According to Bem, humans exhibit
a comprehensive predisposition to comprehend and manage behaviors based on sex-linked associations that construct
gender schema. Thus sex-typing is an outcome of social inuences that set standards for gender dichotomy and gender-
related behaviors (Bem, 1981). The BSRI is a survey that is comprised of 60 personality traits (20 masculine, 20 femi-
nine, and 20 neutral) and is self-assessed. Those who wish to measure their degrees of gender rate themselves on a scale
of 1 to 7 depending on the level they believe they exhibit for each trait. The BSRI is scored by adding together all 20
masculine scores and all 20 feminine scores, and then divides each of these totals by 20.
According to Canary and Dindia (1998), sex differences have little effect on personality and social behavior (p.
23). Thus, because Internet usage is a specic human behavior, research suggests that gender should play a more es-
sential role than sex in predicting that behavior. Unfortunately, when studying technology, specically the Internet most
scholars have evaluated the variable of sex in regard to how an individual interacts with the Internet. Human behaviors
and social norms are transferred to the Internet in complex and divergent ways. Restrictions imposed by reality are less
rigid on the Internet creating a sense of virtuality (virtual reality) that may or may not follow culturally imposed so-
cial norms. Numerous scholars have studied the parallels between real and virtual worlds in online gaming and ndings
support the idea that the anonymity of cyberspace adds a new dimension to interaction (Wang & Wang, 2008); therefore
behaviors associated with gender are an example of social norms that may be virtually contextualized. With this in mind,
this study seeks to determine whether or not gender (i.e., a persons psychological identity) is a better predictor of Inter-
net use than sex.
A 44 question survey was constructed to measure the dimensions of Internet usage, gender, and demographics. In-
ternet-usage was broken into 10 basic areas (Internet, e-mail, social networking, blogging, Wikipedia, virtual libraries,
online news, online entertainment news, search engines, and fan websites) and respondents were asked to rate their fre-
quency for using each area. Gender was measured through a short form version of the Bem Sex Role Inventory (BSRI)
adapted from previous studies (Choi, Fuqua, & Newman, 2009). This version of the BSRI was chosen because of its
length and subsequent easy-of-use. In addition, a secondary goal of the study was to test the reliability and validity
of the shortened version. The BSRI (Bem, 1981) was generated to provide a measure of the degrees of masculinity,
femininity, and androgyny according to Bems gender schema theory (Bem, 1981). The original BSRI is a survey that is
comprised of 60 personality traits (20 masculine, 20 feminine, and 20 neutral) and is self-assessed. Those who wish to
measure their degrees of gender rate themselves on a scale of 1 to 7 depending on the level they believe they exhibit for
each trait. The BSRI is scored by adding together all 20 masculine scores and all 20 feminine scores, and then divides
each of these totals by 20. The shortened form of the BSRI asks 10 questions for each category rated on a scale of 1 to 4.
Participants of the study were solicited online through Facebook.com and were provided a link to the survey through
which they could give anonymous answers. A total of 270 people volunteered to take the survey, of which, 166 were fe-
male and 93 were male (11 skipped the demographics section) and they ranged in age from 18 to over 66.
The rst step in analyzing the survey data was to determine whether or not the gender items stayed true to the mas-
culine, feminine, and neutral factors as designed by Bem. Factor analysis conrmed the masculine scale, but indicated a
few items that cross loaded for both the feminine and neutral scales. Using the feminine items suggested by factor analy-
sis resulted in an alpha of .91 compared to an alpha of .90 using the original Bem items for the feminine scale. With this
in mind, it was decided to stay true to the original Bem short-form inventory scales to improve the construct validity of
the inventory.
Running an ANOVA using masculine and feminine as dependant and sex as the factor showed that both the feminine
and masculine scales were sound measures with a moderate effect size. A factor analysis of the 10 Internet-use items
revealed 2 factors, which can be described as Internet tools (Web 1.0) and Internet media (Web 2.0) and both scales had
acceptable Cronbachs alpha scores.
The purpose of this study was to determine whether or not gender was a better predictor of Internet use than sex. With
this in mind, correlations were run between the various factors with the following results:
1. There was a positive correlation, r(259) = .30, between sex and the feminine factor and a negative correlation.
r(259) =-15,. between sex and the masculine factor. Due to the fact that males were coded as a 1 and females
were coded as a2, these correlations provide support to the reliability of the gender measures. The male sex
was associated with the masculine factor and the female sex was associated with the feminine factor. These as-
sociations support the basis of gender roles being socially imposed scripts for behavior for each sex.
2. Sex was positively correlated with social networking, r(249) = .22, and negatively correlated with Wikipedia,
r(256) = -.16. This indicates that females scored higher frequency levels for social media and males scored
higher frequency levels for Wikipedia.
3. The feminine gender factor was positively correlated with the use of social networking, r(239) = .15, and
entertainment news, r(246) = .23. The masculine gender factor was positively correlated with the use of virtual
libraries, r(251) = .29, and search engines, r(249) = .13.
4. Running discriminant classifcation, the number of males predicted to be females was 34 and the number of
females predicted to be male was 47. This means that 81 (30%) of the participants were misrepresented by their
Are people more true to their authentic personalities online because of the more permissive social norms created by
virtuality? Yee (2006) found that female gamers scored signicantly higher on the relationship aspect of the social
component of online games, which seems to indicate that female gamers follow the same relationship-oriented gender
roles as they do in the real world. Yet Yee also found no sex difference in the social component as a whole. Previous
scholars have focused on sex differences in online behavior rather than gender differences. Research by Bem (1981)
suggests that evaluating masculinity and femininity as two independent dimensions better predicts behavior than evaluat-
ing gender as opposite ends of a spectrum. More conclusively, since Internet use is a specic human behavior, gender
should play a more essential role than sex in predicting gender roles (i.e., behavior) online. Jackson, Ervin, Gardner,
& Schmitt (2001) found that females used e-mail more than did males, which they claimed was consistent with their
stronger motive for interpersonal communication (p. 374). They also found that males used the Web more than did fe-
males, which the scholars claimed was consistent with their stronger motive for information (p. 374). Still more recent
research has also shown that online socializing is valued by both men and women (Grifths, Davies, & Chappell, 2004).
In contrast, participants in this study overwhelming indicated that they most enjoyed using the Internet for BOTH gather-
ing information and social interaction (76% of participants indicated they enjoyed both equally, compared with 19.5%
who primarily preferred gathering information and only 4.6% who primarily preferred social interaction). Sixty percent
of online gamers admit to swapping gender with a character (Grifths, Davies, & Chappell). Although gender swapping
is likely inuenced by the quality and appearance of the type of character that can be created rather than prescribed be-
havior of the characters sex, it seems logical that gender swapping would have a signicant inuence on virtual gender
norms. This idea just brings to light the importance of the fact that anyone can be anyone or say anything on the Internet,
thus it seems logical that the psychological identity (i.e., gender) of a person would have a greater inuence on their on-
line communication and behavior than sex. Future research, according to Bem (1994), should challenge the conventional
cultural standards that men are innately masculine and women are innately feminine. Has the Internet already marginal-
ized gender roles? Further research needs to be done into personality factors that inuence Internet use because sex is no
longer a reliable predictor. As one scholar put it, these issues can only be fully examined by researching how identities
are created online and how they compare to realworld relationships (Schrock, 2009, p.8).
Abbate, J. (1999). Inventing the internet. Cambridge, MA: The MIT Press.
Bem, S.L. (1981). Gender schema theory: a cognitive account of sex typing. Psychological Review, 88, 354-365.
Bem, S.L. (1994). In a male-centered world, the female differences are transformed into female disadvantages. The Chronicle
of Higher Education, B1-B3.
Canary, D.J., & Dindia, K. (1998). Sex differences and similarities in communication: Critical essays and empirical investiga-
tions of sex and gender interactions. Mahwah, NJ: Lawrence Erlbaum Associates.
Choi, N., Fuqua, D.R., & Newman, J.L. (2009). Exploratory and conrmatory studies of the structure of the Bem sex role in-
ventory short form with two divergent samples. Educational and Psychological Measurement, 69 (4), 696-705.
Cole, E. (2009). Feminist pedagogy in the online classroom. Proceedings of the Society for Information Technology & Teacher
Education International Conference 2009 (pp. 314-321). Chesapeake, VA: AACE.
DiMaggio, P., Hargittai, E., Neuman, W.R., & Robinson, J.P. (2001). Social implications of the Internet. Annual Review Sociol-
ogy, 27, 301.
EMarketer (2008, April 1). U.S. women out-geeking men. Retrieved from http://www.emarketer.com/Article.aspx?id=1006124.
Fallows, D. (2005). How women and men use the Internet. Pew Internet & American Life Project. Retrieved from http://www.
pewinternet.org/Reports/2005/How-Women-and-Men-Use-the-Internet.aspx.
Gorski, P. (2001). Understanding the digital divide from a multicultural education framework. EdChange Mulicultural Pavil-
ion: Digital Divide & EdTech. Retrieved from http://www.edchange.org/multicultural/net/digdiv.html.
Grifths, M.D., Davies, M.N, & Chappell, D. (2004). Demographic factors and playing variables in online computer gaming.
CyberPsychology & Behavior, 7 (4), 479-487.
Hargittai, E., & Shafter, S. (2006). Difference in actual and perceived online skills: The role of gender. Social Science Quar-
terly, 87 (2), 432.
Internet World Stats (2009). Top 20 countries with the highest Internet users. Retrieved from http://www.internetworldstats.
com/top20.htm
Jackson, L. A., Ervin, K.S., Gardner, P.D., & Schmitt, N. (2001). Gender and the internet: Women communicating and men
searching. Sex Roles, 44(5/6), 363 - 379.
Jackson, L.A., Yong, Z., Kolenic III, A., Fitzgerald, H.E., Harold, R., & VonEye, A. (2008). Race, gender, and information tech-
nology use: The new digital divide. CyberPsychology & Behavior, 11 (4), 437.
Kuhlen, R. & Michels, D. (2006). Collaborative e-learning an opportunity to identify and to overcome gender barriers. Pro-
ceedings for World Conference on E-Learning in , Government, Healthcare, and Higher Education 2006 (pp. 2593-2601).
Chesapeake, VA: AACE.
NAS Recruitment Communications (2006). Generation Y: The Millennials. Ready or not, here they come. NAS Insights re-
trieved from http://www.nasrecruitment.com/talenttips/NASinsights/GenerationY.pdf.
Pearson, J. & Davilla, R. (2001). The gender construct: Understanding why men and women communicate differently. In L. P.
Arliss & D. J. Borisoff (Eds.) Women and men communication: Challenges and changes (pp. 3-14). Prospect Heights, IL:
Waveland Press.
Rose, D. & Jones, G. (2009). Second thoughts on the digital divide. Proceedings of the Society for Information Technology &
Teacher Education International Conference 2009 (pp. 722-727). Chesapeake, VA: AACE.
Schrock, A. (2009). Examining social media usage: Technology clusters and social network site membership. First Monday,
14 (1/5), retrieved from http://rstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/2242/2066.
Suler, J. (2004). The online disinhibition effect. CyberPsychology & Behavior, 7 (3), 321.
Wang, C.C., & Wang, C.H. (2008). Helping others in online games: Prosocial behavior in cyberspace. CyberPsychology & Be-
havior, 11 (3), 344-346.
Weiser, E.B. (2000). Gender differences in Internet use patterns and Internet application preferences: A two-sample compari-
son. CyberPsychology & Behavior, 3 (2), 167.
Yee, N. (2006). Motivations for play in online games. CyberPsychology, 9 (6), 772-775.
Meeting ISTE Competencies with a Problem-Based Video Framework 243
Meeting ISTE Competencies with a Problem-Based Video Framework
YVONNE SKORETZ & AMY COTTLE
Digital learners demand digital teaching. In 2008, the International Society for Technology in Education (ISTE) re-
sponded with publishing NETS (National Educational Technology Standards) for Teachers, which outlined the standards
for digital age teaching (International Society for Technology in Education, 2008). To implement these standards in the
classroom, teachers are challenged with using technology in meaningful ways to enhance students creativity and innova-
tion, communication and collaboration skills, research and informational uency, critical thinking, problem solving and
decision making, digital citizenship, and technology operations and concepts (ISTE, 2008). Many teachers, recognizing
the need to prepare their students for a 21
century future, are searching for ways to meet these standards.
This article provides a framework for creating digital videos within a problem-based learning context to facilitate
students attainment of the competencies outlined by ISTE. When digital video is used as a tool within a problem-based
learning context, students not only learn the skills needed to complete the particular assignment, but learn how to transfer
this knowledge to future contexts. As students produce an artifact of their learning, the digital video, the authenticity of
the task places students at the center of real-world issues to enhance meaningful learning. This activity encourages stu-
dents to think critically, solve problems, reason, take risks, and negotiate (BBC News World Edition, 2002). It is the ac-
quisition of these learning and innovation skills that prepares students to live and work in an increasingly complex world
(Partnership for 21
Century Skills, 2007).
Problem-Based Learning (PBL) is dened as a learner-centered approach driven by an ill-structured problem.
Learners conduct research and apply knowledge to create a feasible solution (Savery, 2006). Students are not led down
specic paths of learning outcomes (Strobel & Van Barneveld, 2009), for there is not one right answer to the open-
ended problem (Learning Theories Knowledgebase, 2009). Rather than delivering content, PBL requires teachers to be
facilitators of learning by creating tasks and conditions where student thinking involves inquiry, dialogue, and skill build-
ing (Buck Institute of Education, 2007). Students learn how to plan and communicate in a rigorous, relevant, and engag-
ing environment that supports authentic inquiry and student autonomy.
Problem-based learning centers on constructivist principles based on Piaget and Vygotskys research ndings that
individuals construct knowledge through environmental interactions unique to each individual (Piaget, 1972; Vygotsky,
1978). Students experience learning through cognitive, social, and experiential connections, as a result, changing the fo-
cus on teaching to a focus on learning (Major & Palmer, 2001). Teachers become facilitators of knowledge, and students
no longer passively receive knowledge, but are actively searching for information. Gardners (1993) theory of multiple
intelligences also supports the use of PBL to enhance students intelligences and a need for a more personalized school
Studies show that compared to traditional teaching practices, students learning in a problem-based learning curricu-
lum are signicantly more competent and skilled, and have an increase in long-term retention of information acquired
during the project (Strobel & Van Barneveld, 2009). A meta-synthesis review on 43 research studies compared knowl-
edge learned and the ability to apply knowledge between traditional curriculums and problem-based curriculums. Find-
ings indicated students learning in a problem-based learning curriculum performed at higher knowledge levels on tasks
that required them to take their understanding of a concept and apply this knowledge (Gijbels, Dochy, Van den Bossche,
& Segers, 2005).
Problem-based learning is designed to encourage students to discover, lter and integrate information rather than just
acquire content (Keeling, 2008), in order to practice what Bloom categorizes as higher-order thinking skills (Bloom,
1956). Information is evaluated, analyzed, and synthesized as new information is being linked to old information. Stu-
244 Skoretz and Cottle
dents go beyond merely acquiring new information to blending that information with prior knowledge and experience
to arrive at a new understanding. Decisions must be made to determine the best way to represent that understanding, as
students reect on how to turn their abstract understanding into something tangible. Creating a digital video is a natural
extension of this learning process.
Digital videos have captured both teachers and students attention. Teachers are using digital videos in the class-
room to introduce and reinforce course content through short video segments known as learning objects (Nugent, 2005).
While digital videos are engaging, they also have the potential to raise student achievement (Boster, Meyer, Roberto, &
Inge, 2002). Todays student, however, yearns to be more than a consumer of digital video content. Growing up im-
mersed in digital media, the Net Generation is comfortable with not only interacting with digital media but controlling it
as well. With a voice and a need to be heard, they care deeply about social issues and believe they can make a difference
(Tapscott, 1998).
In Hobbs (2006) study, he found that digital video was most commonly used to deliver content and less common-
ly used to encourage communication, research, self-expression, and problem-solving. In a sample of 130 middle and
secondary teachers, a survey revealed 60% of those surveyed used television, video, and lm frequently in their class-
rooms as a teaching tool. When asked to provide an example of how this media tool was specically used in the class-
room to meet educational goals, 16% of the responses included the use of technology tools to create or analyze informa-
tion. Only a few respondents reported the use of assignments that required students to create a video product to demon-
strate learning.
Richardson (2006) stated that it was no longer enough to consume information; we must engage with that infor-
mation and share what we have learned (p. 26). Hung, Keppell, and Jong (2004) agreed. The authors contended that
when students were given the opportunity to create digital video products, the otherwise passive learner can be propelled
into becoming an active learner capable of creating new knowledge. Braun (2006) investigated how digital media can
be used as a tool to teach composition and rhetoric. Based upon the ndings in his study, he concluded that creating as-
signments in which students produced digital videos was a preferred way to teach rhetoric and analytic thinking as com-
pared to using the medium of print only as in the traditional assignment of writing a paper. Increased engagement led to
increased learning for students.
Lippincott (2007), too, stressed the need for teachers to enable students to be content creators. While students learn
to work with technology, they were learning not only content knowledge but a plethora of other skills that can be trans-
ferred from one setting to another. In a pilot study on the effects of student creation of digital video on student learning,
researchers conrmed their key beliefsthat high quality teaching underpinned by good subject knowledge forms a
sound basis for creative learning with ICT (BBC News World Edition, 2002, p. 1).
Many benets exist for integrating video within a PBL. One, motivation for learning increases as students are given
the opportunity to express themselves. Often times, students who struggle with reading and writing are not as effective
in expressing their voice using print. Two, creating digital video products provides the opportunity for individual expres-
sion for all students, no matter what instructional level. When students problem solve and make decisions about how
technology can be used as a tool to best meet their needs, student responsibility for their own learning increases (Bull
& Kajder, 2004). Three, when students integrate multimedia into their assignments, depth of learning increases. Four,
students become better consumers of multimedia products when they interact with content from four perspectives: re-
searcher, author, designer, and writer (International Society for Technology in Education, 2000).
Problem-Based Video Framework
The Video Production Framework is designed to create a problem-based learning context in which students create a
video to represent their learning. Each stage of the framework describes the activities students complete and many of the
stages include a Just In Time Technology lesson at the time students need it to understand the purpose, value, or strat-
egy for using technology (McKenzie, 2003). The timing of these lessons is critical in providing support and helping
students use the information when they need to use it (Warschauer and Grimes, 2005).
Stage 1: Presentation of the Scenario
Problem-based learning begins with the creation of a problem-based learning scenario. Scenarios are based on real
world problems and issues and are deve