Role of context in usability evaluations: A review <s> DISCUSSION <s> There are new challenges to us, as researchers, on how to design and evaluate new mobile applications because they give users access to powerful computing devices through small interfaces, which typically have limited input facilities. One way of overcoming these shortcomings is to utilize the possibilities of multimodality. We report in this paper how we designed, developed, and evaluated a multimodal mobile application through a combination of laboratory and field studies. This is the first time, as far as we know, that a multimodal application has been developed in such a way. We did this so that we would understand more about where and when users envisioned using different modes of interaction and what problems they may encounter when using an application in context. <s> BIB001 </s> Role of context in usability evaluations: A review <s> DISCUSSION <s> Constructive interaction provides natural thinking-aloud as test subjects collaborate in pairs to solve tasks. Since children may face difficulties in following instructions for a standard think-aloud test, constructive interaction has been suggested as evaluation method when usability testing with children. However, the relationship between think-aloud and constructive interaction is still poorly understood. We present an experiment that compares think-aloud and constructive interaction. The experiment involves 60 children with three setups where children apply think-aloud or constructive interaction in acquainted and non-acquainted pairs. Our results show that the pairing of children had impact on how the children collaborated in pairs and how they would afterward assess the testing sessions. In some cases, we found that acquainted dyads would perform well as they would more naturally interact and collaborate while in other cases they would have problems in controlling the evaluations. <s> BIB002 </s> Role of context in usability evaluations: A review <s> DISCUSSION <s> This paper focuses on the interaction between test participants and test facilitator in two variants of the think-aloud method. In a first, explorative study, we analyzed think-aloud transcripts from two usability tests: a concurrent think-aloud test and a constructive interaction test. The results of our analysis show that while the participants in both studies never explicitly addressed the facilitator, the think-aloud participants showed more signs of awareness of the facilitator than the participants in the constructive interaction test. This finding may have practical implications for the validity of the two methods. <s> BIB003 </s> Role of context in usability evaluations: A review <s> DISCUSSION <s> Preface. Acknowledgements. PART I: INTRODUCTION. Chapter 1: Possibilities. Key points. 1.1 Introduction. 1.2 What are mobile devices?. 1.2.1 Communication or information device? 1.2.2 Appliance or swiss-army knife? 1.2.3 Cherished device or commodity tool? 1.3 Impoverished or extraordinary interfaces? 1.3.1 The Fastap keypad. 1.3.2 Peephole displays. 1.3.3 Accommodating human capabilities and limitations. 1.4 Impoverishing interactions? 1.4.1 Reasons for poor design. 1.4.2 Impacts of poor design. 1.5 Outline of the rest of this book. Summary. Workshop questions. Designer tips. Chapter 2: Products for people. Key points. 2.1 Introduction. 2.2 Useful. 2.2.1 Function before form. 2.2.2 Evolving uses. 2.3 Usable. 2.3.1 Usable in itself. 2.3.2 Usable in life. 2.4 User experience. 2.4.1 Strong identity. 2.4.2 Interaction as package. 2.5 Technology acceptance. Summary. Workshop questions. Designer tips. Chapter 3: Innovating. Key points. 3.1 Introduction. 3.2 Technology-centered approaches. 3.3 Transferring from the desktop. 3.3.1 Applications. 3.3.2 Interface styles. 3.4 Building on past mobile success. 3.5 Drama. 3.6 Frameworks for human-centered thinking. Summary. Workshop questions. Designer tips. PART II: DEVELOPING EFFECTIVE MOBILE APPLICATIONS. Chapter 4: Interaction design. Key points. 4.1 Introduction. 4.2 Designing what? Designing how? 4.3 Understanding users. 4.3.1 From biology to psychology. 4.3.2 Field studies. 4.3.3 Direct questioning. 4.3.4 Distilling the findings. 4.4 Developing prototype designs. 4.4.1 Shaping and managing the design space. 4.4.2 Prototyping. 4.5 Evaluation. 4.5.1 Testing with users. 4.5.2 Testing in the absence of users. 4.6 Iterative development. 4.7 Multiple viewpoints. 4.7.1 Many techniques and tools. 4.7.2 Many disciplines. 4.7.3 Participation and collaboration. 4.8 From interaction design to deployment. Summary. Workshop questions. Designer tips. Chapter 5: Watching, asking, probing. Key points. 5.1 Introduction. 5.2 Focusing the study. 5.2.1 How focused should the study be? 5.2.2 Finding people to study. 5.3 Top techniques for understanding people. 5.3.1 Observing. 5.3.2 Enquiring. 5.3.3 Diary studies. 5.3.4 Discount methods. 5.3.5 Focus groups. 5.3.6 Creatively engaging methods. 5.4 Making sense of observations. 5.4.1 Activities. 5.4.2 Analysis perspectives. 5.5 Personas and scenarios. 5.5.1 Personas. 5.5.2 Scenarios. Summary. Workshop questions. Designer tips. Chapter 6: Prototypes. Key points. 6.1 Introduction. 6.2 What is a prototype? 6.3 Different prototypes for different purposes. 6.4 Low-fidelity. 6.4.1 Self-checking. 6.4.2 Communicating with others. 6.4.3 Interaction prototyping. 6.4.4 Empowering users. 6.5 Higher-fidelity. 6.5.1 Deciding what to prototype. 6.5.2 Hardware and software integration. 6.6 Finishing the process. 6.6.1 Evolutionary. 6.6.2 Revolutionary. 6.6.3 Process. 6.7 Issues in prototyping. 6.7.1 Some considerations. 6.8 A final note on development. Summary. Workshop questions. Designer tips. Chapter 7: Evaluation. Key points. 7.1 Introduction. 7.2 Classifying evaluation. 7.3 'Quick And Dirty'. 7.4 Conceptual model extraction. 7.5 Direct observation. 7.5.1 Finding out what users are thinking. 7.5.2 How to record observations. 7.5.3 How to not bias the experiment. 7.5.4 Happy users. 7.6 Interviews. 7.7 Questionnaires. 7.8 Non-user methods. 7.8.1 Heuristic evaluation. 7.8.2 No people whatsoever. 7.9 Experimental evaluation. 7.9.1 Hypothesis. 7.9.2 The users. 7.9.3 Tasks. 7.9.4 Experiment design. 7.9.5 Conducting experiments. 7.9.6 Experimental results. 7.10 Considering context - evaluating mobile systems. 7.10.1 Physical context. 7.10.2 Technological context. 7.10.3 Social context. 7.10.4 Other contexts. 7.11 Complementary evaluation. 7.12 Conclusion. Summary. Workshop questions. Designer tips. PART III: DESIGN GALLERY - DIRECTIONS AND GUIDELINES. Chapter 8: Controlling Complex Functions. Key points. 8.1 Introduction. 8.2 Menus and memory. 8.3 Hierarchical menus. 8.3.1 Learning structure. 8.3.2 Improving classification. 8.3.3 Context information. 8.4 Icons. 8.5 Manuals. 8.5.1 Online manuals. 8.5.2 Website manuals. 8.6 No menus?. 8.6.1 Data structures. 8.6.2 Alternatives. 8.6.3 Design ideas - data structures. 8.6.4 Evaluation via experiment. 8.7 More complex menus. 8.8 Some concluding thoughts. Summary. Workshop questions. Designer tips. Chapter 9: Information access. Key points. 9.1 Introduction. 9.2 Small-screen impacts. 9.2.1 Lessons from the past. 9.2.2 Impact on browsing complex content. 9.2.3 Impact on searching. 9.3 Designs for browsing. 9.3.1 Guidelines. 9.3.2 Schemes. 9.3.3 Packaging content. 9.4 Improving search. 9.4.1 Assessing sets of results. 9.4.2 Judging the value of individual results. 9.5 Mobile information ecologies. 9.5.1 Fitting in. 9.5.2 Case study: the laid-back search tool. 9.5.3 Peer-to-peer schemes. Summary. Workshop questions. Designer tips. Chapter 10: Beyond text - using images on mobile devices. Key points. 10.1 Introduction. 10.2 Ethnography. 10.2.1 Where have all the photographs gone? 10.2.2 Digital ethnography. 10.3 Finding photos. 10.3.1 Meta-data. 10.3.2 Meta-data management. 10.4 Browsing photos. 10.4.1 Timeline. 10.4.2 Treemaps. 10.4.3 RSVP. 10.4.4 Speed Dependent Automatic Zooming. 10.5 Downscaling case study. 10.5.1 Arranging photos. 10.5.2 Screen size. 10.5.3 Write once, run anywhere. 10.5.4 Meanwhile, back with the users. 10.5.5 User testing. 10.5.6 Platform. 10.6 Advanced technology. 10.7 What are photos for? 10.7.1 What are we sharing? 10.7.2 Using audio with photographs. 10.7.3 Video. 10.8 Looking to the future. Summary. Workshop questions. Designer tips. Chapter 11: Impacting the community impacting the world. Key points. 11.1 Introduction. 11.2 The digital divide. 11.3 Mobiles work. 11.3.1 The rise and rise of mobile technology. 11.4 Planning a project. 11.5 That culture thing. 11.6 Case studies. 11.6.1 Empowering people - Cybertracker. 11.6.2 Education. 11.6.3 Communitization. 11.7 Call to arms. Summary. Workshop questions. Designer tips. Resources. References. Index. <s> BIB004 </s> Role of context in usability evaluations: A review <s> DISCUSSION <s> This paper describes the involvement of teachers as facilitators in user-based evaluation of educational software. Findings from a case study that compare the behaviour of teachers and designers during the evaluation indicate that there is no strong evidence to support the current practice of teachers' tangential participation in the evaluation of educational software. Further research is needed to better understand the role teachers can play in this type of evaluation so that their knowledge and experience can positively contribute to the evaluation process and outcomes, whilst retaining the children's valuable contribution. <s> BIB005 </s> Role of context in usability evaluations: A review <s> DISCUSSION <s> Environmental conditions arguably influence mobile website usability, especially as users on the move may be exposed to fluctuating contextual conditions. Since the aspects of context cannot be readily simulated in a mobile usability study, the development of theories and models explaining how the context of use may affect usability has yet to be sufficiently addressed. In this paper we propose a model hypothesizing on the effects of environmental distractions on the usability of mobile sites. Environmental distractions are categorized into auditory, visual and social and we suggest both direct and moderated effects of these distractions on efficiency, effectiveness, and perceived satisfaction. The paper contributes to the mobile HCI literature by developing a concrete set of empirically testable hypotheses and a method for their subsequent assessment. A preliminary test (N=30) was conducted to investigate our main hypotheses. Results confirmed that environmental distractions have a direct effect on mobile website usability. <s> BIB006 </s> Role of context in usability evaluations: A review <s> DISCUSSION <s> In usability evaluations, experiments are often conducted in closed laboratory situations to avoid disturbing influences. Due to non-realistic usage assumptions, this approach has important shortcomings when mobile Human Computer Interactions (m-HCI) have to be evaluated. Field studies allow to perform experiments close to real-world conditions, but potentially introduce influences caused by the environment. In this paper, we aim at distinguishing application shortcomings from environmental disturbances which both potentially cause decreased user performance. Our approach is based on monitoring environmental conditions during the usability experiment, such as light, acceleration, sound, temperature, and humidity, and relating them to user actions. Therefore, a mobile context-framework has been developed based on a small Wireless Sensor Network (WSN). First results are presented that point at increased delays and error rates of user tasks under induced environmental disturbances. Additionally, we demonstrate the potential of environmental monitoring for understanding user performance. <s> BIB007 </s> Role of context in usability evaluations: A review <s> DISCUSSION <s> Usability testing has traditionally been conducted in dedicated laboratory. Alternatively, field study can be an option to conduct the usability testing. As HCI researchers who work with children, we have experiences in conducting usability tests in both natural and laboratory environments. We tested two different children applications in two different environments: testing drawing applications at their preschool and testing an educational game in our usability laboratory. Thus, this paper highlights our experiences in conducting both usability tests. Although the literature suggests that laboratory and field studies should be done for two different purposes, our experiences have told us that we can actually integrate the steps of field studies into the guidelines for lab tests. These steps are particularly useful for the novice designers and researchers who have never done usability evaluations with children before. <s> BIB008 </s> Role of context in usability evaluations: A review <s> DISCUSSION <s> This study proposes a new methodology for conducting synchronous remote usability studies using a three-dimensional virtual usability testing laboratory built using the Open Wonderland toolkit. This virtual laboratory method is then compared with two other commonly used synchronous usability test methods: the traditional lab approach and WebEx, a web-based conferencing and screen sharing approach. A study was conducted with 48 participants in total, 36 test participants and 12 test facilitators. The test participants completed 5 tasks on a simulated e-commerce website. The three methodologies were compared with respect to the following dependent variables: the time taken to complete the tasks; the usability defects identified; the severity of these usability defects; and the subjective ratings from NASA-TLX, presence and post-test subjective questionnaires. The three methodologies agreed closely in terms of the total number defects identified, number of high severity defects identified and the time taken to complete the tasks. However, there was a significant difference in the workload experienced by the test participants and facilitators, with the traditional lab condition imposing the least and the virtual lab and the WebEx conditions imposing similar levels. It was also found that the test participants experienced greater involvement and a more immersive experience in the virtual world condition than the WebEx condition. These ratings were not significantly different from those in the traditional lab condition. The results of this study suggest that participants were productive and enjoyed the virtual lab condition, indicating the potential of a virtual world based approach as an alternative to the conventional approaches for synchronous usability testing. <s> BIB009
We reviewed many works which took into consideration the importance of the physical as well as the social contexts in usability evaluations. The summary of the relevant work is given in table 2. The entries in the table depict the type of context that was addressed in the work, whether context affected the outcomes of the work and the target participants. Physical context in the table refers to both the laboratory evaluation and field evaluation. Social context refers to the people involved in the test. Both refer to the physical and the social context taken together. BIB006 Mobile website BIB007 Mobile  Mobile commerce application BIB008 Drawing application  Commercial website BIB009 E-commerce website BIB001 Multimodal mobile application BIB002 Mobile phone  Video tapes BIB003 Online library catalogues Physical context is addressed by many studies whereas, the social context in usability evaluations is explored by a fewer studies. Most of the studies show that the context (physical and social) has an impact on the outcomes of the evaluations. The focus on physical context in usability evaluations indicates that this aspect of context is considered important. The choice is between evaluating in an artificial setting such as the laboratory or in a more natural setting through field evaluations. Each has its own strengths and weaknesses. Jones & Marsden BIB004 state that the social context in usability evaluation is equally important. The main objective of creating a social context in usability evaluation is to facilitate effective and efficient evaluations. Pardo et al. BIB005 examined the effect of teacher's involvement in usability testing with children. They found that children cannot provide proper feedback on the learning goals they have not experienced before. In such cases the input of the other stakeholders such as the teachers will be beneficial. Some studies also reported that, compared to evaluations involving adult participants, studies involving children are mostly affected by the context. Children show varying behaviour when they are tested in the laboratory environment and when they are tested in the field. They feel relaxed and confident in their own environment. Children also show varying behaviour when they are accompanied by the people they are well acquainted with and with the people they are not acquainted with. Almost 80% of the work that we surveyed shows a clear impact of the context on the outcomes of the usability evaluations. Studies concerning the mobile applications usability testing are the most affected by the context. With mobile applications, better insight to the usability problems can be unveiled in the field than in the lab. None the less, usability evaluations involving adults may also be effected by the context. While previous research studies in usability evaluations have largely focused on the physical aspects of context in usability evaluations, work on social aspects of context is scarce. Behaviour of the test participants is affected by the context which may in turn affect the results of usability evaluations. Therefore, behaviour and settings are inseparable units as claimed by Barker's theory of behaviour settings. More research taking into account both the social and physical context together is needed to uncover the importance of context in usability evaluations.