Role of context in usability evaluations: A review <s> Based on the social context <s> Computer professionals have a need for robust, easy-to-use usability evaluation methods (UEMs) to help them systematically improve the usability of computer artifacts. However, cognitive walkthrough (CW), heuristic evaluation (HE), and thinking- aloud study (TA)-3 of the most widely used UEMs-suffer from a substantial evaluator effect in that multiple evaluators evaluating the same interface with the same UEM detect markedly different sets of problems. A review of 11 studies of these 3 UEMs reveals that the evaluator effect exists for both novice and experienced evaluators, for both cosmetic and severe problems, for both problem detection and severity assessment, and for evaluations of both simple and complex systems. The average agreement between any 2 evaluators who have evaluated the same system using the same UEM ranges from 5% to 65%, and no 1 of the 3 UEMs is consistently better than the others. Although evaluator effects of this magnitude may not be surprising for a UEM as informal as HE, it is cer... <s> BIB001 </s> Role of context in usability evaluations: A review <s> Based on the social context <s> Constructive interaction provides natural thinking-aloud as test subjects collaborate in pairs to solve tasks. Since children may face difficulties in following instructions for a standard think-aloud test, constructive interaction has been suggested as evaluation method when usability testing with children. However, the relationship between think-aloud and constructive interaction is still poorly understood. We present an experiment that compares think-aloud and constructive interaction. The experiment involves 60 children with three setups where children apply think-aloud or constructive interaction in acquainted and non-acquainted pairs. Our results show that the pairing of children had impact on how the children collaborated in pairs and how they would afterward assess the testing sessions. In some cases, we found that acquainted dyads would perform well as they would more naturally interact and collaborate while in other cases they would have problems in controlling the evaluations. <s> BIB002 </s> Role of context in usability evaluations: A review <s> Based on the social context <s> This paper focuses on the interaction between test participants and test facilitator in two variants of the think-aloud method. In a first, explorative study, we analyzed think-aloud transcripts from two usability tests: a concurrent think-aloud test and a constructive interaction test. The results of our analysis show that while the participants in both studies never explicitly addressed the facilitator, the think-aloud participants showed more signs of awareness of the facilitator than the participants in the constructive interaction test. This finding may have practical implications for the validity of the two methods. <s> BIB003
The effect of social context in usability evaluations can be understood by examining some of the previous works in the domain. Work by Benedikte et al. BIB002 involved conducting usability testing with 60 children with three setups where children apply think-aloud or constructive interaction in acquainted and non-acquainted pairs. The results show that the pairing of children had impact on how the children verbalized and collaborated in pairs during the testing sessions. The children in pairs had a high level of verbalization, but often they were more talking aloud than actually thinking aloud. The acquainted dyads were significantly more satisfied with their own performance and they did not feel it demanded a lot of effort for them. Study by Jacobsen et al.  examines the evaluator effect in the usability tests. In their study four HCI research evaluators, all familiar with the theory and practice of usability analyzed four video tapes. The evaluators were asked to detect and describe all problems in the interface based on analyzing the four tapes in a preset order, without any time constraints. The results indicate that only 20% of the 93 unique problems were detected by only a single evaluator. Severe problems were detected by more often by all four evaluators (41%) and less often by only one evaluator (22%), however, the evaluator effect remained substantial. Evaluator effect has also been probed by Hertzum and Jacobsen BIB001 , where they found that different evaluators evaluating the same system with same usability evaluation methods detect substantially different sets of usability problems in the system. van den Haak & de Jong BIB003 analyzed the interaction between test monitor and participants in concurrent think aloud (CTA) method and constructive interaction (CI) test. The results indicate that the presence of the test monitor has most notably affected the CTA participants but has also has its impact on CI participants. They found that a more serious threat to the validity of both the CTA method and CI method was that the participants acknowledged the test monitor as an evaluator of their actions.