Author(s): Kaczmarek, T; Kobsa, A; Sy, R; Tsudik, G | Abstract: User errors while performing security-critical tasks can lead to undesirable or even disastrous consequences. One major factor influencing mistakes and failures is complexity of such tasks, which has been studied extensively in prior research. Another important issue which hardly received any attention is the impact of both accidental and intended distractions on users performing security-critical tasks. In particular, it is unclear whether, and to what extent, unexpected sensory cues (e.g., auditory or visual) can influence user behavior and or trigger mistakes. Better understanding of the effects of intended distractions will help clarify their role in adversarial models. As part of the research effort described in this paper, we administered a range of naturally occurring -- yet unexpected -- sounds while study participants attempted to perform a security-critical task. We found that, although these auditory cues lowered participants' failure rates, they had no discernible effect on their task completion times. To this end, we overview some relevant literature that explains these somewhat counter-intuitive findings. Conducting a thorough and meaningful study on user errors requires a large number of participants, since errors are typically infrequent and should not be instigated more than once per subject. To reduce the effort of running numerous subjects, we developed a novel experimental setup that was fully automated and unattended. We discuss our experience with this setup and highlight the pros and cons of generalizing its usage.
To the best of our knowledge, there has been no prior usability study utilizing a fully automated (unattended) experimental setup with a video-projected experimenter. However, there have been precedents with virtually attended experiments and unattended online surveys, in many fields, most notably, psychology. There is a sizable body of work supporting validity and precision of such unattended online experiments, as compared to more traditional attended experiments in a lab setting. In particular, @cite_33 found no significant difference in psychometric data collected from an attended experiment in a lab setting and its online, virtually attended counterpart. This is further supported by @cite_8 in the comparison of data collected from unattended online and attended offline questionnaires. Guidelines for creating the best possible draw from the intended population base are provided by Birnbaum @cite_17 . Finally, Lazem and Gracanin @cite_9 replicated two classical social psychology experiments, where the experimenter and the three participants were represented as avatars in Second Life instead of being physically co-present. The outcomes were very similar.
@cite_9 @cite_17 @cite_33 @cite_8
1489557789 2135944616 2160641905 2112508839
One of the most disconcertingly unnatural properties of most virtual environments (VEs) is the ability of the user to pass through objects. I hypothesize that passive haptics, augmenting a high-fidelity visual virtual environment with low-fidelity physical objects, will markedly improve both sense of presence and spatial knowledge training transfer. The low-fidelity physical models can be constructed from cheap, easy-to-assemble materials such as styrofoam, plywood, and particle board. The first study investigated the effects of augmenting a visual-cliff environment with a slight physical ledge on participants' sense of presence. I found when participants experienced passive haptics in the VE, they exhibited significantly more behaviors associated with pit avoidance than when experiencing the non-augmented VE. Changes in heart rate and skin conductivity were significantly higher than when they experienced the VE without passive haptics. The second study investigated passive haptics' effects on performance of a real-world navigation task after training in a virtual environment. Half of the participants trained on maze navigation in a VE augmented with a styrofoam physical model, while half trained in a non-augmented VE but were given visual and audio contact cues. The task was to gain as much information as possible about the layout of the environment. Participants knew before the VE session that their training would be tested by navigating an identical real maze environment while blindfolded. Significant differences in the time to complete the blindfolded navigation task and significant differences in the number of collisions with objects were found between the participants trained in an augmented VE and the participants trained in a non-augmented VE. 11 of 15 participants trained without passive haptics bumped into the next-to-last obstacle encountered in the testing session and turned the wrong direction to navigate around it: only 2 of 15 participants trained with passive haptics made the same navigation error. On the other hand, the assessment of the participants' cognitive maps of the virtual environment did not find significant differences between groups as measured by sketch maps and object dimension estimation. The Internet can be an effective medium for the posting, exchange, and collection of information in psychology-related research and data. The relative ease and inexpensiveness of creating and maintaining Web-based applications, associated with the simplicity of use via the graphic-user interface format of form-based surveys, can establish a new research frontier for the social and behavioral sciences. To explore the possible use of Internet tools in psychological research, this study compared Web-based assessment techniques with traditional paper-based methods of different measures of Internet attitudes and behaviors in an Italian sample. The collected data were analyzed to identify both differences between the two samples and in the psychometric characteristics of the questionnaires. Even if we found significant differences between the two samples in the Internet attitudes and behaviors, no relevant differences were found in the psychometric properties of the different questionnaires. This result, simila... Given the ease that humans have with using a keyboard and mouse in typical, non-colocated computer interaction, many studies have investigated the value of colocating the visual field and motor workspaces using immersive display modalities. Significant understanding has been gained by previous work comparing physical tasks against virtual tasks, visuomotor colocation versus non-colocation, and even visuomotor rotational misalignments in virtual environments (VEs). However, few studies have explored all of these paradigms in context with each other, and it is difficult to perform interstudy comparisons because of the variation in tested motor tasks. Therefore, using a stereoscopic fish tank display setup, the goal for the current study was to characterize human performance of a 3D Fitts' point-to-point reaching task using a stylus-based haptic interface in the physical, colocated non-colocated, and rotated VE visualization conditions. Five performance measures---throughput, efficiency, initial movement error, corrective movements, and peak velocity---were measured and used to evaluate task performance. These measures were studied in 22 subjects (11 male, 11 female, ages 20--32) performing a 3D variant of Fitts' serial task under 10 task conditions: physical, colocated VE, non-colocated VE, and rotated VEs from 45---315° in 45° increments. Hypotheses: All performance measures in the colocated VE were expected to reflect significantly reduced task performance over the real condition, but also reflect increased performance over the non-colocated VE condition. For rotational misalignments, all performance measures were expected to reflect the highest performance at 0°, reduce to the lowest performance at 90°, and rise again to a local maximum at 180° (symmetric about 0°). Results: All performance measures showed that the colocated VE condition resulted in significantly lower task performance than the physical condition and higher mean performance than the non-colocated VE condition, but the difference was not statistically significant. Also, rotation misalignments showed that task performance was mostly reduced to a minimum at 90°, 135°, and 225°. We conclude that colocated VEs may not significantly improve point-to-point reaching performance over non-colocated VEs. Also, visual rotations of ±45° affected throughput, efficiency, peak velocity, and initial movement error, but the number of corrective movements were not affected until ± 90°. Web-facing companies, including Amazon, eBay, Etsy, Facebook, Google, Groupon, Intuit, LinkedIn, Microsoft, Netflix, Shop Direct, StumbleUpon, Yahoo, and Zynga use online controlled experiments to guide product development and accelerate innovation. At Microsoft's Bing, the use of controlled experiments has grown exponentially over time, with over 200 concurrent experiments now running on any given day. Running experiments at large scale requires addressing multiple challenges in three areas: cultural organizational, engineering, and trustworthiness. On the cultural and organizational front, the larger organization needs to learn the reasons for running controlled experiments and the tradeoffs between controlled experiments and other methods of evaluating ideas. We discuss why negative experiments, which degrade the user experience short term, should be run, given the learning value and long-term benefits. On the engineering side, we architected a highly scalable system, able to handle data at massive scale: hundreds of concurrent experiments, each containing millions of users. Classical testing and debugging techniques no longer apply when there are billions of live variants of the site, so alerts are used to identify issues rather than relying on heavy up-front testing. On the trustworthiness front, we have a high occurrence of false positives that we address, and we alert experimenters to statistical interactions between experiments. The Bing Experimentation System is credited with having accelerated innovation and increased annual revenues by hundreds of millions of dollars, by allowing us to find and focus on key ideas evaluated through thousands of controlled experiments. A 1 improvement to revenue equals more than $10M annually in the US, yet many ideas impact key metrics by 1 and are not well estimated a-priori. The system has also identified many negative features that we avoided deploying, despite key stakeholders' early excitement, saving us similar large amounts.