Process Mining is a famous technique which is frequently applied to Software Development Processes, while being neglected in Human-Computer Interaction (HCI) recommendation applications. Organizations usually train employees to interact with required IT systems. Often, employees, or users in general, develop their own strategies for solving repetitive tasks and processes. However, organizations find it hard to detect whether employees interact efficiently with IT systems or not. Hence, we have developed a method which detects inefficient behavior assuming that at least one optimal HCI strategy is known. This method provides recommendations to gradually adapt users' behavior towards the optimal way of interaction considering satisfaction of users. Based on users' behavior logs tracked by a Java application suitable for multi-application and multi-instance environments, we demonstrate the applicability for a specific task in a common Windows environment utilizing realistic simulated behaviors of users.
developed a framework for the OpenOffice.org Suite which enables the logging of HCIs while interacting with OpenOffice @cite_1 . This tool enables many opportunities to analyze user's behavior in OpenOffice applications. However, it is limited to OpenOffice only and does not track HCI between applications and instances outside the OpenOffice.org Suite. Unfortunately, no research has been conducted on user's behavior logs observed by OpenOffice in order to recommend user's interactions.
@cite_1
2071889901 2338918436 2168053878 2096214539
We introduce a research framework which enables to use the OpenOffice.org as a platform for HCI research, particularly for performing user studies or prototyping and evaluating intelligent user interfaces. We make two contributions: (1) we introduce an innovative hybrid logging technique which provides high-level, rich and accurate information about issued user commands, command parameters and used interaction styles. Our logging technique also avoids an unwanted requirement for further complex processing of logged user interface events to infer user commands, which must be performed on most loggers these days. (2) Our logging tool acts as a component object in OpenOffice.org with an easy-to-use Application Program Interface (API) which enables, along with deep OpenOffice.org programmability, to use OpenOffice.org as a research framework for developing and evaluating intelligent user interfaces. Recent years have witnessed the increasing emphasis on human aspects in software engineering research and practices. Our survey of existing studies on human aspects in software engineering shows that screen-captured videos have been widely used to record developers' behavior and study software engineering practices. The screen-captured videos provide direct information about which software tools the developers interact with and which content they access or generate during the task. Such Human-Computer Interaction (HCI) data can help researchers and practitioners understand and improve software engineering practices from human perspective. However, extracting time-series HCI data from screen-captured task videos requires manual transcribing and coding of videos, which is tedious and error-prone. In this paper we report a formative study to understand the challenges in manually transcribing screen-captured videos into time-series HCI data. We then present a computer-vision based video scraping technique to automatically extract time-series HCI data from screen-captured videos. We also present a case study of our scvRipper tool that implements the video scraping technique using 29-hours of task videos of 20 developers in two development tasks. The case study not only evaluates the runtime performance and robustness of the tool, but also performs a detailed quantitative analysis of the tool's ability to extract time-series HCI data from screen-captured task videos. We also study the developer's micro-level behavior patterns in software development from the quantitative analysis. The ability of a computer to detect and appropriately respond to changes in a user's affective state has significant implications to human-computer interaction (HCI). In this paper, we present our efforts toward audio-visual affect recognition on 11 affective states customized for HCI application (four cognitive motivational and seven basic affective states) of 20 nonactor subjects. A smoothing method is proposed to reduce the detrimental influence of speech on facial expression recognition. The feature selection analysis shows that subjects are prone to use brow movement in face, pitch and energy in prosody to express their affects while speaking. For person-dependent recognition, we apply the voting method to combine the frame-based classification results from both audio and visual channels. The result shows 7.5 improvement over the best unimodal performance. For person-independent test, we apply multistream HMM to combine the information from multiple component streams. This test shows 6.1 improvement over the best component performance This paper describes technology and tools for intelligent human-computer interaction (IHCI) in which human cognitive, perceptual, motor and affective factors are modeled and used to adapt the H-C interface. IHCI emphasizes that human behavior encompasses both apparent human behavior and the hidden mental state behind behavioral performance. IHCI expands on the interpretation of human activities, known as W4 (what, where, when, who). While W4 only addresses the apparent perceptual aspect of human behavior the W5+ technology for IHCI described in this paper addresses also the why and how questions, whose solution requires recognizing specific cognitive states. IHCI integrates parsing and interpretation of nonverbal information with a computational cognitive model of the user which, in turn, feeds into processes that adapt the interface to enhance operator performance and provide for rational decision-making. The technology proposed is based on a general four-stage interactive framework, which moves from parsing the raw sensory-motor input, to interpreting the user's motions and emotions, to building an understanding of the user's current cognitive state. It then diagnoses various problems in the situation and adapts the interface appropriately. The interactive component of the system improves processing at each stage. Examples of perceptual, behavioral, and cognitive tools are described throughout the paper Adaptive and intelligent HCI are important for novel applications of computing, including ubiquitous and human-centered computing.