Multimodality can make (especially mobile) device interaction more efficient. Sensors and communication capabilities of modern smartphones and tablets lay the technical basis for its implementation. Still, mobile platforms do not make multimodal interaction support trivial. Building multimodal applications requires various APIs with different paradigms, high-level interpretation of contextual data, and a method for fusing individual inputs and outputs. To reduce this effort, we created a framework that simplifies and accelerates the creation of multimodal applications for prototyping and research. It provides an abstraction of information representations in different modalities, unifies access to implicit and explicit information, and wires together the logic behind context-sensitive modality switches. In the paper, we present the structure and features of our framework, and validate it by four implemented demonstrations of different complexity.
For rapid development of context-sensitive applications in the research context, several toolkits and frameworks have been presented, starting with the by @cite_1 . The toolkit by Fogarty and Hudson @cite_5 supports the development of applications that use sensor-based statistical models. Thereby, contextual information can be used to adapt certain settings automatically. With our framework, we focus on wiring of functionality, as often the impacting factors for a desired action are quite clear. There exist alternative approaches (e.g., support vector machines, Bayesian or neural networks), but machine learning often makes it hard to understand a certain action has been performed. Some toolkits address particular use cases, e.g. physical mobile interaction @cite_19 or proxemic interaction @cite_10 . Du and Wang @cite_16 developed a model and implementation framework to simulate context events with Symbian phones. used JCOP (Context-Oriented Programming) extensions to the Java language that support context-dependent execution for Android programming @cite_14 , which, however, complicates a quick integration in any existing Android project and working environment.
@cite_14 @cite_1 @cite_19 @cite_5 @cite_16 @cite_10
2006015625 2051712055 1980661875 2210641851
The behavior of mobile applications is particularly affected by their execution context, such as location and state a the mobile device. Among other approaches, context-oriented programming can help to achieve context-dependent behavior without sacrificing modularity or adhering to a certain framework or library by enabling fine-grained adaptation of default behavior per control-flow. However, context information relevant for mobile applications is mostly defined by external events and sensor data rather than by code and control flow. To accommodate this, the JCop language provides a more declarative approach by pointcut-like adaptation rules. In this paper, we explain how we applied JCop to the development of Android applications for which we extended the language semantics for static contexts and modified the compiler. Additionally, we outline the successful implementation of a simple, proof-of-concept mobile application using our approach and report on promising early evaluation results. Applications running on mobile devices are heavily context-aware and adaptive, leading to new analysis and testing challenges as streams of context values drive these applications to undesired configurations that are not easily exposed by existing validation techniques. We address this challenge by employing a finite-state model of adaptive behavior to enable the detection of faults caused by (1) erroneous adaptation logic, and (2) asynchronous updating of context information, which leads to inconsistencies between the external physical context and its internal representation within an application. We identify a number of adaptation fault patterns, each describing a class of faulty behaviors that we detect automatically by analyzing the system's adaptation model. We illustrate our approach on a simple but realistic application in which a cellphone's configuration profile is changed automatically based on the user's location, speed and surrounding environment. Context has been playing an increasingly important role in areas such as object detection, scene understanding, and image segmentation. Although many different types of contextual cues have been successfully explored, most of them only consider the pair-wise relationship between objects or parts. Several models utilize the high-order relationship for encoding contextual information. However, they mainly use a single contextual cue. In this paper, we present a novel high-order contextual descriptor (HOOD) to measure the strength of interactions among objects within an image. Heterogeneous contextual cues like semantic, spatial, and scale contexts are jointly integrated into HOOD to define the high-order interactions. The strength of these interactions are inferred by applying Bayes’ rule on the pure dependence of the involved objects. As a result, an object-level graph is constructed to represent the contextually consistent interactions. Moreover, we propose a HOOD based object localization framework to verify the effectiveness of HOOD. Experimental results on two benchmark datasets including SUN09 and PASCAL2007 show that our framework outperforms the state-of-the-art context based object localization methods. Finally, we apply HOOD on two multimedia applications: structured image retrieval and out-of-context object detection, which demonstrates the potential usages of HOOD. With the development of video-sharing websites, P2P, micro-blog, mobile WAP websites, and so on, sensitive videos can be more easily accessed. Effective sensitive video recognition is necessary for web content security. Among web sensitive videos, this paper focuses on violent and horror videos. Based on color emotion and color harmony theories, we extract visual emotional features from videos. A video is viewed as a bag and each shot in the video is represented by a key frame which is treated as an instance in the bag. Then, we combine multi-instance learning (MIL) with sparse coding to recognize violent and horror videos. The resulting MIL-based model can be updated online to adapt to changing web environments. We propose a cost-sensitive context-aware multi- instance sparse coding (MI-SC) method, in which the contextual structure of the key frames is modeled using a graph, and fusion between audio and visual features is carried out by extending the classic sparse coding into cost-sensitive sparse coding. We then propose a multi-perspective multi- instance joint sparse coding (MI-J-SC) method that handles each bag of instances from an independent perspective, a contextual perspective, and a holistic perspective. The experiments demonstrate that the features with an emotional meaning are effective for violent and horror video recognition, and our cost-sensitive context-aware MI-SC and multi-perspective MI-J-SC methods outperform the traditional MIL methods and the traditional SVM and KNN-based methods.