We address the problem of incrementally modeling and forecasting long-term goals of a first-person camera wearer: what the user will do, where they will go, and what goal they seek. In contrast to prior work in trajectory forecasting, our algorithm, DARKO, goes further to reason about semantic states (will I pick up an object?), and future goal states that are far in terms of both space and time. DARKO learns and forecasts from first-person visual observations of the user's daily behaviors via an Online Inverse Reinforcement Learning (IRL) approach. Classical IRL discovers only the rewards in a batch setting, whereas DARKO discovers the states, transitions, rewards, and goals of a user from streaming data. Among other results, we show DARKO forecasts goals better than competing methods in both noisy and ideal settings, and our approach is theoretically and empirically no-regret.
Wearable cameras have been used for various human behavior understanding tasks @cite_4 @cite_30 @cite_19 @cite_15 @cite_21 because they give direct access to detailed visual information about a person's actions. Leveraging this feature of FPV, recent work has shown that it is possible to predict where people will look during actions @cite_19 and how people will use the environment @cite_35 .
@cite_30 @cite_35 @cite_4 @cite_21 @cite_19 @cite_15
1990930131 2240290602 1539870437 2963006768
For understanding the behavior, intent, and environment of a person, the surveillance metaphor is traditional; that is, install cameras and observe the subject, and his her interaction with other people and the environment. Instead, we argue that the first-person vision (FPV), which senses the environment and the subject's activities from a wearable sensor, is more advantageous with images about the subject's environment as taken from his her view points, and with readily available information about head motion and gaze through eye tracking. In this paper, we review key research challenges that need to be addressed to develop such FPV systems, and describe our ongoing work to address them using examples from our prototype systems. Wearable cameras can gather first-person images of the environment, opening new opportunities for the development of systems able to assist the users in their daily life. This paper studies the problem of recognizing personal contexts from images acquired by wearable devices, which finds useful applications in daily routine analysis and stress monitoring. To assess the influence of different device-specific features, such as the Field Of View and the wearing modality, a dataset of five personal contexts is acquired using four different devices. We propose a benchmark classification pipeline which combines a one-class classifier to detect the negative samples (i.e., images not representing any of the personal contexts under analysis) with a classic one-vs-one multi-class classifier to discriminate among the contexts. Several experiments are designed to compare the performances of many state-of-the-art representations for object and scene classification when used with data acquired by different wearable devices. The emergence of new pervasive wearable technologies (e.g. action cameras and smart glasses) calls attention to the so called First Person Vision (FPV). In the future, more and more everyday-life videos will be shot from a first-person point of view, overturning the classical fixed-camera understanding of Vision, specializing the existing knowledge of moving cameras and bringing new challenges in the field of video processing. The trend in research is going to be oriented towards a new type of computer vision, centred on moving sensors and driven by the need for new applications for wearable devices. We identify hand tracking and gesture recognition as an essential topic in this field, motivated by the simple realization that we often look at our hand, even while performing the simplest tasks in everyday life. In addition, the next frontier in user interfaces are hands-free devices. In this work we argue that applications based on FPV may involve information fusion at various complexity and abstraction levels, ranging from pure image processing to inference over patterns. We address the lowest, by proposing a first investigation on hand detection from a first-person point of view sensor and some preliminary results obtained fusing colour and optic flow information. Wearable cameras, such as Google Glass and Go Pro, enable video data collection over larger areas and from different views. In this paper, we tackle a new problem of locating the co-interest person (CIP), i.e., the one who draws attention from most camera wearers, from temporally synchronized videos taken by multiple wearable cameras. Our basic idea is to exploit the motion patterns of people and use them to correlate the persons across different videos, instead of performing appearance-based matching as in traditional video co-segmentation localization. This way, we can identify CIP even if a group of people with similar appearance are present in the view. More specifically, we detect a set of persons on each frame as the candidates of the CIP and then build a Conditional Random Field (CRF) model to select the one with consistent motion patterns in different videos and high spacial-temporal consistency in each video. We collect three sets of wearable-camera videos for testing the proposed algorithm. All the involved people have similar appearances in the collected videos and the experiments demonstrate the effectiveness of the proposed algorithm.