Personal devices (e.g. laptops, tablets, and mobile phones) are conventional in daily life and have the ability to store users' private data. The security problems related to these appliances have become a primary concern for both users and researchers. In this paper, we analyse first-person-view videos to develop a personalized user authentication mechanism. Our proposed algorithm generates provisional image-based passwords which benefit a variety of purposes such as unlocking a mobile device or fallback authentication. First, representative frames are extracted from the egocentric videos. Then, they are split into distinguishable segments before a clustering procedure is applied to discard repetitive scenes. The whole process aims to retain memorable images to form the authentication challenges. We integrate eye tracking data to select informative sequences of video frames and suggest a blurriness-based method if an eye-facing camera is not available. To evaluate our system, we perform experiments in different settings including object-interaction activities and traveling contexts. Even though our mechanism produces variable graphical passwords, the log-in effort for the user is comparable with approaches based on static challenges. We verified the authentication challenges in the presence of a random and an informed attacker who is familiar with the environment and observed that the time required and the number of attempts are significantly higher than for the legitimate user, making it possible to detect attacks on the authentication system.
Visual context recognition is a further area closely related to our research. Furnari @cite_25 analysed egocentric videos to facilitate the recognition of personal contexts. They recorded the videos with different camera models in various locations. Each image is represented with a holistic descriptor (or the gist'' of the scene), local features (Scale-invariant feature transform or SIFT) encoded by Improved Fisher Vectors, and values from the penultimate layer of Convolutional Neural Networks. Castro @cite_6 combined Convolutional Neural Networks and Random Decision Forests to identify human activities from egocentric images. Even though our work focuses on personal surroundings, we do not define a fixed set of contexts to be identified. In particular, our image selection method, which relies on segmentation and clustering, aims to group video frames based on their visual characteristics without prior knowledge on the contexts and contents.
@cite_25 @cite_6
1947727767 2198618282 2033639255 854053868
Understanding how images of objects and scenes behave in response to specific ego-motions is a crucial aspect of proper visual development, yet existing visual learning methods are conspicuously disconnected from the physical source of their images. We propose to exploit proprioceptive motor signals to provide unsupervised regularization in convolutional neural networks to learn visual representations from egocentric video. Specifically, we enforce that our learned features exhibit equivariance i.e. they respond systematically to transformations associated with distinct ego-motions. With three datasets, we show that our unsupervised feature learning system significantly outperforms previous approaches on visual recognition and next-best-view prediction tasks. In the most challenging test, we show that features learned from video captured on an autonomous driving platform improve large-scale scene recognition in a disjoint domain. Understanding how images of objects and scenes behave in response to specific ego-motions is a crucial aspect of proper visual development, yet existing visual learning methods are conspicuously disconnected from the physical source of their images. We propose to exploit proprioceptive motor signals to provide unsupervised regularization in convolutional neural networks to learn visual representations from egocentric video. Specifically, we enforce that our learned features exhibit equivariance, i.e, they respond predictably to transformations associated with distinct ego-motions. With three datasets, we show that our unsupervised feature learning approach significantly outperforms previous approaches on visual recognition and next-best-view prediction tasks. In the most challenging test, we show that features learned from video captured on an autonomous driving platform improve large-scale scene recognition in static images from a disjoint domain. Identifying handled objects, i.e. objects being manipulated by a user, is essential for recognizing the person's activities. An egocentric camera as worn on the body enjoys many advantages such as having a natural first-person view and not needing to instrument the environment. It is also a challenging setting, where background clutter is known to be a major source of problems and is difficult to handle with the camera constantly and arbitrarily moving. In this work we develop a bottom-up motion-based approach to robustly segment out foreground objects in egocentric video and show that it greatly improves object recognition accuracy. Our key insight is that egocentric video of object manipulation is a special domain and many domain-specific cues can readily help. We compute dense optical flow and fit it into multiple affine layers. We then use a max-margin classifier to combine motion with empirical knowledge of object location and background movement as well as temporal cues of support region and color appearance. We evaluate our segmentation algorithm on the large Intel Egocentric Object Recognition dataset with 42 objects and 100K frames. We show that, when combined with temporal integration, figure-ground segmentation improves the accuracy of a SIFT-based recognition system from 33 to 60 , and that of a latent-HOG system from 64 to 86 . While egocentric video is becoming increasingly popular, browsing it is very difficult. In this paper we present a compact 3D Convolutional Neural Network (CNN) architecture for long-term activity recognition in egocentric videos. Recognizing long-term activities enables us to temporally segment (index) long and unstructured egocentric videos. Existing methods for this task are based on hand tuned features derived from visible objects, location of hands, as well as optical flow. Given a sparse optical flow volume as input, our CNN classifies the camera wearer's activity. We obtain classification accuracy of 89 , which outperforms the current state-of-the-art by 19 . Additional evaluation is performed on an extended egocentric video dataset, classifying twice the amount of categories than current state-of-the-art. Furthermore, our CNN is able to recognize whether a video is egocentric or not with 99.2 accuracy, up by 24 from current state-of-the-art. To better understand what the network actually learns, we propose a novel visualization of CNN kernels as flow fields.