We introduce Pixie, a novel, camera based two factor authentication solution for mobile and wearable devices. A quick and familiar user action of snapping a photo is sufficient for Pixie to simultaneously perform a graphical password authentication and a physical token based authentication, yet it does not require any expensive, uncommon hardware. Pixie establishes trust based on both the knowledge and possession of an arbitrary physical object readily accessible to the user, called trinket. Users choose their trinkets similar to setting a password, and authenticate by presenting the same trinket to the camera. The fact that the object is the trinket, is secret to the user. Pixie extracts robust, novel features from trinket images, and leverages a supervised learning classifier to effectively address inconsistencies between images of the same trinket captured in different circumstances. Pixie achieved a false accept rate below 0.09 in a brute force attack with 14.3 million authentication attempts, generated with 40,000 trinket images that we captured and collected from public datasets. We identify master images, that match multiple trinkets, and study techniques to reduce their impact. In a user study with 42 participants over 8 days in 3 sessions we found that Pixie outperforms text based passwords on memorability, speed, and user preference. Furthermore, Pixie was easily discoverable by new users and accurate under field use. Users were able to remember their trinkets 2 and 7 days after registering them, without any practice between the 3 test dates.
Similar to Pixie, cameras integrated in wearable devices have been used to capture the input for authentication. Van @cite_49 exploit the smartwatch camera to provide the device with an input (e.g. PIN) that is drawn on a canvas, then use image processing techniques to interpret the captured input. @cite_62 propose to pair and unlock smartglasses with the user smartphone by exploiting the glass camera to scan a QR code that is displayed on the user's phone screen. Similarly, @cite_39 use the smartglass camera to scan a QR code that is displayed on a point-of-service terminals (e.g. ATM) to connect to a cloud server for obtaining an OTP.
@cite_62 @cite_49 @cite_39
2294497986 2025636515 2076463985 1973831058 2963006768 2408661706 2395297283 1586237437 2042819305 2083545271 13223599 2252656680 2046378201 2293349265 2535437746 1539870437 2098910624 2344680582 2461714859 1980481605
Wearable computing devices have become increasingly popular and while these devices promise to improve our lives, they come with new challenges. This paper focuses on user authentication mechanisms for the Google Glass device (Glass). Glass only has three sources of input: a camera, a microphone, and a touchpad. This limited set of interfaces makes the use of standard passwords infeasible or cumbersome. We therefore propose a One-Time-Password (OTP) authentication scheme, Glass OTP that uses the Glass camera to scan a QR code displayed on the user’s smartphone. We implement a proof of concept Glass lock screen which unlocks only upon scanning an OTP generated by the companion Android smartphone application. We also discuss the reliability, security, and convenience of the proposed solution as compared to the current solutions in use. Smartwatches can facilitate several tasks that are performed on mobile devices. However, due to their limited size, touch interaction can be cumbersome. Although alternative input modalities such as speech input could be used, these can also introduce other issues regarding privacy or ease of use. Consequently, HCI researchers have been exploring novel input techniques for smartwatches. In this paper, we introduce WatchMe, a smartwatch application that uses the smartwatch camera and image processing techniques to allow for providing input on a drawing canvas composed of everyday objects and surfaces. We rely on a cloud OCR engine to retrieve text from captured images. Combining these characteristics, we illustrate some scenarios in which WatchMe could be used, such as a novel method for two-factor authentication. This paper presents Glass Unlock, a novel concept using smart glasses for smartphone unlocking, which is theoretically secure against smudge attacks, shoulder-surfing, and camera attacks. By introducing an additional temporary secret like the layout of digits that is only shown on the private near-eye display, attackers cannot make sense of the observed input on the almost empty phone screen. We report a user study with three alternative input methods and compare them to current state-of-the-art systems. Our findings show that Glass Unlock only moderately increases authentication times and that users favor the input method yielding the slowest input times as it avoids focus switches between displays. Today's smartphones provide services and uses that required a panoply of dedicated devices not so long ago. With them, we listen to music, play games or chat with our friends; but we also read our corporate email and documents, manage our online banking; and we have started to use them directly as a means of payment. In this paper, we aim to raise awareness of side-channel attacks even when strong isolation protects sensitive applications. Previous works have studied the use of the phone accelerometer and gyroscope as side channel data to infer PINs. Here, we describe a new side-channel attack that makes use of the video camera and microphone to infer PINs entered on a number-only soft keyboard on a smartphone. The microphone is used to detect touch events, while the camera is used to estimate the smartphone's orientation, and correlate it to the position of the digit tapped by the user. We present the design, implementation and early evaluation of PIN Skimmer, which has a mobile application and a server component. The mobile application collects touch-event orientation patterns and later uses learnt patterns to infer PINs entered in a sensitive application. When selecting from a test set of 50 4-digit PINs, PIN Skimmer correctly infers more than 30 of PINs after 2 attempts, and more than 50 of PINs after 5 attempts on android-powered Nexus S and Galaxy S3 phones. When selecting from a set of 200 8-digit PINs, PIN Skimmer correctly infers about 45 of the PINs after 5 attempts and 60 after 10 attempts. It turns out to be difficult to prevent such side-channel attacks, so we provide guidelines for developers to mitigate present and future side-channel attacks on PIN input. Wearable cameras, such as Google Glass and Go Pro, enable video data collection over larger areas and from different views. In this paper, we tackle a new problem of locating the co-interest person (CIP), i.e., the one who draws attention from most camera wearers, from temporally synchronized videos taken by multiple wearable cameras. Our basic idea is to exploit the motion patterns of people and use them to correlate the persons across different videos, instead of performing appearance-based matching as in traditional video co-segmentation localization. This way, we can identify CIP even if a group of people with similar appearance are present in the view. More specifically, we detect a set of persons on each frame as the candidates of the CIP and then build a Conditional Random Field (CRF) model to select the one with consistent motion patterns in different videos and high spacial-temporal consistency in each video. We collect three sets of wearable-camera videos for testing the proposed algorithm. All the involved people have similar appearances in the collected videos and the experiments demonstrate the effectiveness of the proposed algorithm. The proliferation of wearable devices, e.g., smartwatches and activity trackers, with embedded sensors has already shown its great potential on monitoring and inferring human daily activities. This paper reveals a serious security breach of wearable devices in the context of divulging secret information (i.e., key entries) while people accessing key-based security systems. Existing methods of obtaining such secret information relies on installations of dedicated hardware (e.g., video camera or fake keypad), or training with labeled data from body sensors, which restrict use cases in practical adversary scenarios. In this work, we show that a wearable device can be exploited to discriminate mm-level distances and directions of the user's fine-grained hand movements, which enable attackers to reproduce the trajectories of the user's hand and further to recover the secret key entries. In particular, our system confirms the possibility of using embedded sensors in wearable devices, i.e., accelerometers, gyroscopes, and magnetometers, to derive the moving distance of the user's hand between consecutive key entries regardless of the pose of the hand. Our Backward PIN-Sequence Inference algorithm exploits the inherent physical constraints between key entries to infer the complete user key entry sequence. Extensive experiments are conducted with over 5000 key entry traces collected from 20 adults for key-based security systems (i.e. ATM keypads and regular keyboards) through testing on different kinds of wearables. Results demonstrate that such a technique can achieve 80 accuracy with only one try and more than 90 accuracy with three tries, which to our knowledge, is the first technique that reveals personal PINs leveraging wearable devices without the need for labeled training data and contextual information. Traditional user authentication methods using passcode or finger movement on smartphones are vulnerable to shoulder surfing attack, smudge attack, and keylogger attack. These attacks are able to infer a passcode based on the information collection of user’s finger movement or tapping input. As an alternative user authentication approach, eye tracking can reduce the risk of suffering those attacks effectively because no hand input is required. However, most existing eye tracking techniques are designed for large screen devices. Many of them depend on special hardware like high resolution eye tracker and special process like calibration, which are not readily available for smartphone users. In this paper, we propose a new eye tracking method for user authentication on a smartphone. It utilizes the smartphone’s front camera to capture a user’s eye movement trajectories which are used as the input of user authentication. No special hardware or calibration process is needed. We develop a prototype and evaluate its effectiveness on an Android smartphone. We recruit a group of volunteers to participate in the user study. Our evaluation results show that the proposed eye tracking technique achieves very high accuracy in user authentication. Credit card fraud is a common problem in today's world. Financial institutions have registered major loses till today due to users being exposed of their credit card information. Shoulder-surfing or observation attacks, including card skimming and video recording with hidden cameras while users perform PIN-based authentication at ATM terminals is one of the common threats for common users. Researchers have struggled to come up with secure solutions for secure PIN authentication. However, modern day ubiquitous wearable devices, such as the Google Glass have presented us with newer opportunities in this research area. In this paper, we propose Secure-PIN-Authentication-as-a-Service (SEPIA), a secure obfuscated PIN authentication protocol for ATM and other point-of-service terminals using cloud-connected personal mobile and wearable devices. Our approach protects the user from shoulder-surfers and partial observation attacks, and is also resistant to relay, replay, and intermediate transaction attacks. A SEPIA user utilizes a Google Glass or a mobile device for scanning a QR code on the terminal screen to prove co-location to the cloud-based server and obtain a secure PIN template for point-of-service authentication. SEPIA ensures minimal task overhead on the user's device with maximal computation offloaded to the cloud. We have implemented a proof-of-concept prototype to perform experimental analysis and a usability study for the SEPIA architecture. We introduce an efficient camera relocalization approach which can be easily integrated into real-time 3D reconstruction methods, such as KinectFusion. Our approach makes use of compact encoding of whole image frames which enables both online harvesting of keyframes in tracking mode, and fast retrieval of pose proposals when tracking is lost. The encoding scheme is based on randomized ferns and simple binary feature tests. Each fern generates a small block code, and the concatenation of codes yields a compact representation of each camera frame. Based on those representations we introduce an efficient frame dissimilarity measure which is defined via the block-wise hamming distance (BlockHD). We illustrate how BlockHDs between a query frame and a large set of keyframes can be simultaneously evaluated by traversing the nodes of the ferns and counting image co-occurrences in corresponding code tables. In tracking mode, this mechanism allows us to consider every frame pose pair as a potential keyframe. A new keyframe is added only if it is sufficiently dissimilar from all previously stored keyframes. For tracking recovery, camera poses are retrieved that correspond to the keyframes with smallest BlockHDs. The pose proposals are then used to reinitialize the tracking algorithm. Harvesting of keyframes and pose retrieval are computationally efficient with only small impact on the run-time performance of the 3D reconstruction. Integrating our relocalization method into KinectFusion allows seamless continuation of mapping even when tracking is frequently lost. Additionally, we demonstrate how marker-free augmented reality, in particular, can benefit from this integration by enabling a smoother and continuous AR experience. Camera network systems generate large volumes of potentially useful data, but extracting value from multiple, related videos can be a daunting task for a human reviewer. Multicamera video summarization seeks to make this task more tractable by generating a reduced set of output summary videos that concisely capture important portions of the input set. We present a system that approaches summarization at the level of detected activity motifs and shortens the input videos by compacting the representation of individual activities. Additionally, redundancy is removed across camera views by omitting from the summary activity occurrences that can be predicted by other occurrences. The system also detects anomalous events within a unified framework and can highlight them in the summary. Our contributions are a method for selecting useful parts of an activity to present to a viewer using activity motifs and a novel framework to score the importance of activity occurrences and allow transfer of importance between temporally related activities without solving the correspondence problem. We provide summarization results for a two camera network, an eleven camera network, and data from PETS 2001. We also include results from Amazon Mechanical Turk human experiments to evaluate how our visualization decisions affect task performance. Wearable cameras capture a first-person view of the world, and offer a hands-free way to record daily experiences or special events. Yet, not every frame is worthy of being captured and stored. We propose to automatically predict “snap points” in unedited egocentric video—that is, those frames that look like they could have been intentionally taken photos. We develop a generative model for snap points that relies on a Web photo prior together with domain-adapted features. Critically, our approach avoids strong assumptions about the particular content of snap points, focusing instead on their composition. Using 17 hours of egocentric video from both human and mobile robot camera wearers, we show that the approach accurately isolates those frames that human judges would believe to be intentionally snapped photos. In addition, we demonstrate the utility of snap point detection for improving object detection and keyframe selection in egocentric video. Every day the use of videos from mobile devices as evidence in legal proceedings is more usual and common.Image source acquisition identification is a branch of digital forensic analysis.We use of sensor pattern noise for source identification.We make a series of experiments which emulate similar situations to those that may occur in reality. Recording videos on smartphones and other mobile devices, given their enormous popularity, is currently very common. The portability of these devices facilitates their use for recording videos in a wide variety of situations, including while witnessing criminal activities. These videos can be later used as evidence in legal proceedings. Therefore, the forensic analysis of videos taken with mobile device videos is important, and could serve for legal and also investigative purposes. It is necessary, however, to use techniques that are quite specific to this type of devices, given some peculiar features of their cameras. In this paper, we will address the issue of video source acquisition identification by presenting a technique based on sensor noise and wavelet transform extraction from video key frames. These frames are extracted using an efficient algorithm that takes their content into account, improving the selection of frames to be analyzed over past proposals. The scheme presented consists of four stages: (1) Key frames extraction, (2) sensor pattern noise extraction, (3) feature extraction, and (4) classifier training and prediction. We also present experimental results that support the validity of the techniques used and show promising results. In this paper, we propose and evaluate Use Your Illusion, a novel mechanism for user authentication that is secure and usable regardless of the size of the device on which it is used. Our system relies on the human ability to recognize a degraded version of a previously seen image. We illustrate how distorted images can be used to maintain the usability of graphical password schemes while making them more resilient to social engineering or observation attacks. Because it is difficult to mentally "revert" a degraded image, without knowledge of the original image, our scheme provides a strong line of defense against impostor access, while preserving the desirable memorability properties of graphical password schemes. Using low-fidelity tests to aid in the design, we implement prototypes of Use Your Illusion as i) an Ajax-based web service and ii) on Nokia N70 cellular phones. We conduct a between-subjects usability study of the cellular phone prototype with a total of 99 participants in two experiments. We demonstrate that, regardless of their age or gender, users are very skilled at recognizing degraded versions of self-chosen images, even on small displays and after time periods of one month. Our results indicate that graphical passwords with distorted images can achieve equivalent error rates to those using traditional images, but only when the original image is known. This paper proposes an efficient and effective scheme to applying the sliding window approach popular in computer vision to 3D data. Specifically, the sparse nature of the problem is exploited via a voting scheme to enable a search through all putative object locations at any orientation. We prove that this voting scheme is mathematically equivalent to a convolution on a sparse feature grid and thus enables the processing, in full 3D, of any point cloud irrespective of the number of vantage points required to construct it. As such it is versatile enough to operate on data from popular 3D laser scanners such as a Velodyne as well as on 3D data obtained from increasingly popular push-broom configurations. Our approach is “embarrassingly parallelisable” and capable of processing a point cloud containing over 100K points at eight orientations in less than 0.5s. For the object classes car, pedestrian and bicyclist the resulting detector achieves best-in-class detection and timing performance relative to prior art on the KITTI dataset as well as compared to another existing 3D object detection approach. Detecting and classifying targets in video streams from surveillance cameras is a cumbersome, error-prone and expensive task. Often, the incurred costs are prohibitive for real-time monitoring. This leads to data being stored locally or transmitted to a central storage site for post-incident examination. The required communication links and archiving of the video data are still expensive and this setup excludes preemptive actions to respond to imminent threats. An effective way to overcome these limitations is to build a smart camera that analyzes the data on-site, close to the sensor, and transmits alerts when relevant video sequences are detected. Deep neural networks (DNNs) have come to outperform humans in visual classifications tasks and are also performing exceptionally well on other computer vision tasks. The concept of DNNs and Convolutional Networks (ConvNets) can easily be extended to make use of higher-dimensional input data such as multispectral data. We explore this opportunity in terms of achievable accuracy and required computational effort. To analyze the precision of DNNs for scene labeling in an urban surveillance scenario we have created a dataset with 8 classes obtained in a field experiment. We combine an RGB camera with a 25-channel VIS-NIR snapshot sensor to assess the potential of multispectral image data for target classification. We evaluate several new DNNs, showing that the spectral information fused together with the RGB frames can be used to improve the accuracy of the system or to achieve similar accuracy with a 3x smaller computation effort. We achieve a very high per-pixel accuracy of 99.1 . Even for scarcely occurring, but particularly interesting classes, such as cars, 75 of the pixels are labeled correctly with errors occurring only around the border of the objects. This high accuracy was obtained with a training set of only 30 labeled images, paving the way for fast adaptation to various application scenarios. The emergence of new pervasive wearable technologies (e.g. action cameras and smart glasses) calls attention to the so called First Person Vision (FPV). In the future, more and more everyday-life videos will be shot from a first-person point of view, overturning the classical fixed-camera understanding of Vision, specializing the existing knowledge of moving cameras and bringing new challenges in the field of video processing. The trend in research is going to be oriented towards a new type of computer vision, centred on moving sensors and driven by the need for new applications for wearable devices. We identify hand tracking and gesture recognition as an essential topic in this field, motivated by the simple realization that we often look at our hand, even while performing the simplest tasks in everyday life. In addition, the next frontier in user interfaces are hands-free devices. In this work we argue that applications based on FPV may involve information fusion at various complexity and abstraction levels, ranging from pure image processing to inference over patterns. We address the lowest, by proposing a first investigation on hand detection from a first-person point of view sensor and some preliminary results obtained fusing colour and optic flow information. Lifelogging devices are spreading faster everyday. This growth can represent great benefits to develop methods for extraction of meaningful information about the user wearing the device and his her environment. In this paper, we propose a semi-supervised strategy for easily discovering objects relevant to the person wearing a first-person camera. Given an egocentric video images sequence acquired by the camera, our algorithm uses both the appearance extracted by means of a convolutional neural network and an object refill methodology that allows to discover objects even in case of small amount of object appearance in the collection of images. An SVM filtering strategy is applied to deal with the great part of the False Positive object candidates found by most of the state of the art object detectors. We validate our method on a new egocentric dataset of 4912 daily images acquired by 4 persons as well as on both PASCAL 2012 and MSRC datasets. We obtain for all of them results that largely outperform the state of the art approach. We make public both the EDUB dataset and the algorithm code. The ubiquity of wearable and implantable devices has sparked a new set of mobile computing applications that leverage the abundant information from sensors. For many of these applications, ensuring the security of communication between legitimate devices is a crucial problem. In this paper, we design Walkie-Talkie, a shared secret key generation scheme that allows two legitimate devices to establish a common cryptographic key by exploiting users' walking characteristics (gait). The intuition is that the sensors on different locations on the same body experience similar accelerometer signals when the user is walking. However, the accelerometer also captures motion signals produced by other body parts (e.g., swinging arms). It is shown that a Blind Source Separation (BSS) technique can extract the informative signal produced by the unique gait patterns. Our experimental results show that the keys generated by two independent devices on the same body are able to achieve up to a 100 bit agreement rate. To demonstrate the feasibility, the proposed key generation scheme is implemented on modern smartphones. The evaluation results show that the proposed scheme can run in real-time on modern mobile devices and incurs low system overhead. Users are increasingly vulnerable to inadvertently leaking sensitive information through cameras. In this paper, we investigate an approach to mitigating the risk of such inadvertent leaks called privacy markers. Privacy markers give users fine-grained control of what visual information an app can access through a device's camera. We present two examples of this approach: PrivateEye, which allows a user to mark regions of a two-dimensional surface as safe to release to an app, and WaveOff, which does the same for three-dimensional objects. We have integrated both systems with Android's camera subsystem. Experiments with our prototype show that a Nexus 5 smartphone can deliver near realtime frame rates while protecting secret information, and a 26-person user study elicited positive feedback on our prototype's speed and ease-of-use. The rapid deployment of sensing technology in smartphones and the explosion of their usage in people's daily lives provide users with the ability to collectively sense the world. This leads to a growing trend of mobile healthcare systems utilizing sensing data collected from smartphones with without additional external sensors to analyze and understand people's physical and mental states. However, such healthcare systems are vulnerable to user spoofing attacks, in which an adversary distributes his registered device to other users such that data collected from these users can be claimed as his own to obtain more healthcare benefits and undermine the successful operation of mobile healthcare systems. Existing mitigation approaches either only rely on a secret PIN number (which can not deal with colluded attacks) or require an explicit user action for verification. In this paper, we propose a user verification scheme leveraging unique gait patterns derived from acceleration readings in mobile healthcare systems to detect possible user spoofing attacks. Our framework exploits the readily available accelerometers embedded within smartphones for user verification. Specifically, our user spoofing attack mitigation scheme (which consists of three components, namely Step Cycle Identification, Step Cycle Interpolation, and Similarity Score Computation) is used to extract gait patterns from run-time accelerometer measurements to perform robust user verification under various walking speeds. Our experiments using 322 smartphone-based traces over a period of 6 months confirm that our scheme is highly effective for detecting user spoofing attacks. This strongly indicates the feasibility of using smartphone based low grade accelerometer to conduct gait recognition and facilitate effective user verification without active user cooperation.