In a typical video conferencing setup, it is hard to maintain eye contact during a call since it requires looking into the camera rather than the display. We propose an eye contact correction model that restores the eye contact regardless of the relative position of the camera and display. Unlike previous solutions, our model redirects the gaze from an arbitrary direction to the center without requiring a redirection angle or camera display user geometry as inputs. We use a deep convolutional neural network that inputs a monocular image and produces a vector field and a brightness map to correct the gaze. We train this model in a bi-directional way on a large set of synthetically generated photorealistic images with perfect labels. The learned model is a robust eye contact corrector which also predicts the input gaze implicitly at no additional cost. Our system is primarily designed to improve the quality of video conferencing experience. Therefore, we use a set of control mechanisms to prevent creepy results and to ensure a smooth and natural video conferencing experience. The entire eye contact correction system runs end-to-end in real-time on a commodity CPU and does not require any dedicated hardware, making our solution feasible for a variety of devices.
Although some of the earlier work employed temporal smoothing techniques @cite_2 , earlier gaze correction and redirection solutions overall tried to correct the gaze constantly, without a control mechanism. Therefore, the use of a general-purpose gaze redirector for video conferencing would lead to unnatural results particularly when the user is not engaged or moves away from a typical use case.
@cite_2
2082997166 1944180106 2038685445 1998609690
Effective communication using current video conferencing systems is severely hindered by the lack of eye contact caused by the disparity between the locations of the subject and the camera. While this problem has been partially solved for high-end expensive video conferencing systems, it has not been convincingly solved for consumer-level setups. We present a gaze correction approach based on a single Kinect sensor that preserves both the integrity and expressiveness of the face as well as the fidelity of the scene as a whole, producing nearly artifact-free imagery. Our method is suitable for mainstream home video conferencing: it uses inexpensive consumer hardware, achieves real-time performance and requires just a simple and short setup. Our approach is based on the observation that for our application it is sufficient to synthesize only the corrected face. Thus we render a gaze-corrected 3D model of the scene and, with the aid of a face tracker, transfer the gaze-corrected facial portion in a seamless manner onto the original image. We revisit the well-known problem of gaze correction and present a solution based on supervised machine learning. At training time, our system observes pairs of images, where each pair contains the face of the same person with a fixed angular difference in gaze direction. It then learns to synthesize the second image of a pair from the first one. After learning, the system gets the ability to redirect the gaze of a previously unseen person by the same angular difference as in the training set. Unlike many previous solutions to gaze problem in videoconferencing, ours is purely monocular, i.e. it does not require any hardware apart from an in-built web-camera of a laptop. Being based on efficient machine learning predictors such as decision forests, the system is fast (runs in real-time on a single core of a modern laptop). In the paper, we demonstrate results on a variety of videoconferencing frames and evaluate the method quantitatively on the hold-out set of registered images. The supplementary video shows example sessions of our system at work. In video conferences, one user can either look at the remote image of the other captured by a video camera, or look at the camera that is capturing her him, but not two tasks at the same time. The lack of eye contact caused by this misalignment substantially reduces the effectiveness of communication with an unpleasant feeling of disconnectedness. We propose an approach to bring eye contact back while user looks at the remote image of the other by a novel system composed by a Time-of-Flight depth sensor and traditional stereo. The key success of this system is to faithfully recover scene's depth. In this 2.5D space, the controlling of the user's eye-gaze becomes relatively easier. To evaluate the performance of the system, we conducted two user studies. One focuses on subjects have been trained to be familiar to eye gaze displayed in images; another is blind evaluation that subjects have no prior knowledge about eye gaze. Both evaluations show that the system can bring desktop participants closer to each other. Much research has been done in the past for segmenting lecture videos by detecting slide transitions. However, they do not perform well on certain kinds of videos recorded under non-stationary settings: the changes of a camera position or focus during a lecture. Since such non-stationary settings greatly affect visual properties of slides, the existing approaches utilizing global features and a global threshold, often have trouble in computing similarities between slides. In this paper, we propose a highly accurate method for lecture video segmentation by using SIFT and an adaptive threshold. By using SIFT, we can reliably match two slides whose contents are the same but are visually different. We also propose an adaptive threshold selection algorithm that detects slide transitions accurately by considering characteristics of features. Through various experiments that use real lecture videos, we show that our method provides 30 improvement in the average F1-score over other existing methods.