EgoReID: Person re-identification in Egocentric Videos Acquired by Mobile Devices with First-Person Point-of-View

Widespread use of wearable cameras and recording devices such as cellphones have opened the door to a lot of interesting research in first-person Point-of-view (POV) videos (egocentric videos). In recent years, we have seen the performance of video-based person Re-Identification (ReID) methods improve considerably. However, with the influx of varying video domains, such as egocentric videos, it has become apparent that there are still many open challenges to be faced. These challenges are a result of factors such as poor video quality due to ego-motion, blurriness, severe changes in lighting conditions and perspective distortions. To facilitate the research towards conquering these challenges, this paper contributes a new, first-of-its-kind dataset called EgoReID. The dataset is captured using 3 mobile cellphones with non-overlapping field-of-view. It contains 900 IDs and around 10,200 tracks with a total of 176,000 detections. Moreover, for each video we also provide 12-sensor meta data. Directly applying current approaches to our dataset results in poor performance. Considering the unique nature of our dataset, we propose a new framework which takes advantage of both visual and sensor meta data to successfully perform Person ReID. In this paper, we propose to adopt human body region parsing to extract local features from different body regions and then employ 3D convolution to better encode temporal information of each sequence of body parts. In addition, we also employ sensor meta data to determine target's next camera and their estimated time of arrival, such that the search is only performed among tracks present in the predicted next camera around the estimated time. This considerably improves our ReID performance as it significantly reduces our search space.


Introduction
Person Re-Identification (ReID) aims at associating the same pedestrian across multiple cameras [11,54]. This task has drawn increasing attention in recent years due to its importance in applications, such as surveillance [43], activity analysis [26] and tracking [48]. Person ReID remains a challenging problem because of complex variations in camera viewpoints, human poses, lighting, occlusions, and background clutter.
Currently, in almost all attempts to solve person ReID problem, the source of data is from fixed cameras with wider field of view (FOV). Mostly, in such datasets, full bodies of pedestrians are captured and intra camera variations of targets are very small, as pose of targets often do not change within the same camera. On the contrary, videos from a moving first-person point-of-view (POV) devices pose unique challenges, such as frequent occlusion of targets due to close proximity of camera and targets, blurriness caused by camera motion, sever lighting and background changes, both within and across  Figure 1: (a) shows three tracklets of the same identity from the same camera, its evident to see that due to both camera and target motions a target is captured in different poses, backgrounds and illuminations. (b) each row shows tracklets of the same identity captured from camera 1, 2 and 3, respectively. Due to close proximity of target and the cameras, different body regions of the target are missing in Camera 1 and 2. cameras, appearance difference due to different perspectives. Moreover, due to constant motion of both target and cameras, there is a frequent inter camera appearance changes between sequences of the same target. This is mainly due to the fact that a target could be recorded in several poses in a single camera as shown in Fig. 1 (a). The scene is more dynamic as compared to a traditional fixed camera scenario.
Besides the unique challenges posed by the nature of the video, these devices (such as cellphones) provide additional source of information. Most new generation devices are equipped with additional sensors such as gyros, accelerometers, magnetometers, GPS and are Internet-enabled, it is now possible to obtain large amounts of first-person point-of-view (POV) data un-intrusively.
Aiming to facilitate the research towards such unique video domain, we propose a new dataset called EgoReID. Different from existing datasets EgoReID presents several new features. 1) 3 synchronized mobile cellphones with non-overlapping FOV are used to record the videos. 2) Throughout the recording, all 3 cameras are moving around campus covering larger area, resulting in complex scene transformation and background. 3) 12 sensor meta data along with the videos are also collected. 4) YOLO9000 [30] and FCDSC [39] are used for pedestrian detection and tracklet generation, respectively, which will be provided with the dataset.
In a typical person ReID pipeline, first, each pedestrian video sequence is represented by a single feature vector and then is matched in a task-specific metric space, where the feature vectors of same target are expected to have smaller distance than that of different target [46,23,24]. These approaches show impressive results, when target sequences do not undergo drastic view changes or occlusion. However, in egocentric videos pedestrians undergo drastic change in both appearance and pose due to frequent motion of both camera and targets. Furthermore, frequent occlusion of different body parts of targets is common due to close proximity of a target to the camera as shown in Fig. 1 To address aforementioned challenges posed by the first-person POV videos, we propose a framework which uses human semantic parsing to extract local features from human body parts. It is natural to make use of semantic body regions in this particular setup, since mostly pedestrians are in close proximity from cameras, thus, different body regions are more visible. However, most of Person Re-ID datasets deal with image-based re-ID problem and do not deal with videos, therefore very few video-based Person re-ID datasets are available for training video deep networks for re-ID. In this paper, we propose a novel approach to leverage network trained on 10 image-based re-ID datasets for video-based Person Re-ID problem. Here, we employ a model, trained on these image-based datasets, to extract frame level local features for each semantic region, then we employ 3D convolutions to encode the temporal information in each sequence of regions. Instead of directly encoding the whole sequence, we use sequence of each regions to learn discriminative local features and compare respective regions separately to find a match. In addition to traditional video features, we successfully demonstrate the use of additional sensor meta data to complement appearance cues. In particular, we use meta data information to estimate target's next camera and its time of arrival, that way we significantly reduce the search space by pruning several unreliable pairs which violate the estimated time of arrival constraint.
The contributions of this paper can be summarized as follows: 1) A unique and challenging EgoR-eID dataset is collected and will be released with detections, tracks and sensor meta data. Compared to existing datasets, EgoReID has a unique feature which can help advance research in solving realistic and challenging task of person re-identification in a first-person POV videos. 2) We propose a new model to solve person ReID problem in egocentric videos, where human semantic parsing is employed to extract local visual cues and compare corresponding semantic regions to find a match. 3) We propose a method to employ sensor meta data information to significantly reduce our search space by first predicting the next camera of a target then estimating its time of arrival, thus, we only search for our match among tracks which satisfy the time constraint.

Related Works
In this section, we review related works in the areas of video-based person re-identification and egocentric vision.
Video-based person re-identification: The success of deep learning in wide range of computer vision areas has been inspiring a lot of studies in person re-identification. In recent years, researchers are considering more realistic scenarios such as larger dataset [53,52], complex scenarios [55,31] and combining different modalities such as text descriptions in their approaches [56,19].
The success of deep learning in image classification [16], has inspired several studies in person re-identification. The effectiveness of Convolutional Neural Network (CNN) in learning discriminative image representations from large scale datasets have been exploited effectively in [40,41,25,17,6,51,58,49,38].
Utilization of video data is further facilitated by the powerful feature learning ability of Convolutional Neural Networks.
In video-based ReID [29,58,46,42,59,28], the learning algorithm is given pair of video sequences instead of images. Authors in [29], introduce an RNN model to encode temporal information. The features from all timesteps are then combined using temporal pooling to give an overall appearance feature for the complete sequence and then compute the feature similarity of two videos. In [42] the most discriminative video fragments are selected from noisy/incomplete image sequences of people from which reliable space-time and appearance features can be computed, whilst simultaneously learning a video ranking function for person ReID. Authors in [28], introduce a new space-time person representation by encoding multiple granularities of spatio-temporal dynamics in form of time series. In [58] attention weights are combined per-frame with visual features and the forward propagated RNN hidden variables, in order to weaken the influence of the noisy samples. Authors in [45] employ spatial pooling layer to select regions from each frame, while temporal attention pooling is performed to select informative frames over the sequence, both pooling guided by the information from their distance matching. In [36], the dual attention mechanism is used, in which both intra-sequence and intersequence attention strategies are employed for feature refinement and feature-pair alignment, respectively.
Egocentric vision: Recently, visual analysis of egocentric videos has been a hot topic in computer vision, and the work ranging from object detection [9] to recognizing daily activities [8,7], predicting gaze behavior [23,3], and video summarization [27] have been proposed.

EgoReID Dataset
In this section, we briefly introduce our new EgoReID dataset.

Dataset Description
During collection, we use 3 synchronized Samsung mobile cellphones with non-overlapping field of views. Throughout the period of recording, all 3 cameras were constantly moving around and covered larger area. To generate tracklets, we first use YOLO9000 [30] to detect pedestrians and then employ FCDSC tracker [39] to generate trackles. Sample tracklets are shown in Fig. 1 and more are given in the supplementary material. EgoReID consists of around 900 different identities with 10,200 tracklets with a total of 176,000 human detection bounding boxes. Camera 1 and 2 have 190 IDs in common while camera 2 and 3 share 256 IDs in common. Around 103 IDs are present in all three cameras. 3164, 2748 and 4353 tracklets are present in camera 1,2 and 3, respectively.
Evaluation protocol: Similar to most of previous datasets, we utilize the Cumulative Matching Characteristics (CMC) curve to evaluate the ReID performance. For each query, multiple true positives could be returned. Therefore, we also consider person ReID as a retrieval task, and also employ mean Average Precision (mAP) as the evaluation metric.

Proposed Method
This section briefly presents the proposed model which consists of 3 main components: frame-level feature extraction, semantic segmentation and videolevel feature extraction. The overview of the proposed network is shown in Figure 3.
Our model is inspired by the model [14], where human parsing is successfully applied in imagebased ReID. Important feature of their model is that it is trained on ten different dataset providing robust system. In the unique nature of our dataset, since targets are close to the cameras, different body regions of a target are clearly visible, therefore above method is pretty suitable for our problem. However, their method is image-based, in this paper, we extend their method to videos and achieve significantly better results as compared to the existing state-of-the-art method [35] on our dataset.
Among several ways to extend the image-based model to videos, one is to inflate [4] the bottom 2D layers to 3D, such that it accepts a video clip and generates its single compact representation. Since we have two modules: feature extraction and segmentation to process a video, we need to inflate both of them. Even though we can employ the abovementioned modification over the feature extraction module, since we have enough labeled video dataset for training our extended model, however, our semantic segmentation module suffers from the lack of semantic segmentation annotations at video level. Therefore, we apply feature extraction and semantic segmentation at frame level and then pool each feature map using different segmentation regions. This is followed by 3D convolution over feature activa-tion map, with attention applied to each region, to better encode the temporal information in the video sequence.
Below, first we describe Feature Extraction and Semantic Segmentation Modules followed by videobased re-ID .
Feature sequence extraction module: In our model, we adopt Inception-V3 [37] as a backbone of our feature extraction module. Given a video sequence X ∈ R H×W ×3×S of length S, where each frame (RGB) is of size H ×W , each frame in video at a time-step is passed to Inception-V3 to produce the frame-level feature maps. At the top of the Inception-V3, using bilinear interpolation, the feature maps are scaled up to the size of the segmentation maps.
Semantic segmentation module: For the segmentation module, as in the feature extraction module, we use the Inception-V3 architecture. However, we make two important modification to this architecture compared to the one used for feature extraction. The first modification is that the output stride of the model is reduced from 32 to 16 to get the feature maps with adequate resolution for semantic segmentation task. The extra computation cost resulting from this modification is eliminated by the replacing the convolutions in the last Inception block with the dilated convolutions [47]. The second modification is the use of atrous spatial pyramid pooling [5], instead of global average pooling to exploit multi-scale information. Refer to [14] for more details about the feature extraction and semantic segmentation modules.
Video-based person ReID model: Given a video sequence, each frame in a video at a time-step is passed to an Inception-V3 to produce the frame-level feature maps. Simultaneously, our semantic segmentation module is fed the same sequence, which generates segmentation (probability) maps of 3 semantic regions for each frame in the sequence. We pool the output of our feature extractor model using the 3 segmentation maps. This results in a set of 3 maps, each focusing on a different body region of the sequence. Next, to encode the temporal information, we pass the activation sequence of each region to 3 consecutive 3D convolutions that do not share the weights. As a result of this encoding process, a tensor is obtained for each body region and feature vectors are constructed using the Global Average Pool-  1  12  23  34  45  56  67  78  89  100  111  122  133  144  155  166  177  188  199  210  221  232  243  254  265  276  287  298 Rotation -X   Figure 3: Proposed Network: First, the input video clip consisting of S frames are input to the Inception-V3 models, the feature maps and 3 segmentation maps for each frame (f-1 to f-S) are generated in the upper and lower branches of the model, respectively. We pool the feature maps using the segmentation maps and obtain sequences of 3 feature maps, each focusing on a different body regions: lower, upper and full. Next, we aggregate the temporal information within these sequences utilizing 3D convolutions, which operate with the stride of 2 and with the kernel size of 3 for the temporal 1 for the spatial dimensions. With this encoding process, we obtain a tensor of size (30x30x2048) for each body region and apply global average pooling (GAP) to construct the feature vectors. In addition to these vectors, we construct another feature vector taking the average of the output feature maps of Inception-V3 (top branch) and applying GAP on the resulting tensor.
ing (GAP) on these tensors. We also use one more feature vector, constructed by taking the average of the feature maps produced in the feature extractor module and applying GAP on it. During the training of the model, we pass the 3 feature vectors of the body regions to different soft-max layers that do not share the weights and compute the classification losses separately. We do not use the global features while training the model. At the test time, we concatenate all region and global features to get the final descriptors for the input videos.

Improving ReID Employing Sensor
Meta-data In this section, we discuss the proposed method for employing sensor meta data information to further help refine our re-identification results. In particular, we use the heading, speed and GPS (longitude and latitude) of the camera, which are captured by our recording devices.
Let T a i represents an observation of a person i in camera a. Its trajectory is given by a set of points where t s and t τ represent the time it entered and exited camera a, respectively. ρ(T a i ) denote appearance feature representation of tracklet T a i generated by the proposed model. GPS and heading of camera a at time t τ are respectively represented by C a g t τ and C a h t τ . The heading of a target i in camera a, T a h i , is inferred from the camera a s heading at time t τ , that is, if target is moving in the same direction as camera (target is walking away from the camera), then T a h i = C a h t τ . Where as, if a target is moving towards the camera, like the target in Fig. 4, then T a h i = C a h t τ + 180 0 , that is, we assign the opposite heading w.r.t the camera's heading. The direction of target's movement is determined from the tracklet direction.
Estimating the next camera: Naïvely, one can select the closest camera to the target at time, t τ , as the next camera without considering the target's direction of motion. As can be seen from the example in Figure.4, such approach will end up selecting the wrong camera, which is located on the other side of the target. So, first, we need to select candidate cameras which are spatially located in the direction of target's heading. Then, we select the closest camera among those based on their GPS distance from the The black doted lines shows the distance between camera a and the other two cameras. The green boxes and arrows respectively represent a target's tracklet and its heading. In this particular scenario, its evident that camera j should be selected as the next camera, since the target is heading towards it, even though camera m is closest camera to the target. And the estimated time is computed using the first case of eq. 1, as both camera j (next camera) and target have the same heading. target at time t τ . In this set up, we not only select the closest camera but also we ensure the next camera is located along the way of target's motion. Lets define a function H(GP S 1 , GP S 2 ), which returns heading angle between two GP S points. In particular, it determines which direction GP S 2 is located w.r.t GP S 1 and is computed as follows: where, X = sin(λ 2 − λ 1 ) * cos(ϕ 2 ) and Y = cos(ϕ 1 ) * sin(ϕ 2 )−sin(ϕ 1 ) * cos(ϕ 2 ) * cos(λ 2 −λ 1 ). (λ 1 , ϕ 1 ) and (λ 2 , ϕ 2 ) represent tuple of longitude and latitude of GP S 1 and GP S 2 , respectively.
For a camera, C m , to be in a candidate set, K, of target T a i , the following should hold: . That is, for a camera C m to be selected as a candidate, its heading w.r.t the location of current camera of the target, C a g t τ , is the same as target's heading T a h i . Finally, the next camera for target i in camera a is estimated by selecting the closest camera at time t τ among the candidate cameras, and is given by is GPS of target i of camera a at time t τ and due to very close proximity of the cameras to the targets, we can approximate target's GPS by the camera a s GPS at time t τ , C a g t τ . Estimating time of arrival: Next, we estimate the time required to travel between camera a (current camera) and j (the next camera). Since our cameras topology is very dynamic (due to camera motion), we implicitly compute the time as a division of distance and speed. The speed of a target, T a v i , is estimated to be 1.3 m/s, which is the average walking speed of individual [32]. While the speed of camera j, C j v , can be inferred from the sensor metadata data. The distance between the next camera, C j , and the target is computed between their GPS. Formally, the time required by target, T a i , to reach camera j from camera a can be computed as follows: where T a h i is the heading of target i at the time of exit, t τ , from camera a and is inferred from the heading of camera a at time t τ .
Note that in eq. 1, since both target and camera are moving, time computation depends on the direction of their motion. If both are moving in the same direction (i.e. same heading), first case in eq. 1, then we divide the distance by the difference of their speeds, otherwise, we estimate the time by dividing the distance with the sum of their speed.
Finally, we impose our time constraint on our appearance similarity. The final affinity between query target, T a i , and the rest of tracks in camera j is updated as follows: where ψ(ρ(T a i ), ρ(T j l )) is appearance similarity between two tracks features, T j l,t s is an entrance time of track T j l and |T j | is total number of tracks in camera j. As can be inferred from eq. 2, during ReID, we only compare track T a i with tracks in camera j which appear after the estimated time of arrival of track T a i at camera j, E T a i ,C j . Thus, we can significantly reduce our search space by pruning several wrong matching which violate our time constraint.

Datasets
In addition to the EgoReID, we present evaluation on widely used, fixed cameras dataset, MARS [52]. It consists of 6 cameras and 1261 different pedestrians. There are 625 identities for training and 636 identities for testing.
Evaluation settings: Training/Testing split of EgoReID dataset contains 567 identities for training and 309 identities for testing. For MARS dataset, we follow the same setting as the authors of the dataset. To evaluate performance for each algorithm, we report the Cumulative Matching Characteristic (CMC) metric and mean average precision (mAP).
Training the Network: We first train the semantic segmentation module of our proposed model using Look Into Person (LIP) [10] dataset. We then train our feature extractor module using the probability maps produced with segmentation module. Since only 3 semantic regions are used in the model, we group the probability maps of different regions to create the probability maps for foreground, upperbody and lower-body. During the training of the feature extractor module, following the same settings mentioned in [14], we use a training set consists of 10 image-based re-identification datasets. We use a training set, which is constructed by the aggregation of 10 image-based re-identification datasets. Next, we fine-tune the feature extractor by image-based training using the images from the video datasets (MARS or EgoReID). In the last step of the training, we freeze feature extractor and train 3D convolution layers using video clips of 15 frames. As mentioned in Section 4, there are three consecutive 3D convolution layers for each region. Each of these layers produce a tensor of size (30 × 30 × 2048), by encoding the temporal information with the kernel size of (1 × 1 × 3) and with the stride of 1 and 2 in spatial and temporal dimension, respectively. We construct the feature vectors from these tensors applying GAP and compute the loss for each region by performing multi-class classification separately. Finally, the loss of the model is obtained by summing the three losses. More implementation details are provided in the supplementary material.
Ablation study of our approach: We investigate the effect of each component of our model by conducting several experiments. In Table. 1, we show the results of each component in the proposed network. We evaluate the effects of each body regions and sensor meta data. As can be noted from Table. 1, lower body features perform worst on EgoReID, while they achieve reasonable result on MARS. This is mainly due to the fact that in EgoReID dataset lower body is frequently missing as the camera is very close to the target. We can also observe from Table. 1 that, jointly using features from different body regions, leads to improvements in the performance than using only one body region.
We can also see that the proposed approach for using sensor information to prune several unreliable matching pairs, significantly improves the ReID performance of our approach on EgoReID dataset. As shown in Table. 1, we are able to improve rank-1 and mAP by around 15% and 12%, respectively.  To further show the effectiveness of the proposed model, we compare our results with [14], where we apply their method on frame by frame bases and then employ average pooling over temporal dimension to generate features from each region. As can be seen from Table. 2, our approach gives significantly better results in both rank-1 and mAP (i.e. 20% and 34% respectively). This shows the effectiveness of the proposed 3Dconv module in encoding the temporal information.  Table 2: Comparison of SPReID [14] and the proposed approach on MARS dataset.

MARS
Comparison to state-of-the-art Methods: In Table. 3 and Table. 4, we compare our approach against the state-of-the-art methods on EgoReID and MARS, respectively. As can be observed from Table. 3, our approach significantly outperforms state-of-theart approache on EgoReID dataset. This shows the effectiveness of our human semantic region based local feature extraction approach on our dataset. This is mainly due to different body parts of pedestrians being clearly visible in EgoReID. In Table. 4, we observe that our method has competitive r-1 results, while it has better results in r-5 and mAP. This further demonstrate the robustness of our approach in handling different domains of video.

Conclusion
In this paper, we presented a new EgoReID dataset which is captured using 3 mobile cellphones with non-overlapping FOV. EgoReID presents substantial variations in lighting, scene, background, human pose, etc. Compared with existing video-based ReID datasets, EgoReID poses unique and realistic challenges to person ReID task. Moreover, we have recorded 12 sensor meta data for each video. We have also proposed a novel approach which employs human semantic parsing to extract local discriminative features and compare corresponding semantic regions to find a match. Experiments conducted on MARS and EgoReID showed the effectiveness of our approach in different video domains. In addition, We have also successfully employed sensor meta data information to determine target's next camera and its estimated time of arrival, thus, we only search for a target in the predicted camera around the estimated time of arrival. This significantly improved our ReID performance by reducing our search space. Fig. 5 shows sample tracklets from each camera. Each row contain tracklets of the same person, while each column represent different camera. As can be noted from fig. 5 , our dataset contains several illumination, pose and background changes both within and across cameras. Fig. 6 and 7 provide more detailed statistics on our dataset. For instance, camera 3 captures the most IDs and tracklets. Camera 2 captures the second most IDs but ranked third in total number of tracklets it contain.

C. Qualitative Results
Person re-identification: In Fig. 16 -17, given different example queries from EgoReID dataset, top 10 retrieved tracklets from the gallery are shown for different methods. In all examples, the first row is a result from Inception-v3 where we extract frame level global features and perform average pooling for each tracklet. The second row represent results where we apply [14] on every frame in a tracklet and apply average pooling along the temporal dimension. While the third and fourth rows show results of our method before and after employing sensor meta data information. Correctly matched tracklets are shown with green border while the incorrect ones are show in red. To avoid clutter, we only show three frames per tracklet.

D. Implementation Details
We train the feature extractor module for 200K iterations using the training set consists of 10 imagebased ReID datasets (Market-1501 [53], CUHK01 [21], CUHK02 [20], CUHK03 [22], DukeMTMC-reID [56], 3DPeS [1], PRID [13], PSDB [44], Shinpuhkan [15] and VIPeR [12]), then fine-tune it on the video datasets for 20K iterations. The initial learning rates for these two processes are 0.01 and 0.001, respectively, and we decay them 10 times with the rate of 0.9 by utilizing the exponential shift. Similarly, we use 0.001 as the learning rate while training the 3D convolution layers and decay it 10 times. While training the feature extractor, the batch size, momentum and weight decay are 15, 0.9 and 0.0005 for the respective values. To train the 3D layers, we set the batch size to 6 and the length of the clips to 15. The input images with the size of 512 × 170 are used to train both the feature extractor module and the 3D convolution layers.
We follow the same settings mentioned in [14] to train the semantic segmentation module. We set the initial learning rates to 0.01, 0.1 and 0.1 for the Inception-V3 backbone, atrous spatial pyramid pooling and the 1 × 1 convolution layer (classification layer), respectively, and decay them 10 times. The minibatch size is set to 8 and the input images with the size of 512 × 512 are used. The other settings are similar to the mentioned above. We perform the training of the re-identification and the segmentation models using Nesterov Accelarated Gradient [2] and used the pre-trained InceptionV3 models on ImageNet [33]. Figure 5: Sample tracklets from EgoReID. Each row corresponds to different identities, while each column represent tracklets from cameras 1, 2 and 3. (from left to right). As can be noted, in addition to changes in background, pose and illumination of the same tracklet across camera (each row), due to camera motion, different tracklets from the same camera (each column) are also captured with different background, illumination and poses. This makes our dataset challenging but more close the reality.