Memory Warps for Learning Long-Term Online Video Representations

This paper proposes a novel memory-based online video representation that is efficient, accurate and predictive. This is in contrast to prior works that often rely on computationally heavy 3D convolutions, ignore actual motion when aligning features over time, or operate in an off-line mode to utilize future frames. In particular, our memory (i) holds the feature representation, (ii) is spatially warped over time to compensate for observer and scene motions, (iii) can carry long-term information, and (iv) enables predicting feature representations in future frames. By exploring a variant that operates at multiple temporal scales, we efficiently learn across even longer time horizons. We apply our online framework to object detection in videos, obtaining a large 2.3 times speed-up and losing only 0.9% mAP on ImageNet-VID dataset, compared to prior works that even use future frames. Finally, we demonstrate the predictive property of our representation in two novel detection setups, where features are propagated over time to (i) significantly enhance a real-time detector by more than 10% mAP in a multi-threaded online setup and to (ii) anticipate objects in future frames.

Introduction
Motion is a crucial intermediary for human visual perception to learn about its environment and relate to it [1,2]. By encapsulating motion cues, video represents a rich medium for computer vision to understand and analyze the visual world. While the advent of convolutional neural networks (CNNs) has led to rapid improvements in learning spatial features, a persistent challenge remains to learn efficient representations that derive significant benefits from long-term temporal information in videos.
In this paper, we learn online video representations that incorporate multi-scale information on longer time horizons and design practical frameworks that achieve accuracy, efficiency and predictive power. Analogous to the Gestalt principle of common fate [3,4], we hypothesize that temporal coherence by accounting for motion across frames allows learning powerful representations while achieving greater invariance to blur, lighting, pose and occlusions. While our frameworks are The work was conducted as part of Tuan-Hung Vu's internship at NEC Labs America. We propose a multi-scale memory that efficiently aggregates image evidence over longer time horizons and also accounts for camera and object motion by feature warping, which enables learning better representations that lead to higher accuracy.
applicable to diverse problems, we demonstrate them with the specific example of object detection in videos.
In recent years, object detection in videos has attracted significant interest with benchmarks such as ImageNet VID [5] or Youtube-8M [6]. A popular approach has been to use detected bounding boxes computed independently for each frame using a strong CNN-based model [7,8,9,10] and to do temporal reasoning through tracking [11], re-scoring detections [12] and performing sequential non-maximum suppression [13]. While such methods improve over per-frame baselines, we explore the benefits of leveraging temporal information to learn better underlying feature representations. A few recent works temporally aggregate features to improve representation power [12,14], but use a fixed set of nearby frames and do not maintain causality or efficiency. In contrast, we propose a video representation that composes information across time in an online fashion (see Figure 1), which is not only faster, but also enables predictive applications.
In particular, Section 3.1 proposes a novel network structure, termed MemNet, that holds a memory of the feature representation, which is updated at every frame based on image observations solely from the past and warped from one frame to the next to account for observer and scene motions. We use a displacement field for warping similar to [14], but encoding memory allows retaining information from further in the past, while requiring only a single warp computation perframe. This is 2.3 times faster than [14], which requires as many warps as the number of temporally aggregated frames. 1 Further, in Section 3.2, we propose a hierarchical network structure, termed ClockNet due to similarity to [16,17], which extends MemNet by operating at multiple temporal scales. This allows efficiently leveraging information from even longer temporal horizons, which improves the representation power, as demonstrated by our experiments.
We apply our video representation to object detection in Section 4.1 and evaluate it on the ImageNet VID [5] data set in Section 5.1. Our proposed architectures improve over per-frame baselines by up to 2.2% in mean average precision (mAP). 2 We achieve state-of-the-art results close to flow-guided fea-ture aggregation (FGFA) [14] without having access to future frames and with considerably lower runtime, while outperforming a causal variant of FGFA.
A key benefit of the online nature of our video representation is that it imparts predictive abilities, which enables novel applications. First, in Section 4.2, we enhance the accuracy of an online real-time detector, by leveraging a stronger but less efficient detector in another thread. While the strong detector lags due to higher latency, our memory warping enables propagating and aligning its representation with the real-time detector, boosting the accuracy of the latter by more than 10% mAP, with no impact on speed or online operation (see Section 5.3). This is non-trivial, since parallelizing standard detectors in an online setup is not straightforward. Next, our predictive warping of video representations enables anticipating features in future frames, which allows solving visual tasks without actually observing future images. Sections 4.3 and 5.2 demonstrate this for the novel application of anticipating objects in future frames.
Finally, we note that our contributions are architecture-independent. The speed, accuracy and predictive benefits of our online representation are available for any detection method on video inputs.

Related Work
Learning representations for videos has been a long-standing goal in computer vision and many directions have been explored. Donahue et al. [20] or Srivastava et al. [21] rely on recurrent neural networks (RNNs) like LSTMs [22] to propagate feature representation from still frames over time. However, unlike our approach, features are propagated without explicit knowledge of motion in the scene, as we elaborate on in Section 3.3. 3D CNNs provide more freedom to learn motionspecific kernels (although motion is also not explicitly used) and were successfully used in [23,24] for tasks like video captioning or action recognition, but come with considerably more parameters to learn and typically more computational costs. Recent works have also considered unsupervised learning with pretext tasks [25], however, we focus on efficient learning of supervised representations.
Two-stream architectures like [26,27,28] combine features extracted from both images and motion (optical flow) to boost the representational power. While 3D CNNs learn motion-specific features implicitly, two-stream architectures explicitly take optical flow as input. In both cases, however, this information is not used to transform features over time to compensate for observer and scene motions.
The recently proposed flow-guided feature aggregation framework (FGFA) [29,14], on the other hand, explicitly warps convolutional feature maps between frames for better alignment when aggregating them. The warping function is triggered by a learned displacement field initialized with FlowNet [30]. However, FGFA [14] uses a fixed temporal window of nearby frames, requires as many warp computations as the length of the window, and compromises causality, i.e., integrates features from future frames. In contrast, we introduce a feature memory that is warped from one frame to another, saving computation time and allowing for a longer temporal horizon without looking into future frames. Our temporal multi-scale variant, ClockNet, is inspired by [16,17]. While sharing the idea of efficiently extending the temporal horizon, we also motivate and experimentally demonstrate the multi-scale aspect of this architecture. LSTMs [22] are also designed with a similar goal of a long-term representation for temporal data. The feature warping and propagation proposed in this work complements [16,17,22] with an effective way of dealing with spatial memory maps that describe real video data where observer and the scene itself can be in motion.
We showcase our ideas for object detection in videos, which has recently attracted lots of interest, partly due to the ImageNet VID challenge [5]. Besides [12,14] that work on the feature level, most recent approaches for detection in video leverage the temporal data on a higher level, e.g., by tracking objects or specifically designed post-processing mechanisms [31,11], which are orthogonal to our contributions. Besides object detection, the proposed feature can also be used for other tasks operating on videos like semantic segmentation [32,17] or action recognition [27,28,26]. Visual anticipation has also been shown for segmentation [33] and actions [34], but we differ in demonstrating for supervised object detection and in explicitly accounting for motion in the learned representation.

Feature propagation via memory networks
The goal of this paper is to improve the feature representation for objects in videos, by leveraging temporal information and motion. Exploiting past frames can also help predictions in the current frame when occlusions or motion blur distorts image evidence, cf. [14]. We propose to continuously aggregate and update features over time to provide a stable and powerful representation of the scene captured by the video, which is illustrated in Figure 2.

Aggregating features over time
Given a single image I ∈ R h I ×w I ×3 , a convolutional neural network (CNN) with parameters Θ F first extracts a feature map F ∈ R h F ×w F ×d F , where d F is the number of feature maps and we typically have h F = 1 16 h I and w F = 1 16 w I . In the following, we show how these single image feature representations are effectively aggregated over time. While we use a single feature map per image for ease of presentation, note that we can easily handle multiple feature maps at different resolutions to handle scale variations, which was shown to be useful in [7,35]. Tracking features over time: In every frame t, we hold a feature map M t ∈ R h F ×w F ×d F that acts as a memory on the feature representation of the video. Since the scene is dynamic and the camera is moving, the same objects will appear at different locations of the image plane in frames t − 1 and t. In order for the memory of the past frame M t−1 to benefit detection in the current frame t, M t−1 needs to be transformed according to the scene dynamics. Similar to [14], we use bilinear sampling to implement this transformation, where φ(·) is the bilinear sampling function and D (t,t−1) ∈ R h F ×w F ×2 is a displacement (or flow) field between frames t and t − 1, which is estimated by a CNN with parameters Θ D . This CNN is a pre-trained FlowNet [30], which takes images I t and I t−1 as input and predicts the displacement, but we fine-tune the parameters Θ D for the task at hand. Note that for fast computation of the displacement field, we feed FlowNet with half-resolution images and up-scale the displacement field. Also note that in the absence of ground truth data for the displacement field, this CNN predicts displacements suitable for the task at hand, which is demonstrated in Section 5.1.
Updating with image evidence: After having transformed the memory to the current frame t, i.e.,M t , we need to aggregate the newly available image evidence F t extracted by the feature CNN into the memory, which defines one step of the proposed MemNet. We implement (and experimentally evaluate) two variants of the aggregation function ψ(·). The first is a parameter-free combination that leads to exponential decay of memory over time, and the second is a weighted combination of memory and image features, with α M , α F ∈ R h F ×w F ×1 and α M + α F = 1. The weights are computed by a small CNN with parameters Ψ M and Ψ F operating on M and F , respectively, and the constraint α M + α F = 1 is always satisfied by sending the concatenated output of the CNNs through a per-pixel softmax function. The parameters of the weight-CNNs are automatically learned together with the rest of the network without any additional supervision. In the first frame t = 1, we simply assign the memory M 1 to be the feature representation of the image F 1 . Training MemNet: Training the proposed video representation requires a supervisory signal from a task module that is put on top of the memory features M . In general, the task module can be anything, even an unsupervised task like predicting future frames [36]. In this work we explore object detection in videos, where the supervisory signal comes from a combination of object localization and classification loss functions, see Section 4.1.
All parts of our representation can be trained end-to-end. Since bilinear sampling and the grid generation of the warping module are both differentiable [37], we can back-propagate gradients over time to previous frames, to the image feature extractor, as well as to the FlowNet generating the displacement fields.
While the network architecture in theory allows gradients to flow over the memory warping module to learn a good feature propagation, it also opens a shortcut for minimizing the loss because image evidence is available at every frame. While for some tasks past information is truly essential for prediction in the present, for several tasks the image of the current frame already provides most evidence for a good prediction (or at least a signal to minimize the loss). To encourage the network to learn a good feature propagation module, we randomly drop image evidence with probability 0.8 at frame t, which we found to improve results by a few percentage points.

Extending the temporal scale
The basic MemNet operates on just a single temporal scale, which has limited capability to leverage information at a larger temporal horizon. While, in theory, information from the whole video sequence is contained in the feature representation of the current frame t, this portion can be vanishingly small, particularly for the aggregation function relying on the exponential decay.
We thus propose to use a clock-work structure similar to [16,17] that operates on multiple temporal scales, which we denote ClockNet and illustrate in Figure 3. Formally, instead of having a single memory feature map, we have K memories M k t at frame t with k ∈ {1, . . . , K}, each of them operating at different rates. In our implementation, we update memory M k every 2 k−1 frames with new image evidence, although other schedules are also possible. Note that when K = 1, the basic MemNet is obtained.
In order to exchange information across the different time scales k, we aggregate all memory maps at a single frame t by simply averaging them, i.e., As with the feature map aggregation in the basic MemNet, different strategies for combining feature maps are possible. We chose the simpler parameter-free averaging, as a more complex learning-based weighting scheme did not show any performance gains. The aggregated memory M t can then be used as input to any task-specific modules.

Discussion
Our proposed video representations have a simple and intuitive structure, can be trained end-to-end and fulfill the basic requirements for a fast and causal system that can be applied to videos in any real-world application. In contrast to FGFA [14], the proposed model does not look at future frames and is also not limited to a specific temporal horizon in the past, rather can carry information from the whole (past) sequence in its memory. An even longer temporal horizon is utilized by the ClockNet architecture.
There also exists a relation to convolutional recurrent neural networks (cRNN) [38,39], however, with one crucial difference. While cRNNs keep their hidden memory fixed across spatial dimensions (h t = RNN(h t−1 , x t )), our model enables the memory to be spatially aligned with observer and scene motion in the actual video content (h t = RNN(warp(h t−1 , D t,t−1 ), x t )), see Figure 4. While our aggregation function ψ(·) for new input and previous hidden states is simple, we did not observe any improvements for our particular applications with more complex architectures like LSTM [22] or GRU [40].

Detection, Propagation and Anticipation in Videos
To demonstrate the benefits of propagating features over time with the proposed MemNet and ClockNet, we show its impact for three practical applications.

Object detection in videos
While we can use the proposed memory features M from Section 3 for any downstream task, in this paper, we focus on object detection in videos. Modern object detectors like Faster-RCNN [10], R-FCN [15], SSD [9] or RetinaNet [8] all have a similar high-level structure in the sense that they all rely on a convolutional neural network to extract features F from a single image. The detection-specific modules applied on top of F define the differences between the detectors, e.g., proposal-based [10,15] or proposal-free [9,8], making F an interface between one generic module and detection-specific modules. Our proposed MemNet and ClockNet operate on F and compute a novel feature representation M , making our video representation applicable to all of these detectors. In this paper, we pick R-FCN [15] because it has shown a good trade-off between accuracy and speed, with a publicly available code base.
Given a representation M t of a video sequence at frame t, the object detector first computes object proposals with a region proposal network (RPN) as proposed in [10]. Object proposals define potential locations of objects of interest (independent of the actual category) and reduce the search space for the final classification stage. Each proposal is then classified into one of C categories and the corresponding proposal location is further refined. In contrast to Faster-RCNN [10], the per-proposal computation costs in R-FCN are minimal by using position-sensitive ROI pooling. This special type of ROI pooling is applied on the output of the region classification network (RCN).

Real-time detection by propagating strong features
Assuming an input stream capturing images at 20 frames-per-second (FPS), we ideally want an object detector that can process one image in less than 50 ms to avoid latency in the output. One easy option to speed-up a modern object detector is to use a more light-weight feature extraction CNN, e.g., by using ResNet-18 instead of ResNet-101. Note that this is a viable option for any detection framework, e.g., Faster-RCNN [10], R-FCN [15], YOLO [41,42] or SSD [9]. However, accuracy will decrease. Here, we explore another option to speed-up a modern object detector. Instead of using a single model, we demonstrate how to exploit two models with complementary properties running simultaneously (but asynchronously) on two threads (two GPUs) to achieve both speed and accuracy, using our feature propagation.
We run a fast detector, R-FCN-18 (i.e., R-FCN with ResNet-18 [43]) in one thread and a slower but also stronger detector, ClockNet-101, in the other thread. R-FCN-18 runs at the required frame rate and can provide output for every frame, however at a lower quality than ClockNet-101 could do if no time requirements existed. The main problem with the strong object detector is that it will always have some delay (or latency) ∆ to produce an output. If ∆ is too large for a practical system, the strong detector is not usable. It is important to note that achieving a speed-up with two GPUs is not trivial in a real-time setting. For the offline case it is easy to distribute computation of different images on multiple GPUs. However, this is not an option for streaming data. In Section 5.3, we still empirically compare with two alternative baselines that also leverage two GPUs.
With our design, on the other hand, we can still leverage the strong features by making up for the delay via feature propagation. We compute the displacement field between frame t + ∆ and t and warp the strong features M 101 t into the current frame t + ∆, where the fast object detector has already computed features F 18 t+∆ , see Figure 5. We boost the representational power of R-FCN-18 by combining the feature maps. Again, we take the average of both features (the dimensionality is the same), but more advanced aggregation schemes are possible. We experimentally evaluate this application in Section 5.3.

Anticipating features
Another application of the proposed feature propagation is future prediction or anticipation. Features from the current frame t are propagated to a future frame t + ∆, where the task network is applied to make predictions. In the previous application of Section 4.2, we utilize feature propagation over several frames, but the displacement fields are still computed from image evidence, similar to [29]. For a true visual anticipation, however, future images are not available.
We propose to extrapolate the displacement fields into future frames and use them to propagate the feature (or memory) maps. For demonstration, we use a simple extrapolation technique. Given the two previous displacement fields D t−1,t−2 and D t,t−1 , we compute the difference of aligned displacement vectors (with bilinear sampling), which gives us the acceleration of pixels. We then employ a simple constant acceleration motion model to each displacement vector and extrapolate for one or multiple frames. Obviously, this extrapolation technique has limitations but it is sufficient for our demonstrations of feature anticipation. We analyze the quality of the anticipated features in Section 5.2 by measuring the object detection quality in future frames.

Experiments
Our experimental evaluation focuses on the performance of our feature propagation and aggregation methods for object detection in videos. In Sections 5.1, 5.2 and 5.3, we evaluate the performance of the proposed MemNet and ClockNet on the three applications introduced in Section 4, respectively. Dataset: All our experiments are conducted on the ImageNet VID data set [5], which is most suitable for object detection in videos and has been the testbed for most recent approaches for this task [12,11,29,14]. ImageNet VID is a large scale data set consisting of 5344 video clips, captured at frames rates between 25 and 30 FPS and divided into training, validation and testing sets with 3862, 555 and 937 clips respectively. Each clip is fully annotated with bounding box tracks of 30 different object classes. Implementation details: We use the ResNet-101 architecture [43] as the basic feature extractor in all experiments with the exception of some ablation studies. In particular, we useà trous convolutions as in [44,14] to increase the feature resolution. Similar to [14] the extracted features are passed through a 3×3 convolutional layer and a non-linear activation (ReLU [45]) before we provide them as inputs to MemNet and ClockNet. To estimate displacement fields we use FlowNet [30]. All the parameters are jointly fine-tuned end-to-end. In general, we closely follow the experimental setup of [14] using their publicly available MXNet [46] implementation. All models are trained for 2 epochs with an initial learning rate of 0.001, which is decreased by a factor of 10 after 4 3 epochs. We train our models on the same mix of ImageNet DET and ImageNet VID training sets as in [14]. All experiments, including runtime measurement, are done on an NVIDIA TITAN Xp. For training, we use 4 GPUs.

Object detection in videos
Object detection in videos aims at localizing objects in every video frame, i.e., estimating bounding boxes around objects associated with a confidence score. Evaluation metrics: We measure detection performance as mean average precision (mAP) over all object classes, where we additionally differentiate between fast, medium and slowly moving objects using the subsets of videos introduced in [14]. We also measure the average runtime per frame in milliseconds (ms) for each model (using the same framework and GPU setup). Baselines: We compare the proposed MemNet and ClockNet models with several baselines. The first one is the per-frame object detector itself (R-FCN [15,14]) that does not exploit temporal information. The second baseline is FGFA [14], which, for every frame, aggregates features from nearby frames both in the past and the future. Obviously, this makes FGFA a non-causal system not applicable to real-time tasks. Note that these two baselines represent two extremes of using temporal information, with R-FCN not exploiting the video at all and FGFA looking not only into the past but also into future frames. For a more fair comparison, we thus created a causal variant of FGFA (Cau. FGFA) that aggregates information only from nearby features in the past but not from future frames. While FGFA can only operate in the off-line setting where future frames are accessible, causal FGFA is an on-line detector, making it the most comparable baseline to our proposed models. Main results: Table 1 summarizes our quantitative results and Figures 7a-7b give qualitative examples. We can first see that all models leveraging temporal data improve over the per-frame baseline R-FCN [15]. Looking into both past and future frames, as FGFA [14] does, gives the best overall results, but comes at a considerable runtime cost and, more importantly, is a non-causal system. Among all causal systems leveraging data only from the past (eight bottom models in Table 1), the proposed ClockNet gives the best results overall and is particularly strong for fast moving objects. Its mAP value is only 0.6 percentage points behind FGFA without having access to future frames.  Looking at the running times, we see that the proposed MemNet is clearly the fastest, except for the still-frame baseline, and ClockNet already ranks second. Both proposed models are faster than causal FGFA and consequently, FGFA [14]. Built upon the detection framework of R-FCN, all those models have the same computational complexity for feature extraction, proposals generation and classification. Therefore, the causes of speed difference are the numbers of flow computations N F and feature warps N W . For FGFA and Causal FGFA, N F and N W equal the number of frames within aggregation range, i.e. 20 and 10, respectively. The ClockNet reported in Table 1 requires N F = 3 and N W = 3, corresponding to its 3 temporal scales, and MemNet has a speed advantage because it only needs N F = 1 and N W = 1 to process an incoming frame.
We also note that the computation time of Causal FGFA can be reduced by aggregating displacement fields in an online manner, see Cau. FGFA* in Table 1, thus reducing N F to 1 as in MemNet. While the runtime is reduced, the error accumulation in online aggregation of displacement fields leads to a significant accuracy drop. To highlight the reduction in runtime from MemNet and ClockNet, we show another trivial way to speed-up FGFA by halving the image resolution, FGFA (half). However, this variant also leads to a large performance drop.
Overall, accuracy and runtime demonstrate the advantages of our memory propagation mechanism for object detection in videos, see Figure 6. Moreover, our memory-based architectures have the additional benefit (over FGFA) of being able to propagate features into future frames, which we analyze in Section 5.2 and demonstrate in a practical application in Section 5.3. Ablation studies: First, we investigate the importance of the length of the sequences used for training MemNet. In the testing phase there is no limitation on the sequence length, but GPU memory is a hard constraint during training as gradients need to flow back through the whole sequence. We can observe in Table 1 that a longer temporal window for training MemNet (MemNet-6 vs. MemNet-3) is indeed beneficial. Second, by comparing different aggregation schemes (Equations (3) and (4)), we can observe that learning an additional weighting gives a small improvement with only a tiny increase in computational costs. We also tried LSTM [22] as aggregation scheme, but did not observe any improvements. Finally, we analyze the importance of multiple temporal scales in ClockNet, which operates on three time axes k = [1,3,4], corresponding to temporal strides of 1, 4 and 8, respectively. To emphasize the benefit of feature aggregation across multiple temporal scales, we compare ClockNet to MemNet trained with longer strides (MemNet-6-strd-4 and MemNet-6-strd-8), given it access to larger temporal horizon during training. We can see from Table 1 that both baselines perform worse than MemNet-6 (with temporal stride of 1), which illustrates the importance of the information provided by actual neighboring frames. Therefore, the success of ClockNet provides a signal for the benefits of a temporal multi-scale architecture, where different temporal scales are complementary to each other. More experiments on various details of our methods can be found in the supplemental material. Fine-tuning FlowNet: Finally, we want to understand the effect of fine-tuning FlowNet during training the video representation for the detection task. Similar to [14], we observe a performance drop in mAP if FlowNet is not fine-tuned. Given that displacement fields after fine-tuning are apparently better for the detection task, we visualize the difference for one example in Figure 7c. One difference that we can observe is that the object tends to move as a whole and ignores the motion of individual parts.

Propagating and anticipating features
In the previous experiment, the appearance feature of the current frame is always available and the main purpose was to analyze the influence of additional information from the past. In this experiment, we want to give more insights into the quality of the feature propagation and anticipation performance, which has several applications as discussed in Section 4. Propagation: In this experiment, we provide image information up to frame t − δ and then propagate up to frame t. Note that displacement fields are still available but only for warping the memory; no image evidence in form of appearance features is available to the detector after frame t − δ. The propagated features are then used to compute the detections. We compare our model with a baseline that takes the detected bounding boxes at frame t − δ and propagates them with the computed displacement fields to frame t. The mean displacement inside a bounding box serves as the translational vector. Table 2 shows the impact of skipping the image features for different amounts of frames for MemNet, ClockNet and the box-propagating baselines. In general, all models gracefully degrade performance with larger δ, but at the same time reduce running time. It is evident that feature propagation outperforms propagation on the bounding box level, particularly for ClockNet. From Figure 7d, we can see that the performance gap between MemNet and ClockNet increases when δ gets larger, demonstrating the impact of multiple temporal scales and the extend time horizon of ClockNet. Anticipation: We next evaluate feature anticipation as discussed in Section 4.3, which differs to the previous experiment since no image information is available at all for frames after t − δ. This requires us to extrapolate the displacement field as   Table 3: Accuracy and runtime of our twothreaded detection setup. The number of GPUs utilized is denoted as #G.
described in Section 4.3. Comparing the anticipation results with the propagation model in Table 2, we can see that our flow extrapolation strategy works well for MemNet which has short temporal stride, although the runtime speed drops due to our current non-optimized implementation of flow extrapolation. Reversely, on ClockNet memories, feature anticipation performs worse than feature propagation. One explanation is that quality of the extrapolated flows is heavily degraded with the long temporal strides of ClockNet.

Fast detection by propagating strong features
In this section we analyze the application described in Section 4.2 and Figure 5. A fast but weak object detector is running in the main thread providing detection output at every frame, while leveraging features from a strong but slow feature extractor. In order to use the strong features, they have to be propagated to the current frame, i.e., aligned over time, to compensate for the delay ∆.
We use R-FCN based on the ResNet-18 architecture as the fast main-thread detector, which runs at 14.3 FPS. The helper thread runs ClockNet based on ResNet-101 (same as in Table 1), which is able to propagate features over time as demonstrated in Section 5.2. In practice, ClockNet-101 runs 2.4 times slower than R-FCN-18. To compensate for additional overhead (e.g., feature propagation), we update ClockNet-101 with image evidence once every 4 frames ("ClockNet-101-FeatProp"). For training the fast detector, i.e., R-FCN-18, we only leverage fixed features coming from the second thread but do not fine-tune ClockNet-101.
Quantitative results: Table 3 shows the results of this experiment. As expected, R-FCN-18 is the fastest model but also gives the worst accuracy. ClockNet-101 is the model shown in Section 5.1 and can be considered an upper bound in terms of accuracy but is very slow compared to R-FCN-18. ClockNet-101-FeatProp is the model running in the helper thread which receives image evidence every 4 frames and uses propagation to make predictions in other frames. The performance drop compared to the upper bound is not much, as also seen in Section 5.2, but the runtime is still high. However, when following the design proposed in Section 4.2, i.e., feeding R-FCN-18 with propagated strong features from ClockNet-101-FeatProp, we observe a significant 10% mAP boost compared to R-FCN-18 which is even close to the ClockNet-101 upper bound, while maintaining the low runtime of R-FCN-18.
Comparison to two-thread baselines: Recall that achieving speed-up with two GPUs in a real-time (data-streaming) setting is non-trivial as parallelizing frames over multiple GPUs is not possible. We still evaluate alternative baselines that leverage two GPUs. First, a trivial two times speed-up can be achieved by splitting the image into two halves, running the detector individually and merging the detections before NMS, denoted "(split)" in Table 3. While this gives the expected speed-up, the performance drop is significant, which can be explained by the fact that objects in Imagenet-VID are mostly centered, thus effectively truncating them. The second baseline is more evolved and similar to our proposed design. However, instead of propagating features, (delayed) detections from the strong model are propagated to align with the faster detector ("BoxProp"), as in Section 5.2. As in the previous experiment, we observe that feature-level propagation is superior to propagating bounding boxes.

Conclusions
Our work proposes a long-term online video representation that effectively leverages past information. We rely on a memory that is updated regularly with image evidence and is warped over time for a proper spatio-temporal alignment of features. We also extend this representation to multiple temporal scales, thus aggregating information from even farther in the past. Our experiments illustrate the benefits of each component, as well as a demonstration of two practical applications: feature anticipation and a real-time multi-threaded object detector. For future works, we plan to investigate the effectiveness of our learned video representation for action recognition and segmentation in videos. We also plan to further explore the utility of feature anticipation and postulate that causally learned representations like ours have inherent advantages for visual anticipation. The supplemental material contains the following items: -Section 1: Detailed per-category results of our main evaluation -Section 2: Additional ablation studies of the proposed MemNet -Section 3: Additional qualitative results of the proposed video detectors -Section 4: Additional details on the feature anticipation experiment 1 Per-category results on ImageNet-VID data set The main results in Table 1 of the main paper show the mean of the average precision value (AP) across all 30 categories of the ImageNet-VID data set. In Table 1, we provide the per-category results for the main methods: R-FCN [15], FGFA [14], Causal FGFA, MemNet-6 and ClockNet. To better illustrate the performance gain of leveraging temporal data, we show the per-category AP gain of ClockNet over R-FCN [15] in Figure 1, where each sub-figure shows different subsets of the data set corresponding to the columns of Table 1 in the main paper: "all" corresponds to all frames of the data set. "fast, medium, slow" correspond to subsets of the frames where objects move fast, normal (medium fast) and slow. These subsets have been defined in [14].

Additional results of ablation studies
This section provides (i) a comparison between different aggregation schemes for MemNet, (ii) an analysis of the underlying feature representation our memory module is applied to and (iii) additional qualitative examples of the learned displacement fields. Note that we conduct ablation studies of MemNet with only half resolution images for computational reasons, which explains lower accuracies for both, baselines and our proposed methods. First, we complement the evaluation of different aggregation schemes between memory and features from new image evidence in Section 5.1 of the main paper. Table 2a compares simple averaging and the weighting described in the main paper with adaptive weighting from [14] and the recently proposed AdaScan [47]. As already discussed in the main paper, it is interesting to see that simple averaging gives on par results with more complex weighting schemes.
Second, we apply the proposed MemNet on different levels of a ResNet-50 feature extractor: "conv3", "conv4", "conv4+conv5", and "conv5". For this  Table 1: Per-category AP (%) on ImageNet VID validation set. particular application of object detection in videos, one can observe from Table 2b that "conv5" gives the best results overall. Other applications may require different setups.
Finally, we also provide more examples of the learned displacement fields in Figure 2. As in the example in the main paper, we can see that after finetuning FlowNet, objects tend to move as a whole, while different motion of individual parts is suppressed. Moreover, we also see some examples where the initial FlowNet only provides motion for parts of an objects but after fine-tuning the whole object is covered, thus refining the flow.  3 Qualitative results Figure 3 illustrates additional qualitative results of our memory networks and the RFCN baseline [15], as in Figure 7(a-b) of the main paper.

Feature anticipation
In this section, we complement the feature anticipation experiment conducted in Section 5.3 of the main paper. Due to space limitations, we only showed a plot Average Weighting Adaptive Weighting [14] AdaScan [47]   for feature propagation (Figure 7d) but not for anticipation. Here we provide the same figure for feature anticipation, see Figure 4. We can observe that ClockNet is more efficient than MemNet in anticipating features over long distances, which is similar to our observations for feature propagation.


Real-Time 3D Shape of Micro-Details

Motivated by the growing demand for interactive environments, we propose an accurate real-time 3D shape reconstruction technique. To provide a reliable 3D reconstruction which is still a challenging task when dealing with real-world applications, we integrate several components including (i) Photometric Stereo (PS), (ii) perspective Cook-Torrance reflectance model that enables PS to deal with a broad range of possible real-world object reflections, (iii) realistic lightening situation, (iv) a Recurrent Optimization Network (RON) and finally (v) heuristic Dijkstra Gaussian Mean Curvature (DGMC) initialization approach. We demonstrate the potential benefits of our hybrid model by providing 3D shape with highly-detailed information from micro-prints for the first time. All real-world images are taken by a mobile phone camera under a simple setup as a consumer-level equipment. In addition, complementary synthetic experiments confirm the beneficial properties of our novel method and its superiority over the state-of-the-art approaches.

Introduction
3D reconstruction can be considered as one of the major areas for computer vision which provides not only 3D interpretation, but also the surface feature extraction. Photometric Stereo (PS) is a potential technique for this purpose. In PS, several input images taken from a fixed view point under different illumination directions are used for 3D reconstruction. The pioneers of the problem are Woodham [66] and Horn et al. [21]. As it is shown by Woodham [67], the orientation of a Lambertian surface can be uniquely determined when at least three input images are given. During last decades, PS has attracted wide attention in different practical domains such as industrial quality control [11,74,42,25], face recognition [72,75,14,73], Figure 1. A successful 3D reconstruction from very fine details of micro-prints on a 50 euro banknote shown in gray color rectangular parts of (A) and (B). As can be seen, these details are even invisible to human eyes, while our real-time 3D reconstruction from these regions is able to reveal hidden information and features in high amount of details. This information recovery can be used in various applied areas such as detecting security items on financial documents for the fraud detection and also the quality control of any industrial productions that include delicate details such as printed circuits. medical science [44,63], texture classification [51,57,34] and estimation of weather conditions [55]. Another benefit of PS is to generate invariant features utilized in different kinds of recognition tasks [61,58]. However, most of the later works on PS kept two basic building blocks of Woodham's approach, namely Lambertian surface reflection and orthographic camera projection. For this reason, Lambert's law [29] is commonly used in classic computer vision models, which implies that the surface reflects light equally in all directions and ignores the specular component. However, it may easily fails for surfaces whose reflection is non-Lambertian. Corresponding effects (e.g. specularities as occurring on shiny objects) are quite readily observable in real-world images and surfaces with not pure isotropic luminance. So, two major light reflection components in-cluding specular reflection as the high frequency variations and diffuse reflection as the low frequency component should be combined to provide a successful reflectance model [47,40,65]. In order to circumvent difficulties of specularities, commercial instruments for 3D reconstruction use sophisticated techniques such as white light interferometry or scanning focal microscopy, while these laboratory-based devices tend to be large, slow, or at the expense of $100,000 or more. In many works, non-Lambertian effects are considered as outliers. In this context, Miyazaki et al. [38] suggested a median filtering to evade the influence of specular reflection and a preprocessing technique is presented by Yang et al. [70] to remove specularities. A maximum-likelihood estimation [64] and an expectation maximization [69] are other approaches based on assuming specularities as the outliers. While Mukaigawa et al. [39] suggested a random sample consensus based approach in which only diffuse reflection is employed as the underlying model, Tang et al. [60] proposed a coupled Markov random field treating specularities as noise. An approach relying on a maximum feasible subsystem method is introduced by Yu et al. [72], where input images satisfying a Lambertian constraint are selected for reconstruction. In addition, a convex optimization technique is developed by Wu et al. [68] in which specularities are considered as deviations from the Lambertian assumption in the objective function. Zickler et al. [23] provides an analyze to a subspace which is free from highlights for image transformation. Finally, Ikehata et al. [23] suggested a regression procedure that deals with the specular component as a sparse error of an underlying purely diffuse reflectance equation. Mallick et al. [32] considered a transformation of the RGB color space to avoid specularity using the light source color which is not always available. An inpainting technique for the highlights correction is presented by Tan et al. [49] using the color of illuminant and the diffuse component. In [59] and [54] the pixel chromaticity direct analysis and specular-free image are used respectively to specularity correction in which the diffuse component is adjusted to a criterion. This criterion is an indicative of smooth transition of color between diffuse and specular regions and evaluated using a nonlinear shifting of specular pixels intensity and chromaticity. Johnson et al. [24] presented an elastomer using a sensor skin constructed with a metal-flake pigment which is attached to the surface to change its BRDF. However, in all these works, the shape information that is evidently contained in non-Lambertian effects (e.g. highlights) is discarded. The orthographic projection as the second mentioned ingredient is realistic when objects are far away from the camera, but not if they are close since then perspective effects grow to be important. In addition, when a surface is projected orthographically, it can be reconstructed only up to a generalized bas-relief trans-formation. However, the perspective projection as proved in [43], can remove the bas-relief ambiguity and provide the uniqueness of the reconstructed surface. There is only a small number of previous papers dealing with perspective models in PS. To our knowledge, the first of such models has been developed by Lee and Kuo [30]. They approximated the depth function as a linear combination of individual element basis functions. Other perspective methods are applied by Galo and Tozzi [13], Tankus and Kiryati [62] and Mecca et al. [37], while the Lambertian reflectance is applied in all these researches. Concerning more general non Lambertian reflectance models based on perspective PS, Khanian et al. proposed the application of perspective Blinn-Phong model [5,46] for the signal reconstruction and corresponding lightening conditions in [26] and for the 3D reconstruction with different perspective techniques in [27]. An empirical approach is also considered in [36,35]. However, for the derivation of the final applied model, purely diffuse and purely specular reflections should be decoupled in [36,35] . Moreover, the proposed approach in [36] relies on the Fast Marching method [52] where the unknown depth value of the center point should be provided in advance which makes difficulty for real-world applications. Some state-of-the-art in PS methods in which the information of depth should be provided in advance, can be pointed as [7,50] which require the depth to be known on the boundary, and [35] that needs the mean value of depth. In contrast, we do not need any information of depth to be given in advance in our approach as also proved in Figure  1.

Our contributions
In this paper, we provide the following innovations: 1. We proposed an accurate hybrid real-time 3D reconstruction approach which merges the benefits of perspective projection and non-Lambertian reflectance model motivated by Cook-Torrance reflectance [8] to handle complex reflecting characteristics. To the best of our knowledge, this is the first work for the real-time PS which also regarded the application of perspective Cook-Torrance model.

Introducing a Recurrent Optimization Network (RON)
which provides a robust PS framework to obtain a highlydetailed 3D shape from even micro-print texts where their visibility is hard for the human visual system as can be seen in Figure 1. In this Figure, we could reconstruct 3D shape of two regions of A and B with high details, while these regions are very tiny microtexts. The information hidden in these areas became completely clear by our approach. As a result, our technique can be used in different applied areas. 3. Since the performance of many optimization techniques is highly-dependent on the initialization process, we provided a Dijkstra Gaussian Mean Curvature (DGMC) technique to find anchor points which can be applied as a Figure 2. A complex real scene including several objects with different sizes captured by a mobile phone with a simple setup which is used for our 3D shape reconstructions. key point to offer more effective optimization approach. 4. Furthermore, we extended our proposed approach to deal with realistic lightening situation using spatially dependent lighting. 5. We show that our approach is able to provide the realtime 3D shape of objects as captured by an ordinary mobile phone camera without employing controlled laboratory conditions (cf. Figure 2). So, our PS process is conducted by the consumer-level equipment where specialized equipment and professional laboratory settings are not required. Our instruments are easily available for general consumers as should be expected for an ideal solution. Consequently, our method can be readily employed in real-world situations. We demonstrate that this paper provides the basis for a potentially useful 3D reconstruction technique. This improvement breaks inhibitor limitations of a controlled setup and the necessity of working with specific scenes (e.g. Lambertian materials) and turns mobile phone into powerful 3D shape reconstruction tools. 6. Finally, our approach also provides non-uniform colorful albedo from images with diverse color intensities as shown in Figure 12. All these attempts greatly advance the applicability of photometric stereo to various applications, especially for reconstructing 3D surfaces from very tiny structures like microprints.

Cook-Torrance reflectance model
The Cook-Torrance reflectance as a physically based model designed to account for the spectral composition of specularity and cope with the problem of modeling reflections of realistic materials [8]. This physically based model agrees well with actual measured BRDFs [28]. Like recent popular theoretical models, this reflectance model considers this assumption that the large scale BRDF is a collection of microfacets as shiny V-grooves with random sizes and orientations [28]. This hypothesis leads to the microfacet theory for non-perfectly-smooth surfaces. These surfaces are essentially height fields, where the distribution of facet is described statistically. So, the main component of these models is an expression for the distribution of microfacet normals. This property enables them to effectively model many real-world surfaces [45] in which, mirror-like facets are modeled by the Cook-Torrance reflectance to retrieve the local shape from specularity. The Cook-Torrance image irradiance equation is defined as follows: and In this notation, I(x, y) and N (x, y) are the intensity and surface normal at pixel (x, y), respectively. The parameter k d is the diffuse material parameter (albedo) and m is the root mean square slope of the surface microfacets (i.e. the surface roughness).
It is worth to mention that in expression (8), H is used instead of N as suggested in [18]. Let us emphasize that as opposed to an orthogonal viewer direction (0, 0, 1) which is commonly used, we model here a pinhole camera. Therefore, the viewing vector V depends explicitly on (x, y) and also on the depth z. This model is powerful enough to capture the geometrical properties of the scene as recorded by the camera. Moreover, its images resemble what would be seen by human visual system. Hence, pinhole camera is also referred to as perspective camera model in graphics and vision literatures [71].
The vector H is defined to be the halfway vector between the viewing vector V and the light vector L, as the direction to which the microfacet normals need to be oriented to reflect L into V . The light source intensity is denoted by l d . F is the Fresnel term including f λ which is the reflection coefficient (the reflectance of the surface at normal incidence). Here, the distribution of facets is described by D. It should be noticed that in PS, it is necessary to apply a set of non-coplanar light vectors L i (corresponding to the input images I i ) as also investigated in [67,20,19,9]. It is worthwhile noting that in all our experiments (including synthetic and real-world scenes and even for specular images), we will still follow Woodham [66] and use only three input images I i , i = 1, 2, 3. We consider it an indication of the robustness and potential practical usefulness of our method that it works effectively with such a low number of input images. Considering (1) for multiple images leads to a highly non-linear system of equations for the unknown surface normal vectors that should be solved.

Perspective projection and intrinsic parameters of pinhole camera model
Under the perspective projection model, the surface parametrization is performed as follows: The normal vector as the cross product of the partial derivatives of the surface is computed as: where f is the focal length and ∇z(x, y) = (z x , z y ) is the gradient field. On the other hand, to implement the perspective projection of the pinhole camera model, the intrinsic camera parameters should be involved which leads to the CCD camera [17]. To this aim, we apply the following transformation to convert the homogeneous representations pixel coordinate X = (x, y, 1) to image coordinate χ = (µ, ν, f ): where Γ is the matrix of intrinsic parameters of the camera expressed as: Here, ψ x and ψ y are the focal length in x and y directions. ξ describes the skew effect and (δ x , δ y ) denotes the principal point or focal point. We neglect the skew parameter (ξ = 0) as it is zero for most of normal cameras [17].

Lightening sensitivity analysis
Different types of light sources can be defined based on the energy emission from a single point in space with some angularly varying distribution of outgoing lights. At first, we considered the distant light, also known as the directional light or sparse light. This source of energy is an emitter which deposits illumination from the same direction at every point in space. It can be considered also as a light at infinity (e.g. sun). Furthermore, in this kind of light sources, energy strength does not depend on the distance between the light source and the object [45]. In addition, we also equipped our perspective Cook-Torrance model with spatially varying lightening in which the point-wise light source implies how the light varies from one point to another. This assumption facilitates the independent control of intensity. To this aim, a three dimensional vector field of the light L should be imposed instead of a directional light vector. In this case, the rate at which the light falls off w.r.t. the distance should be induced to make the model more consistent with the actual physical aspects. Integrating all the mentioned points, the new spatially dependent light vector L is defined as: with the light attenuation Π: where Υ is the light source position. Since we developed the perspective projection, the surface parametrization as in (9) is imposed in S(x, y) for spatially point light source. As a result, the light direction depends not only on the point (x,y), but also on the depth of the surface itself. So, in this case, an initial guess for the depth of the surface is required.

Optimization process
Using equation (1) for three input images (as the minimum number of input images in PS), we constitute a system of equation , h = 1, 2, 3 and formed by equations given by (1) corresponding to each input image I h (x, y), h = 1, 2, 3 as: To estimate the normal field, we should solve the optimization problem as follows: In optimization process, we update the solution using an iterative procedure: Here, d k is the search direction in iteration k. The main difference between various optimization approaches is caused by the definition of d k .

Quasi-Newton with BFGS updating
In this section, we investigate various procedures that we applied for the nonlinear system of equations described by F (x) = 0. Among many iterative methods, Newton's method is one of the popular approaches. But this method has some drawbacks which can be removed by its modified versions. Here, we used the updated formula known as Quasi-Newton with BFGS suggested independently by [6,12,15,53], which as explained in [48], has much better performance than other modified versions of Newton's method. In this approach, the Quasi-Newton condition (secant condition) is used to find the search direction d k : where B and J(X) are the Hessian matrix approximation and the Jacobian matrix of F (X), respectively. The following expression is known as Quasi-Newton condition or the BFGS formula for updating B: with

Levenberg-Marquardt
Another strategy that we considered is the Levenberg-Marquardt method [31,33] with the search direction determined as: where λ k > 0 and the (J(X k )J(X k ) T +λ k I) −1 is positive definite.

Powell's Dog Leg
In this technique, a strategy is proposed to choose optimization steps based on a parameter called radius of trust region ∆. Three search directions are applied as follows: where Here, we have also used the strategy applied in [41] to choose β and ∆. The general process of mentioned algorithms is illustrated in the last section.

Dijkstra Gaussian Mean Curvature (DGMC) technique
As a heuristic search to find a proper initial value for the normal field, we have devised a technique that consists of two phases. At the first stage, we employ the Dijkstra algorithm [10] to find the nearest point in the outer highlight boundary S to each specular pixel p . This point is denoted as q.
In the second step, we compute the Gaussian Curvature (GC) and the Mean Curvature (MC) properties of p and q, separately: where K 1 is the smallest and K 2 is the biggest eigenvalues of the Hessian matrix of the pixel local neighboring area.
Having those parameter for both points allows us to define their local geometric similarity conditions as follows: In case that q does not meet similarity conditions, the next nearest member of S to p will be nominated.

Recurrent Optimization Network (RON)
We will introduce an intermittent optimization strategy which allows updating not only the surface normal but also the albedo and roughness parameter. At first, we consider a constant albedo and also a constant roughness so that we can obtain the surface normal. After obtaining the normal field, we update the albedo values. Furthermore, the roughness parameter is updated as well. To this aim, we use the following formulation proposed in [16] to update the roughness parameter m, where we already obtained the depth map z by integrating the normal field.
Here,z(x, y) is the mean surface depth, and n is the number of pixels. The whole algorithm of mentioned RON procedure is illustrated in Algorithm 1. Thanks to this concise optimization network with backtracing steps and DGMC technique which are integrated into the photometric stereo algorithm, our method enables PS to work in diverse applications by removing restrictions such as diffuse objects, orthographic projection, laboratory set up and professional equipment.

Integration
As the integrator of the normal field, the energy minimization of the following functional should be considered: which leads to the Euiler-Lagrange equation ∆z = ∇.N . In order to solve the mentioned minimization problem, we applied Generalized Minimal RESidual (GMRES) with the initial solution of Simchony [56].

Experiments
In the case of synthetic experiments, to perform a quantitative evaluation, we will make use of Mean Angular Error of Normal vectors (MAEN) in degrees and also Mean Squared Error (MSE) of depth.

Tests of accuracy on synthetic data
In the following, we discuss our results obtained for the synthetic test images as displayed in Figures 6, 7 and 8. All images are rendered using Blender software. The 3D model of Oldman and Woman images are obtained from [1] and [2] respectively. 3D models of the other synthetic images are publicly available at [3]. All images except Coping (256 × 200) are in the size of 256 × 256. All implementations are performed on an Intel Core i7 processor with 8 GB of RAM.

Comparison with other BRDFs:
In Figure 3 documenting the first set of experiments, we represent the results of comparison between the recent approach suggested by Khanian et al. [27] and also the results of Lambertian reflectance model (as the most common model applied in PS). The model presented in [27] is based on the Blinn-Phong reflectance. We extended our experiments to a wide range of the specular material parameter k s = 1 − k d to evaluate how the varying k s influence the error of 3D reconstruction. It can be noticed that the output of our method achieves higher rate of accuracy over [27] and Lambertian model as k s increases. Whereas, there are some fluctuations when k d k s or k d k s . As illustrated, error rates of presented models are almost similar for low values of k s . However, as k s increases, the superiority of our model over Lambertian and [27] is significant. This observation agrees well with our motivation for focusing on the high frequency variations and model these components explicitly. One particular interpretation for the sharp ascending trend of the error rate for Coping input image (in the case of higher values of k s parameter) is the presence of the flat background region which leads to high amount of shiny highlighted areas in input images. As for the difference between [27] and Lambertian model, we find that the Lambertian still performs a little better on very low values of k s ; however, when k s is larger than 0.3, [27] improves accuracy much more than Lambertian model. As indicated, the MSE comparing our 3D reconstruction and ground truth data, demonstrates that our method is capable of producing accurate 3D reconstructions for the indicated images with specularity. The qualitative evaluations can be also seen as recovered surfaces shown in Figure 6. Our faithful 3D reconstructions with a high amount of details even for tiny surfaces containing low-depth details (e.g. Coping) represent the proficiency of our approach.

Optimization procedures:
As the next experiment shown in Figure 5, we compared various optimization procedures (applied in first and second steps of our RON technique) for several surfaces with diverse k s . It turns out that Dog Leg method achieves higher accuracy in all cases. Although, BFGS and Levenberg approaches follow the same trend for lower values of k s (Levenberg method can slightly outperform BFGS technique for low values of k s ) BFGS performs better as k s increases.

Lightening analysis:
To investigate the 3D reconstruction sensitivity w.r.t. the lightening model, we applied different PS frameworks.  Quantitative and qualitative results of this investigation are shown in Table 3 and Figure 7, respectively. To this end, we evaluated our approach regarding different available lightening conditions. In the first experiment, we consider the distant light source leads to perform directional-light-based PS referred as DLPS. The output images of this simulation resemble the real-world images that are illuminated by sun. The second evaluation is conducted to simulate spatially dependent light resulting in point-wise-light-based PS denoted as PLPS. We observed that the higher accuracy is achieved by applying the point-wise light source and adjusting lightings based on spatially dependent light as presented in (13) and (14). Figure 7 reveals that DLPS has problem to recover some areas since capturing images by directional-light leads to more shadows as shown in Figure  7. On the other hand, using the point-wise light source in the scene and formulating PLPS can provide the higher amount of accuracy (cf.   Table 1.
As it can be seen in the graph, the proposed method can be also favored due to its higher accuracy w.r.t both obtained normal vectors and depth map. To illustrate, our technique can provide accuracy improvement in normal field extraction over [22]

Comparison with several PS methods:
As another evaluation shown in Table 1, we compared our method from different aspects on the set of images with the state-of-the-art approaches: constrained bivariate regression [22], differential ratios [36], dimensionality reduc-tion based on PCA [4] and also least squares-regression [67]. For a fair comparison, normal map are obtained by the finite-difference of depth map for the methods suggested in [22,36] (these schemes provide depth, but not normal vectors) and the depth map is obtained by our integrator (section 5.6.) for [4,67] (these techniques obtain normal vectors, but not depth map). All methods show worse performances than our approach. One reason is that highfrequency variations are mis-classified and leads to the unpredictable errors. As can be seen in Table 1, the proposed strategy dramatically reduces error in computing normal    Table 3. MSE of our 3D shape reconstruction with perspective Cook-Torrance reflectance model in the presence of specularities using different lightening models.
in Table 1 is shown in Figure 4. For example, we could improve the accuracy in extracted normals by 93.15% over [36] and 90.43 % over [67] and provide higher accuracy in depth reconstruction by 73.29% over [36] and 70.46 % over [67]. Furthermore, the first row in Table 1 demonstrates the efficiency of our proposed method. Our method is the first one with significant high speed providing all information of normal vectors, colorful albedo, roughness value and depth map. Our speed improvement over mentioned techniques is represented in Table 2. The running time of [36,67] are comparable, yet their results appear both quantitatively and qualitatively different. The qualitative comparison of our approach with the mentioned methods is shown in Figure  8. The performance ranking of methods is consistent with qualitative results. Our results demonstrate the ability of the proposed scheme to provide a complete reconstruction without any deviation at specularities which outperforms other comparing approaches.

Test of applicability on real-world images
Finally, we evaluated the capability of our approach on a set of real-world images shown in Figures 9,10,11,12 as a test of the practical applicability of our method. All the images used in these experiments are captured by the camera of a Samsung Galaxy S5 smartphone and using a flash. For image acquisition the photographed figurine is standing on a table covered by a black tablecloth as shown in Figure 2. The light directions are measured by considering the camera as the origin of the coordinate system. The recovered 3D shape for the first real-world image is presented in Figure 9. In this case, we created a complex scene composed of numerous small statues with different sizes as a challenging set. However, we could reconstruct the surface with correct geometry where relevant details and main features are recovered faithfully. Let us stress that our realworld images are taken without any laboratory equipment or controlled setups. Nevertheless, the reconstruction result is in our opinion very reasonable. This test demonstrates the robustness of our method and its potential for real-world applications even using consumer-level equipment. Moreover, again for this real-world scene, we applied different PS-based lightnings models for a same light source of a flash. Improved quality of the 3D reconstruction with spatially dependent lights (PLPS) shown in Figure 9 (in terms of the height field of depth map and details) can be an advantage offered by this model. More real-world experiments are presented in Figures 10 and 11. As can be seen, our approach succeeds in producing accurate 3D reconstruction results even for high frequency details of tiny structure of Berlin souvenir statues. The colors of these images are extracted in Figure 12 where the diversity of colors is recovered as well in our non-uniform colorful albedo reconstruction. Let us also point out that our results show the capability of reliable 3D reconstruction for specular materials by using a minimum number of input images (for all synthetic and real-world experiments, we used three input images). Thus we document here that our method works under reasonable practical conditions with inexpensive instruments so that it can be used for many potential applications.

Conclusion
We presented a real-time robust PS which is also benefiting from perspective Cook-Torrance reflectance model to explicitly handle specularities and remove the limitations of working with diffuse materials or orthographic projection. A RON optimization technique based on DGMC approach is introduced to obtain accurate information of normal vectors, depth, roughness and colorful albedo. Our proposed RON and DGMC techniques can be applied in any highly non-linear optimization framework as a key point for providing multi-variable solutions and proper initialization. These innovations provide important steps towards a reliable PS. Furthermore, we equipped our model with the spatially dependent lightening offering more reasonable reconstructions. We have demonstrated the applicability of our method by applying minimum number of input images without any laboratory conditions or facilities. Furthermore, we stretch our approach to its full potential by extracting 3D reconstruction of micro-prints. Our method has the potential to be useful as the basis of future developments.
Coping Horseman Soldier Fighter Figure 6. Left: input images including specularity. Right: our 3D reconstruction results using perspective Cook-Torrance reflectance model. These results illustrate the capability of the proposed method for providing faithful reconstructions with high frequency details even for the fine details of a tiny surface (e.g. first input image).

Carpenter
Oldman Woman Figure 7. First row: input specular images with directional light. Second row: our 3D output using DLPS. Third row: input images with pointwise light. Last row: our 3D output with PLPS. Images produced by directional light are imposed by more specularity and shadows in comparison with images with pointwise light. These shadow parts lead to losing some reconstructed area. Figure 8. Comparison of several PS methods. From left: 1) input image (Militaryman) including high specularities, and the 3D results of 2) our reconstruction applying perspective Cook-Torrance reflectance model without any deviation or artifacts, 3) reconstruction of [67] that could not handle specularity areas, 4) reconstruction of [36] which shows deviations specially around the highly specular areas 5) [4] shows problem in producing complete 3D reconstruction in presence of spcularity and 5) [22] which still includes distortions. Figure 9. From left: 1) input real-world image including several tiny statues captured without laboratory setting cf. Figure 2, 2) our 3D reconstruction with Cook-Torrance reflectance model, 3) our 3D reconstruction using DLPS for the face of the smallest statue in the scene and 4) our 3D reconstruction using PLPS for the same statue. Figure 10. More real-world experiments with specularity, first row) input images including Einstein statue, two Berlin souvenir statues with comprehensive structure and a metallic coin, second row) our 3D reconstruction using DLPS and last row) our 3D reconstruction using PLPS. High quality details and structures recovered by the proposed approach confirm offered advantages and agrees well with them. Figure 11. Some qualitative evaluations for the proposed method: from left) close-up views of our reconstruction results corresponding to the red, blue, green and magenta rectangular areas in Coping as synthetic input and Einstein, Coin and first Berlin statue as the real scenes. It can be seen that our method achieves desirable reconstruction quality even for very fine details. Figure 12. Reconstructed albedo of two color input images shown in Figure 10. As can be seen, our RON approach can produce nonuniform albedo from images with a variety of colors. Since the albedo is not same across the entire real-world surfaces, our variant albedo extraction is more adjusted for real-world situations. It should be noticed that these statues are very tiny surfaces.

Algorithm 2: Quasi-Newton with BFGS updating
Primary input : X 0 that should be computed using our proposed "DGMC" technique explained in section 5.4. of the paper Secondary inputs: ϑ = 0, k = 0 output : Normal vectors for photometric stereo while J(X k ) > ε, and k < k max and ϑ < ϑ max do Compute d k := −(B(X k ) −1 )J(X k ) [X k+1 , dϑ] ← line search (X k , d k ) as applied in reference [33] of the paper Where is the computer accuracy and ε = 1e -12 is considered.


extfieldsofstudy:
Biology
Computer Science
Physics
Sociology

s2fieldsofstudy:
Computer Science
Sociology

text:
Tipping Points for Norm Change in Human Cultures

Humans interact with each other on a daily basis by developing and maintaining various social norms and it is critical to form a deeper understanding of how such norms develop, how they change, and how fast they change. In this work, we develop an evolutionary game-theoretic model based on research in cultural psychology that shows that humans in various cultures differ in their tendencies to conform with those around them. Using this model, we analyze the evolutionary relationships between the tendency to conform and how quickly a population reacts when conditions make a change in norm desirable. Our analysis identifies conditions when a tipping point is reached in a population, causing norms to change rapidly.

Introduction
Social norms are critical in enabling human populations across the world to coordinate and accomplish different tasks. The strength of these social norms, however, differs widely around the globe, as has been established by past neuroscience, field and experimental research [GRN + 11, HG14, HBG15, MKHG15, RGNL15, DNG17,GHJ17]. Some cultures are said to be tight, with strong social norms, typically characterized by high degrees of norm adherence and strict punishment directed towards norm-violators. Other cultures are said to be loose, with weaker norms characterized by a higher acceptance of deviant behavior [GRN + 11, HG14,RGNL15].
Tightness-looseness is a dynamic construct, yet to date, there has been little research on the evolutionary processes that lead to changes in societal norms, the rate at which such changes occurs, and how these processes vary across different cultures. In this paper, we aim to study how cultural differences in the way humans interact and influence each other heavily influence how societal norms are established and the rate at which they change across the world. We use evolutionary game theory (EGT) to examine the causal relationship between an individual's tendency to conform with those around them and the rate at which norms are changed in different cultures (see Section 2 for a brief discussion on EGT). More specifically, our primary contributions in this paper are as follows: • Drawing on recent research in cultural psychology, we propose a game-theoretic model of a culture based on the tendency of an individual to conform with others, vs. being more individualistic in their behavior.
• Using this model, we provide conditions under which a population is open to changing the current norm in a society, depending on the pressure of conformity and the abruptness of the boundary between following and violating the norm.
• Finally, we analyze the rate at which such norm changes occur. We show that tighter cultures are more likely to be initially resistant to norm changes compared to looser cultures. Further, we analyze conditions under which tighter cultures sometimes reach a tipping point, where a large proportion of the population suddenly switch to a new norm.
The rest of the paper is organized as follows. Section 2 provides background and related work. We introduce our proposed model in Section 3 and study the rate of norm change in Section 3.2. In Section 4 we discuss the significance of our results.

Background and Related Work
Evolutionary Game Theory (EGT) was initially proposed to model biological evolution [HS03,Smi82,Wei97], but has been increasingly used to study human cultural evolution. The idea is to represent an interaction among individuals as a normal-form game, where individuals can use different strategies. The game's payoffs represent an individual's evolutionary fitness. In EGT models of cultural evolution, biological reproduction represents social learning: individuals are more likely to adopt strategies from others that produce high fitness and thus strategies that lead to higher fitness become more prevalent over time. While such models use highly simplified abstractions of complex human interactions, they aim to capture the essential nature of the interactions of interest, and thus have been increasingly used to study a wide variety of social and cultural phenomena, (see [DNG16] for a brief tutorial on how such models are generally set up) such as, cooperation and altruism [ ]. There has been, however, much less work done on understanding the processes that lead to change in an already established norm in a population. A related concept, the propagation of information in social networks, has been well-studied (see [CLC13,Jac10,EK10] for an overview), but these works typically do not account for the differences in how individuals interact and influence each other in different cultures. Data science approaches have also explored this question, however, it is very challenging to separate out the various confounding factors (such as institutional influence) to establish clear causal relationships [ZZX15, LGRC12, KYC + 12, LMK + 13]. Finally, note that [Hen01] and [DNG17] have previously studied the processes of norm change in societies, and our work extends these studies in important directions.
Using a more general model of conformist transmission that depends on the degree to which there is an abrupt boundary between following and violating the norm, we study the speed of norm change in different cultures, and show how tipping points during such norm change depends critically on the model of conformist transmission.

Proposed Evolutionary Game-Theoretic Model
Research in cross-cultural psychology has established that the strength of social norms varies considerably across cultures. Further, using historical and ecological data, it has been shown that this is related to the degree of threat that populations face [GRN + 11, HG14]. Stronger norms and sanctions are needed in high-threat situations to coordinate and survive, leading to tighter cultures. By contrast, populations that lack exposure to serious ecological threats can afford to have weaker norms and tolerance for deviance given that they have less need for coordinated social action (looser cultures). We now describe our model.
Consider an infinite, well-mixed population (i.e., each individual can interact with any other individual in the population) that evolves according to the well-known replicator dynamic [HS03]. For simplicity of presentation, suppose each agent may choose one of two possible actions: A and B (see Section 4 for a discussion on the assumptions used in our model). The two actions A and B correspond to possible norms that the society could settle on. Let x A and x B denote the proportions of the population using actions A and B respectively, with 0 ≤ x A , x B ≤ 1 and x A + x B = 1, and let x = (x A , x B ). According to the replicator dynamic, the rate of change in the proportions of agents using each action is given by the following differential equation:ẋ where i ∈ {A, B},ẋ i = dx i /dt (i.e., rate of change of x i ), f i (x) is the fitness of action i, and φ(x) denotes the average fitness of the population, i.e.: The replicator dynamic is based on the idea that the proportion of agents with a particular strategy increases when it achieves expected fitness higher than the average fitness, and vice versa.
Let u A and u B denote the payoffs associated with actions A and B, where 0 < u A , u B < 1 and u A + u B = 1.
To define the fitness function f i , we use the key insight that in loose cultures, individuals tend to choose the action that is most beneficial to them; but in tight cultures, individuals tend to conform to the same action that others use, even if a different action might be more beneficial to each individual. To model this mathematically, we let f i be a weighted combination of the payoff u i and an additional conformism fitness measure θ i that depends on whether the individual is conforming to others in the population. Let m denote the parameter controlling the weighting between these two fitness measures, i.e., the amount of conformist transmission in a population. Thus, we define f i as: where 0 ≤ m ≤ 1, and we define the conformism fitness measure θ i as: where k > 0. Note that we can vary the behavior of the conformism fitness measure θ i using the parameter k (see Figure 1). For example, when k is large, θ i is close to a step function where there is an abrupt boundary between following and violating the norm and agents have a non-zero conformism fitness only if they conform with the majority action: Note that with no conformism whatsoever (m = 0), each action's fitness depends solely on its payoff, i.e., typical of a very loose culture. On the other hand, with 100% conformist transmission (m = 1), i's fitness depends solely on the conformism fitness measure (for the case of θ ∞ i , this means that i's fitness depends solely on whether i is in the majority or the minority of the population). This is more indicative of a very tight culture. For simplicity, for the rest of the paper, we denote θ i (x, k) as θ i and θ ∞ i (x) as θ ∞ i .

When does norm change occur?
Suppose norm B has a higher utility compared to A, i.e., u B > u A . We are interested in analyzing the conditions for which a population shifts from norm A to B (norm change). We can re-write the average fitness to be: We are interested in anlayzing the rate of change in the proportion of B individuals. From (1), (2), and (4), we get:ẋ , if the current norm in the population is A, norm change takes place only if: Thus, norm change takes place only if the population is loose enough, while tighter cultures are more resistant to change. Further, note that θ ∞ A − θ ∞ B = 1 when x A > x B . Thus, when the conformist fitness measure is a step-function θ ∞ i , (6) becomes: m < (u B − u A )/(u B − u A + 1) < 0.5. Thus, for θ ∞ i , norm change occurs only if individuals in a population weigh their individual payoff more than whether they conform with others. Figure 1 (middle) shows a heatmap of how condition (6) varies with u B − u A and k when x B = 0.1. We see that the bound on m increases as u B − u A increases, i.e., a population becomes more likely to switch the norm. On increasing k, we see that the bound on m decreases. This makes intuitive sense, since a higher k makes the difference in conformist fitness between A and B clearer. Thus, in tight cultures, where people tend to agree more on what behaviors are appropriate vs. inappropriate in different situations [GRN + 11], a higher k would lead to more resistance to norm change.

Rate of norm change in tight vs. loose cultures
We are now interested in studying the speed with which norms change in different populations. Consider two possible values of m, namely m 1 and m 2 , with m 2 > m 1 (i.e., m 2 is a more conformist culture than m 1 ). Let the corresponding values ofẋ B be denoted byẋ 1 B andẋ 2 B , respectively. Assume further that both m 1 and m 2 satisfy (6), i.e., norm change takes place in both cultures. Analyzing the difference in the rates of change, from (5) we get:ẋ Note that when x B ≤ 0.5, θ B − θ A ≤ 0, which would mean:ẋ 2 B −ẋ 1 B ≤ 0, i.e., the more conformist culture would be slow to change initially. To analyze the case when x B > 0.5, let's assume x B = 0.5 + , for > 0.
Thus, x A = 0.5 − . From (7), we see that forẋ 2 B −ẋ 1 B > 0, the following condition needs to hold:  We are also interested in studying how the rate of changeẋ B varies with x B as a population switches to norm B, and how this relates to different levels of conformism. We first look at the maximum rate of changė x max B = max x Bẋ B (we numerically calculate this given values for k, m and u B − u A ). Figure 2 (right) plotṡ x max B for different m and k values, where we set u B − u A = 0.7. These values were chosen such that the norm changes from A to B for all the considered combinations (using the bounds from Section 3.1). We see that when k is low, lower conformism leads to higherẋ max B . However, as k increases (i.e., as θ B approaches θ ∞ B ), there is a clear transition, where more conformist cultures end up having a higher maximum rate of change.
This effect is clearer in the left plot of Figure 2, where we show howẋ B varies with x B . We see that when k is low, i.e., when there is no clear difference between A and B in its conformist fitness measure θ,ẋ B changes slowly for both tight and loose cultures, with the loose culture having a higher rate of change. With high k, however, we see that the tighter culture faces a tipping point, resulting in a sudden increase inẋ B with the tighter culture adopting a higher rate of changeẋ B than the loose culture. Thus, using a more general model of conformist transmission over [Hen01], we find that the parameter k, which makes the difference in conformist fitness between following and violating the norm clearer, has a big influence on the pressure of conformity, and thus on the rate at which the norm changes in a society. In summary, in a more conformist culture, initially peer pressure impedes the switch to the more beneficial norm B. But once enough of the population has switched, a tipping point is reached where peer pressure causes the rest of them to switch very rapidly.

Discussion
This paper presents an EGT model that aims to investigate how the cultural dynamics of norm maintenance and norm-change differ across various cultures. We show that tight cultures sometimes experience a tipping point for norm change while loose cultures typically face a more gradual change. The results presented assume an infinite well-mixed population with two possible actions. We believe it would be relatively straightforward to extend our results for multiple actions. Assuming an infinite well-mixed population made our model mathematically tractable and to provide exact conditions for norm change. As future work, it would be interesting to extend this model to the finite population case, where interactions between individuals are dictated by a social network.