Medicine Computer Science
Computer Science-2021.gz:143420
Computer Science
057f3208a5cee94cb32fc895615019d9da57783d
Robust Pose Transfer with Dynamic Details using Neural Video Rendering

Pose transfer of human videos aims to generate a high fidelity video of a target person imitating actions of a source person. A few studies have made great progress either through image translation with deep latent features or neural rendering with explicit 3D features. However, both of them rely on large amounts of training data to generate realistic results, and the performance degrades on more accessible internet videos due to insufficient training frames. In this paper, we demonstrate that the dynamic details can be preserved even trained from short monocular videos. Overall, we propose a neural video rendering framework coupled with an image-translation-based dynamic details generation network (D2G-Net), which fully utilizes both the stability of explicit 3D features and the capacity of learning components. To be specific, a novel texture representation is presented to encode both the static and pose-varying appearance characteristics, which is then mapped to the image space and rendered as a detail-rich frame in the neural rendering stage. Moreover, we introduce a concise temporal loss in the training stage to suppress the detail flickering that is made more visible due to high-quality dynamic details generated by our method. Through extensive comparisons, we demonstrate that our neural human video renderer is capable of achieving both clearer dynamic details and more robust performance even on accessible short videos with only 2k - 4k frames.


INTRODUCTION
R ECENTLY, great progress has been achieved by applying neural networks to video synthesis, especially the human motion transfer task, which aims to transfer the action of a source person depicted in a video to a target one. The most essential challenges for this problem include but are not limited to synthesizing a detail-rich target video and imitating a variety of human motions, which can differ significantly from the training set.
Most existing approaches fall into one of the two categories: image-to-image translation methods [1], [2], [3], [4], [5], [6] which map pose labels to person images, and neural rendering based techniques that learn explicit 3D representations coupled with the traditional graphics rendering pipeline. Despite the powerful representational capacity of deep features that can help generate detail-rich results, image-to-image translation methods rely on blackbox 2D generative networks and often introduce obvious artifacts for poses with a large deviation from the training samples. Hence a great deal of training frames are needed to cover as many poses as possible, e.g., more than 10k frames are used for each subject in [1]. For neural rendering based techniques, the explicit 3D representations are introduced and hence the generative models become more stable. However, they either learn a static texture representation during the training stage e.g. [7], [8], which is temporally invariant and often leads to blurry results, or rely on finely reconstructed 3D models [9], which are difficult to obtain without large amounts of multi-view data. In general, all of these methods degrade when training data is limited to a common monocular video, which is more accessible in our daily life, e.g. short videos from the internet.
To alleviate aforementioned problems, we propose a novel generative framework that seamlessly couples image translation components and neural rendering, to gain the benefits of both and works well even with only a short monocular video. We ease the training difficulty and reduce the data dependency by presenting a novel learnable texture representation together with a corresponding pose-aware dynamic details generation network (D 2 G-Net) embedded in the neural renderer. Specifically, rather than using a traditional static RGB-channel image, we represent the texture with a hybrid feature map, which encodes the static RGB colors explicitly and the dynamic details of human appearance implicitly. Therefore, the hybrid texture serves as a higher level description of human appearance compared with classic texture images. Meanwhile we also predict the texture coordinates of each pixel in each frame directly for more flexible UV mapping, which sidesteps the difficulty of fine 3D reconstruction. By this means, the learned hybrid texture can be adaptively mapped to 2D screen space in a differentiable way without the traditional rasterization rendering pipeline, which is then fed into D 2 G-Net for dynamic detail generation. Since the human appearance details are pose-varying, the generation process of D 2 G-Net is also conditioned on the current pose. Hence the mapped feature can be rendered into a foreground human figure with clear details dynamically. Moreover, to obtain the complete background image, we propose a refinement strategy by integrating the background information from arXiv:2106.14132v2 [cs.CV] 14 Jul 2021 With only an accessible internet video as training data, our neural rendering framework is able to synthesize temporally coherent person images from given pose sequences, while keeping rich details compared with traditional rendering results using static textures. all the video frames with different foreground-background occlusion due to different poses. Finally, we composite the predicted foreground and background together to obtain the final rendered result.
However, the generated video will introduce flickers and jitters if splicing the individually generated frames directly. One feasible way [2] to solve this problem is to bring in video discriminators and the optical flow prediction module. However, such discriminators are difficult to train and the flow module relies on time-consuming computation even during the inference stage. In contrast, we incorporate the temporal loss [10] to our human video generation task, which only affects the training process and efficiently produces much more temporally consistent videos than the SOTA (state-of-the-art) methods, as demonstrated in Table 3, Table 4 and the supplementary video.
Here, we summarize the technical contributions as follows: • A novel end-to-end neural rendering framework for human video generation with dynamic details using accessible monocular video as training data.
• A pose-aware dynamic detail generation network (D 2 G-Net) which renders the learned texture feature into high-fidelity human images with pose-varying details.
• Additional spatial and temporal priors, specifically, explicit 3D human mesh and temporal consistency are integrated to enhance the robustness.

RELATED WORK
Video-based human motion transfer has been extensively studied over the recent decades due to its ability for fast video content production. The early approaches accomplished this task by manipulating existing video footage, e.g. Video Rewrite [11]. Benefiting from the rapid development of deep learning especially Generative Adversarial Networks (GANs) and their variants [12], [13], [14], GANbased methods are becoming increasingly popular. Meanwhile, the neural rendering approach also rises up [7], [9], [15], [16], [17] and learnable components are introduced to classical rendering pipeline. Here we summarize major advances for human pose transfer, namely techniques based on image-to-image translation, and those using neural rendering. Finally, we overview methods dedicated to video synthesis.
Image-to-image Translation. Image-to-image translation was proposed in Pix2pix [18], in which a conditional GAN with a U-net architecture [19] is used to transfer an image from one domain to another by training with paired data. In addition, Pix2pixHD [20] introduced a multi-scale generator, so that it could synthesize high-resolution photorealistic images. Such architectures and training methods were widely adopted in several follow-up methods, some of which [1], [2] aim to translate pose labels to human images with visually pleasing results. However, such fullysupervised learning can easily cause overfitting, leading to poor results when the desired poses are quite different from the training set.
Some other approaches design elaborate pipelines based on image translation architectures for more general poseguided person image synthesis. Ma et al. [5] attempted to generate better details by conditioning on both reference images and pose labels. [3], [4], [6] assisted the process of image translation with flow information. LWGAN et al. [21] presented a novel warping strategy, which uses the projection of 3D models to tackle motion transfer, appearance transfer and novel view synthesis within a unified framework. Balakrishnan et al. [22] segmented the human body into parts and spatially transformed them according to the target pose. However, these methods focus on synthesis of general human images with low resolution, while ours aims to generate high fidelity person-specific video.
Human Image Synthesis based on Neural Rendering. Some approaches performed human body synthesis by employing an explicit 3D representation and neural rendering. Dense pose transfer [8] combined neural rendering and deep neural synthesis via a blending module. Meanwhile, texture inpainting was utilised to ensure plausible results for the body parts occluded in the input image. Textured Neural Avatars [7] learned a full-body neural avatar by estimating an explicit texture map and mapping the input pose to a UV-coordinate image. As static texture maps are used, most high-frequent details are consequently lost in the synthesized results. Note that Liu et al. [9] also proposed to predict dynamic texture depending on poses. However, they employed accurate 3D reconstructions captured by dedicated devices, which are not practical in most application scenarios. In this paper, we propose a part-based texture generator that can predict dynamic textures according to different input poses. Although no high-accuracy 3D reconstructions are involved in training process, it still generates realistic and detail-rich person images.
Video Synthesis. A naive approach to video synthesis is through concatenating synthesized images frame by frame. However, without considering temporal consistency, it will inevitably bring flicker artifacts to generated videos. Early efforts on video synthesis attempted to improve continuity in the time domain via a recurrent neural network [23], [24]. To introduce more fine-grained controls, Chan et al. [1] adopted a conditional GAN [20] and predicted two consecutive frames together to obtain temporally coherent results. Wang et al. [2] proposed to add an optical flow module and a video discriminator to the off-the-shelf image translation network [20], leading to an increased computational complexity. Here we introduce the temporal loss [10] to the human video generation task, which achieves temporally consistent results without additional network modules.

Overview
Our aim is to generate a new video of the target person imitating the specific movements using only a short monocular video as training data, while keeping high fidelity and temporal consistency. To achieve this, we propose a novel neural video rendering framework containing both a hybrid texture representation and an embedded dynamic details generation network, as illustrated in Fig. 2. Specifically, given consecutive pose sequence, we first predict the texture coordinates of each pixel through an UV generator. Meanwhile, we initialize the texture with a learnable feature map. Afterwards, based on the predicted UV coordinates the texture feature is mapped to the screen space, which is then fed into the pose-conditioned dynamic details generation network for detail-rich human foreground image rendering. Moreover, to complete the generated frame we propose to combine the background refinement into the end-to-end training framework, by which means the information of frames with different foreground-background occlusion can be integrated effectively.

Foreground Image Generation
In the classical graphics rendering pipeline, the texture maps are always static RGB images and not conducive to the characterization of high frequency detailed signals, while the geometric deformations are responsible for dynamic details. DTL [9] first proposes to learn a dynamic texture to characterize the high-frequency signals of human appearance. However, its dynamic texture learning is under the supervision of unwrapped video frames, which relies on the fine-grained character model reconstructed from multiview training data. While in our neural rendering pipeline, we ease the need of multi-view training data with an endto-end differentiable framework, which avoid the explicit 3D reconstruction for rendering. To improve the fidelity of generated frames, we also introduce the hybrid texture representation and a corresponding dynamic details generation network (denoted as "D 2 G-Net"). The former contains both explicit static color and implicit details characteristics of human appearance, while the latter is responsible for varying details visualization dynamically in the neural rendering process. Here we start introducing our pipeline from the representation of pose labels, followed by the constitution of hybrid texture representation. Then we expound the neural rendering process, which includes the UV mapping as well as the dynamic details generation from texture feature.
Pose Labels Representation. The pose labels we adopt contain both 2D and 3D features. On the one hand, the 2D pose label is a skeleton image based on keypoints extracted by OpenPose [25]. On the other hand, the 3D label is the projection images of a 3D human mesh, which is obtained using a video-based reconstruction approach [26]. Each pixel of the 3D labels contains 3-channel Laplacian features [27], which are intrinsic characterization of 3D geometry and capture 3D body shapes meanwhile. We adopt 2D keypoints because they are relatively more accurate and easy for tracking, while 3D information can cope with pose ambiguity and self-occlusion. We concatenate 2D and 3D images into a 6-channel image as the pose label.
Hybrid Texture Representation and Initialization. To alleviate the decrease of fidelity caused by static texture map, we introduce hybrid texture representation, which is a high-dimensional feature map consisting of both explicit RGB color and implicit detail characteristics. The hybrid texture enrich the generated details by 1) encoding the appearance details implicitly and compensating the explicit RGB map 2) regularising the UV generator during the joint learning with D 2 G-Net. Technically, the texture formation follows the DensePose system [28], which unwraps the human body into N (N = 24) patches and provides a mapping from each pixel to a certain patch of human surface. Therefore, the hybrid texture is also composited of N parts (denoted as T i , where i = 1, · · · , N ), corresponding to N parts of human body. The hybrid texture contains 18 channels. The first three are initialized as RGB colors, coming from the average value by unwrapping each frame to the texture space according to Densepose, while others are initialized as zero.
Neural Rendering. In this stage, we map the hybrid texture to screen space differentiable by predicting the UV coordinate through an UV generator. The mapped features are then fed into D 2 G-Net for detail-rich human foreground generation.
UV Generation. The traditional rendering pipeline relies on explicit 3D models for texture mapping. However, it is difficult to obtain fine 3D human models from only a monocular video. Moreover, the rasterization blocks the backward gradient since its indifferentiable characteristic, which makes the end-to-end training impossible. In our pipeline we avoid this problem by predicting texture coordinates (UV) of each pixel in each frame directly with a UV generator. To be specific, the UV generator takes current pose labels as input, and outputs the UV coordinates and part probabilities for each pixel in the video frame. The part probabilities P i (i = 0, 1, · · · , N ) have the size H×W , which pose and I t pose are extracted from adjacent video frames. The UV generator takes pose labels as input and predicts the UV coordinates of each pixel, i.e. Iuv. Afterwards, the hybrid texture is mapped to the screen space according to the predicted UV coordinates and then translated to human foreground images with dynamic details. Meanwhile background image is refined during the training process and final synthesized images are obtained through a combination of foreground and background images. Note that adjacent frames are trained as a whole for the implicit learning of temporal coherence in the neural video renderer.  Fig. 3. Illustration of the D 2 G-Net. I hd fg is obtained by mapping the hybrid texture to the screen space. The translation network aims to translate the high-dimensional feature to RGB color with dynamic details under the current pose. Note that the pose label is conditioned on the feature level to guide the translation process.
represents the probability of each pixel belonging to N parts (P 1 , · · · , P N ) or background (P 0 ), while the coordinates C i (i = 1, · · · , N ) have the size of H × W × 2, indicating the UV coordinates of each pixel in the corresponding part.
Here H and W are the height and width of video frames respectively. Then the human foreground of hybrid texture I ht fg can be obtained by where φ is a function that maps the hybrid texture T i to the screen space according to UV coordinates C i . We extract the static component, i.e. RGB channels from I ht fg as I fg , which is a human foreground image without details and serves for subsequent regular loss calculation (Eq. 9). By this means we avoid explicit 3D shape modeling and provide a way for end-to-end training, which enables us to find the best hybrid texture representation and corresponding D 2 G-Net.
Dynamic details generation. In this part, we aim to visualize the implicit details contained in texture feature by translating the the human foreground of hybrid texture, i.e. I ht fg into detail-rich human images, denoted by I fg . Since the same texture feature should be interpreted to different appearance characteristics under different poses (e.g., wrinkles on the belly when bending down while flat clothes when standing erectly), we introduce the pose label as guidance during the dynamic details generation process. We adopt the pix2pixHD [20] generator as the backbone of our D 2 G-Net, which has achieved excellent results on the imagetranslation tasks. The pose label I pose is conditioned by concatenating with I ht fg at the intermediate level, as illustrated in Fig 3. The functionality of D 2 G-Net can be formatted as Note that although I ht fg also contains pose information, the varying feature increase the difficulty of converge during training. We perform the ablation study of pose condition in Sec 4.3.

Background Refinement and Combination
We will complete the generated detail-rich human foreground into a final video frame in this subsection. Since we focus on the human video generation with static background, we optimize the background image during the training process, which utilizes the information from all frames. In order to start from a reasonable initial state, we initialize the background image with a state-of-the-art inpainting network.
We obtain the coarse initial background through frameby-frame human body deduction and inpainting. Specifically, we segment the foreground (human) based on a Unet [19] from the background for each frame in the training video, and then average the inpainted results obtained using the image inpainting approach [29].
During the training stage, the initial background image is updated according to the backpropagated gradient. By this means the information of all frames are aggregated and the inpainted artifacts are eliminated effectively, as illustrated in Fig 4. Since P 0 denotes the probability of each pixel belonging to the background, we can combine the foreground and background by where I syn is the final synthesized frame, I syn is the synthesized frame of static component, i.e. RGB channels from the hybrid texture, and represents element-wise multiplication.
Video frame Initial Background Refined Background Fig. 4. Comparison of the inpainted initial background and the refined background with our method. It could be found that our method could inpaint the background with sharper details.

Temporal Consistency
The video produced by processing each frame individually does not look realistic and natural enough because of the inevitable flickering and jittering artifacts, especially when the dynamic details are presented. To address this problem, we introduce the temporal loss [10] into the human video generation task, which is defined as the L 1 loss between the generated frame I t syn at time t and the warped version of the generated frame I t−1 syn at time t − 1: where f t (·) is a function that warps the generated frame at time t − 1 to time t according to the pre-computed optical flow, and for an image I, I(k) refers to the pixel at k. D = H × W is the dimension of the output. c k denotes the per-pixel confidence of the optical flow: 0 indicates that the optical flow at the corresponding pixel is not credible; while 1 is the opposite. Note that we pre-compute the flow as well as flow confidence on training data, and the flow warping module is not needed during the inference stage, which reduces the inference time effectively, as compared in Table 4.

Full Objective
We denote the UV generator and D 2 G-Net as G uv and G D 2 G respectively. Note that the G uv is pretrained on a large corpus of different characters frames, since the mapping from pose labels to pixel-level UV coordinate is personagnostic. Then we finetune the G uv during the personspecific training for each subject to fit the particular body shape. The pretrain objective is given by minimize the following loss L uv (P,P , C,Ĉ) = L CE (P,P ) + ||C −Ĉ|| 1 where L CE is the cross entropy loss. P and C are the predicted probabilities and UV coordinates of the UV generator, whileP andĈ are the ground truth. Note thatP andĈ can be obtained through existing 3D priors, e.g. rasterization of rigged 3D human models. In practice we use the results of DensePose directly.
Our whole training network can be trained in the endto-end manner. Let I pose , I bg I real and T be the pose label, background image, ground truth image and hybrid texture respectively. The overall objective is formulated as: L supervised consists of a perceptual loss and an L 2 loss, which has the following form: The perceptual loss regularizes the generated result I syn to be closer to the ground truth I real in the VGG-19 [30] feature space, while L 2 loss does similar restraints at the pixel level. Regular loss term. Note that L supervised ( I i syn , I i real ) is the regular loss term calculated from the static component from the hybrid texture. We add this term to stable the UV generator and avoid the unreasonable local optimum due to the concurrent update policy of G uv and G D 2 G . We visualize the effect of regular loss in Fig 11.

EXPERIMENTS
We perform sufficient comparisons to demonstrate the advantage of our approach, which preserves dynamic texture details while only relies on a single accessible monocular video for training. We compare our approach with the state-of-the-art methods under the same data setting. We also verify the importance of each key component in our framework, including the image translation components, pose label condition in the D 2 G-Net, the regular loss term and the temporal loss.

Setup
We start by introducing the experimental setting, which includes the dataset configuration, implementation details and the evaluation metrics.
Dataset. We conduct massive experiments on the dataset consisting of selected videos from iPER dataset [21], several online videos and one own video, each of which lasts about 3 minutes with 25 FPS (2∼4k valid frames). To ensure the data diversity, there are multiple clothes and actions for each subject.
Implementation details. We design the architecture of D 2 G-Net based on the image translation approach [20], and adopt the network in [7] as our UV generator. In the pretraining stage, we train G uv for 5 epochs with data of 10 subjects in the iPER dataset (total 20k frames), which will The two rows correspond to consecutive frames of t and t + 1 respectively. We compare our approach with EDN [1] and Vid2vid [2] both temporally (orange box) and spatially (red box). Our method can produce more reasonable body parts e.g. face and belly, as well as better temporal coherence. be fintuned to any specific subject directly. In the training stage, G D 2 G , G uv , T and I bg are updated jointly for 30 epochs with Adam optimizer and the hyperparameters are set as lr = 0.002, λ Temp = 100, λ f = 10 and λ l2 = 200. For all the experiments, the frame image is resized into the resolution 512 × 512. For each subject we use the Nvidia Tesla V100 GPU to train for about 1 day. More details about the networks are given in supplementary material. Evaluation Metrics. We use different metrics for quantitative evaluation under two conditions. 1) To directly measure the quality of the generated images, we perform selftransfer in which the source and target are the same subject, and then use Structural Similarity Index Measure (SSIM) [31] and Peak Signal-to-Noise Ratio (PSNR) to assess the similarity between source and target images. We split frames of each subject into training and validation sets at the ratio of 9:1 for this evaluation. 2) We also evaluate the performance of cross-subject transfer where the source and target are different subjects, using Fréchet Inception Distance (FID) [32] with feature extracted by InceptionV3 [33] network. It is worth noting that we compute the FID score between the original and generated target images since there exists no ground truth for comparison in this case. However, the above metrics can only reflect the quality of generated results from the holistic perspective. To better measure the robustness of the generative model, we propose a new metric which focuses on poses with a large deviation from the training sample (denoted as "challenging pose"). To be more specific, for each pose i in the validation set, we compute its nearest neighbor distance d i with training samples. Here the distance between two poses is characterized by Euler distance of 2D keypoints. Assuming the poses with larger value of d i are more challenging, M validation samples (M = 10 in our experiments) with largest d i are selected to form the challenging poses. We evaluate the model performance under these poses with SSIM and PSNR, which is denoted as "Robust-SSIM" and "Robust-PSNR" for clarity.
In addition to these metrics calculated on single frames, we also introduce the temporal error E Temp [10] to estimate the temporal coherence of generated image sequences. E Temp is defined to be the average pixel-wise Euclidean color difference between consecutive frames over a video sequence: where T is the total number of frames. This formulation is very similar to Eq. 5, except that we sum the temporal loss The two rows correspond to consecutive frames of t and t + 1 respectively. We compare our approach with TNA [7] and DTL [9] both temporally (orange box) and spatially (red box). Our method can produce clearer details e.g. high-frequency signals on clothes, as well as better temporal coherence.
over all consecutive frame pairs in a video sequence.

Comparison with SOTA
We compare our method against existing state-of-the-art methods Vid2vid [2], Everybody Dance Now [1] (EDN) and Liquid Warping GAN [21] (LWGAN), using official implementations. We also compare with existing neural rendering methods Texture Neural Avatar [7] (TNA) and Dynamic Texture Learning [9] (DTL) based on our re-implementation with the same experimental settings as original, since no source code is available. Note that for TNA, we add our refined background for fair comparison. And for DTL, we use the off-the-shelf UV parameterization (DensePose [28]) since the accurate 3D reconstruction which relies on multiview capture devices is not available. We show those comparison results with SOTA in Table 1. Moreover, since LW-GAN model is not limited to a particular person, we only list the results for reference, but it does not participate in comparison.
Comparison with direct image translation approaches. In general, for all approaches, the quality of generated results decreases when they are tackling the "challenging poses". However, the proposed method outperforms direct image translation approaches (EDN [1] and Vid2vid [2]) significantly, as shown in Fig 5, which demonstrates the robustness of our approach for the "unseen" poses during training.
Comparison with neural rendering approaches. Compared with other neural rendering approaches (TNA [7] and DTL [9]), our method produces the results with richer details and higher fidelity under the same data configuration. The comparison results are shown in Fig 6. Although TNA can still produce reasonable results when the training data is limited to monocular video, its static texture map goes against to the dynamic details generation under different poses. By contrast our approach achieves better performance by interpreting the high-dimensional texture feature to details dynamically. In terms of DTL, the performance degrades significantly in the absence of multi-view data and finely reconstructed 3D model, while our approach is more robust for the common monocular video.
We also visualize the comparison of self-transfer on validation set both spatially and temporally. Fig 7 illustrates the spatially comparison result of "challenging poses", which is stated in Sec 4.1. It can be seen that our approach achieves more robust performance on the "unseen" poses in training set. Fig 8 shows that compared with other rendering-based methods, our approach is able to learn reasonable varying details with the same dynamic characteristics as the ground truth.  7. Robustness comparison. We compare our approach with SOTA methods on "challenging poses" . Our approach outperforms others. Since the tracked poses fail to characterize motion blur in the driving motion, our method might produce clearer results than ground truth.
Driving motion TNA [7] Ours Driving motion DTL [9] Ours t t+1 Fig. 8. Comparison of dynamic details. The two rows correspond to consecutive frames of t and t + 1 respectively. Compared with TNA [7] and DTL [9], our method better delineates details for different poses. We show the results of self-transfer for the reference of ground truth. More pose transfer results will be exhibited in supplementary material.
Overall, our method synthesizes high quality results with rich high-frequent details, and is more robust to the challenging inputs compared with existing SOTA methods. Moreover, our approach, demonstrated in comparisons, produces more temporally consistent videos.
User study. We also conduct a user study to measure the human perceptual quality for pose transfer results. In our experiment we compare videos generated by EDN [1], Vid2vid [2], TNA [7], DTL [9] and ours. Specifically, we show the volunteers a series of videos by each method at the resolution of 512 × 512, and the volunteers are given unlimited time to make responses. Each participant is asked to select 1) The clearest result with rich details 2) The most temporally stable result 3) The overall best result We report our results in Fig 9. Although TNA [7] achieves most temporal coherence votes owing to the static texture map and lack of details, our approach shows more advantages in fidelity. Our results are considered more realistic than all existing methods.

Ablation study
We evaluate the role of each component in our pipeline via convictive ablation studies. Specifically, we assess the effect of image translation components by omitting the D 2 G-Net or remove the implicit component from the hybrid texture. We also investigate the relevance of pose label condition in D 2 G-Net. Finally we demonstrate the importance of regular loss and temporal loss.
Effect of image translation component. We evaluate the effect of image translation component in the neural rendering framework from two aspects and report the results in Table 2.
Importance of D 2 G-Net. We investigate the importance of details generation by training the network without the D 2 G- Ours also outperforms the others in temporal coherence. The numbers in italics are for reference only, not for comparison. Fig. 9. User study results. We report the percentage of participants' choices in three different aspects respectively. Overall, our method has achieved the best results. Net and using only the explicit component of hybrid texture (denoted as "w/o D 2 G"). This variant actually degrades to the static texture rendering and the dynamic details cannot be well characterized. Importance of the implicit component of hybrid texture. In our method we adopt a high-dimensional texture map to introduce both explicit RGB color and implicit detail characteristic. We compare the rendered result without the implicit components in hybrid texture (only "RGB channels") and "All channels", and illustrate the results in Fig 10. It can be seen that the implicit channels are important for clear dynamic appearance details generation.
Relevance of pose label condition. To characterize the dynamic details of human surface from hybrid texture under the varying poses, we condition the D 2 G-Net with current pose labels. Note that although there is implicit pose information in the human foreground of texture feature, the varying feature during training process would disturb the converge of the translation network. We report the result with and without pose label condition (denoted as "w/o condition") in Fig 10. Regular loss. Since our whole network is trained end-toend and both the UV generator and D 2 G-Net are learnable, they will interfering with each other and result in unreasonable local optimum. Therefore we add the regularize loss term to stablize the output of the UV generator. We visualize the results with and without regular loss in Fig 11. Temporal loss. We compare the temporal coherence of generated results with and without temporal loss ("w/o. Temp") using metric defined in Eq. 10, and replace the temporal loss with a two-frame video discriminator ("Video D"). The results are shown in Table 3. It can be seen that temporal loss suppresses the inconsistency between adjacent frames effectively. The generated results are visualized in Fig. 12. Moreover, by adopting the temporal loss we enforce our neural renderer to learn the temporal coherence implicitly and hence the flow warp module is not needed during the inference stage. By this means the computation complexity is reduced and the inference time decreases. We U map V map Results Fig. 11. Ablation study of the regular loss term. The top shows the results without regular loss term and the bottom shows our whole pipeline. It can be seen that the regular loss can stable the output of UV generator effectively and helps produce photo-realistic results.  TABLE 3 Ablation study for temporal loss and video discriminator ("Video D"). It can be seen that the temporal loss help reduce the temporal error and achieve better temporal consistency compared with other variants.
report single 512x512 frame inference time in Table 4 with a Nvidia Tesla P40 GPU.
w/o Temp Video D ours t t+1 Fig. 12. Ablation study of the temporal loss. We show two consecutive frames in each column. It can be seen that the temporal loss can help produce more coherent results.
Selection of pose labels. In this part, we perform an ablation study to verify the impact of each input component of our UV generator, including only 2D skeleton ("2D"), normal map (used in DTL [9]) and our full input (("2D+3D"), or "Ours"). The comparison results are shown in Table 5.

Background replacement
Our approach is also able to replace the background by the foreground-background separation. We exhibit one example

Method
Vid2vid [2] Ours Inference time (ms) ↓ 210 90  13. Example of the same generated person with different backgrounds. Our approach is able to generate videos with arbitrary novel backgrounds by replacing the refined background images.

LIMITATIONS AND DISCUSSION
Although our approach is able to generate coherent videos with high fidelity, there are still several limitations. First, we have to train a new model for each specific person, which greatly limits the applicable scenarios of our method. Besides, since our pipeline is based on existing pose tracking approaches, tracking errors will also effect the quality of generated results.

CONCLUSION
We have proposed a new approach for pose transfer on the human video synthesis task. By embedding the learnable hybrid texture into the neural rendering pipeline with dynamic details generation mechanism, our approach is able to generate high-fidelity human frames in an end-to-end manner. In this means, our approach avoids the dependency for abundant training data and serves as a more accessible approach. Furthermore, the temporal loss improves the consistency of consecutive frames significantly. Our approach outperforms SOTA in both fidelity and temporal coherence with training data from the short monocular videos.