Computer Science
Computer Science-2021.gz:143131
Computer Science
4ffd87551ab02eca22bd6d5ad945c8d036e80b1d
ICON: Implicit Clothed humans Obtained from Normals

Current methods for learning realistic and animatable 3D clothed avatars need either posed 3D scans or 2D images with carefully controlled user poses. In contrast, our goal is to learn an avatar from only 2D images of people in unconstrained poses. Given a set of images, our method estimates a detailed 3D surface from each image and then combines these into an animatable avatar. Implicit functions are well suited to the first task, as they can capture details like hair and clothes. Current methods, however, are not robust to varied human poses and often produce 3D surfaces with broken or disembodied limbs, missing details, or non-human shapes. The problem is that these methods use global feature encoders that are sensitive to global pose. To address this, we propose ICON ("Implicit Clothed humans Obtained from Normals"), which, instead, uses local features. ICON has two main modules, both of which exploit the SMPL(-X) body model. First, ICON infers detailed clothed-human normals (front/back) conditioned on the SMPL(-X) normals. Second, a visibility-aware implicit surface regressor produces an iso-surface of a human occupancy field. Importantly, at inference time, a feedback loop alternates between refining the SMPL(-X) mesh using the inferred clothed normals and then refining the normals. Given multiple reconstructed frames of a subject in varied poses, we use SCANimate to produce an animatable avatar from them. Evaluation on the AGORA and CAPE datasets shows that ICON outperforms the state of the art in reconstruction, even with heavily limited training data. Additionally, it is much more robust to out-of-distribution samples, e.g., in-the-wild poses/images and out-of-frame cropping. ICON takes a step towards robust 3D clothed human reconstruction from in-the-wild images. This enables creating avatars directly from video with personalized and natural pose-dependent cloth deformation.


Introduction
Realistic virtual humans will play a central role in mixed and augmented reality, forming a key foundation for the "metaverse" and supporting remote presence, collaboration, education, and entertainment. To enable this, new tools are needed to easily create 3D virtual humans that can be readily animated. Traditionally, this requires significant artist effort and expensive scanning equipment. Therefore, such approaches do not scale easily. A more practical approach would enable individuals to create an avatar from one or more images. There are now several methods that take a single image and regress a minimally clothed 3D human model [6,7,18,21,33,48]. Existing parametric body models, however, lack important details like clothing and hair [29,38,48,53,64]. In contrast, we present a method that robustly extracts 3D scan-like data from images of people in arbitrary poses and uses this to construct an animatable avatar.
We base our approach on implicit functions (IFs), which go beyond parametric body models to represent fine shape details and varied topology. IFs allow recent methods to infer detailed shape from an image [22,24,54,55,65,70]. Despite promising results, state-of-the-art (SOTA) methods struggle with in-the-wild data and often produce humans with broken or disembodied limbs, missing details, high-frequency noise, or non-human shape; see Fig. 2 for examples.
The issues with previous methods are twofold: (1) Such methods are typically trained on small, handcurated, 3D human datasets (e.g. Renderpeople [3]) with very limited pose, shape and clothing variation.
(2) They typically feed their implicit-function module with features of a global 2D image or 3D voxel encoder, but these are sensitive to global pose. While more, and more varied, 3D training data would help, such data remains limited. Hence, we take a different approach and improve the model.
Specifically, our goal is to reconstruct a detailed clothed 3D human from a single RGB image with a method that is training-data efficient and robust to in-the-wild images and out-of-distribution poses. Our method, called ICON, stands for Implicit Clothed humans Obtained from Normals. ICON replaces the global encoder of existing methods with a more data-efficient local scheme; Fig. 3 shows a model overview. ICON takes as input an RGB image of a segmented clothed human and a SMPL body estimated from the image [32]. The SMPL body is used to guide two of ICON's modules: one infers detailed clothed-human surface normals (front and back views), and the other infers a visibility-aware implicit surface (iso-surface of an occupancy field). Errors in the initial SMPL estimate, however, might misguide inference. Thus, at inference time, an iterative feedback loop refines SMPL (i.e., its 3D shape, pose, and translation) using the inferred detailed normals, and vice versa, leading to a refined implicit shape with better 3D details.
We evaluate ICON quantitatively and qualitatively on challenging datasets, namely AGORA [47] and CAPE [41], as well as on in-the-wild images. Results show that ICON has two advantages w.r.t. the state of the art: (1) Generalization. ICON's locality helps it generalize to in-the-wild images and out-of-distribution poses and clothes better than previous methods. Representative cases are shown in Fig. 2; notice that, although ICON is trained on full-body images only, it can handle images with outof-frame cropping, with no fine tuning or post processing.
(2) Data efficacy. ICON's locality helps it avoid spurious correlations between pose and surface shape. Thus, it needs less data for training. ICON significantly outperforms baselines in low-data regimes, as it reaches SOTA performance when trained with as little as 12% of the data.
We provide an example application of ICON for creating an animatable avatar; see Fig. 1 for an overview. We first apply ICON on the individual frames of a video sequence,  Figure 2. SOTA methods for inferring 3D humans from in-the-wild images, e.g., PIFu, PIFuHD, PaMIR, and ARCH++, struggle with challenging poses and out-of-frame cropping (E), resulting in various artifacts including non-human shapes (A,G), disembodied parts (B,H), missing body parts (C,D), missing details (E), and high-frequency noise (F). ICON deals with these challenges and produces high-quality results, highlighted with a green shadow Front view (blue) and rotated view (bronze).
to obtain 3D meshes of a clothed person in various poses. We then use these to train a poseable avatar using a modified version of SCANimate [56]. Unlike 3D scans, which SCANimate takes as input, our estimated shapes are not equally detailed and reliable from all views. Consequently, we modify SCANimate to exploit visibility information in learning the avatar. The output is a 3D clothed avatar that moves and deforms naturally; see Fig. 1-right and Fig. 8b. ICON takes a step towards robust reconstruction of 3D clothed humans from in-the-wild photos. Based on this, fully textured and animatable avatars with personalized poseaware clothing deformation can be created directly from video frames. Models and code are available at https: //icon.is.tue.mpg.de.

Related work
Mesh-based statistical models. Mesh-based statistical body models [29,38,48,53,64] are a popular explicit representation for 3D human reconstruction. This is not only because such models capture the statistics across a human population, but also because meshes are compatible with standard graphics pipelines. A lot of work [18,30,31,33,57,58,66] estimates 3D body meshes from an RGB image, but these have no clothing. Other work estimates clothed humans, instead, by modeling clothing geometry as 3D offsets on top of body geometry [4-7, 34, 50, 63, 72]. The resulting clothed 3D humans can be easily animated, as they naturally inherit the skeleton and surface skinning weights from the underlying body model. An important limitation, though, is modeling clothing such as skirts and dresses; since these differ a lot from the body surface, simple body-to-cloth offsets are insufficient. To address this, some methods [12,27] use a classifier to identify cloth types in the input image, and then perform cloth-aware inference for 3D reconstruction. However, such a remedy does not scale up to a large variety of clothing types. Another advantage of mesh-based statistical models, is that texture information can be easily accumulated through multi-view images or image sequences [6,12], due to their consistent mesh topology. The biggest limitation, though, is that the state of the art does not generalize well w.r.t. clothing-type variation, and it estimates meshes that do not align well to input-image pixels.
Deep implicit functions. Unlike meshes, deep implicit functions [17,44,46] can represent detailed 3D shapes with arbitrary topology, and have no resolution limitations. Saito et al. [54] introduce deep implicit functions for clothed 3D human reconstruction from RGB images and, later [55], they significantly improve 3D geometric details. The estimated shapes align well to image pixels. However, their shape reconstruction lacks regularization, and often produces artifacts like broken or disembodied limbs, missing details, or geometric noise. He et al. [22] add a coarse-occupancy prediction branch, and Li et al. [37] and Dong et al. [20] use depth information captured by an RGB-D camera to further regularize shape estimation and provide robustness to pose variation. Li et al. [35,36] speed up inference through an efficient volumetric sampling scheme. A limitation of all above methods is that the estimated 3D humans cannot be reposed, because implicit shapes (unlike statistical models) lack a consistent mesh topology, a skeleton, and skinning weights. To address this, Bozic et al. [14] infer an embedded deformation graph to manipulate implicit functions, while Yang et al. [65] also infer a skeleton and skinning fields.
Statistical models & implicit functions. Mesh-based statistical models are well regularized, while deep implicit functions are much more expressive. To get the best of both worlds, recent methods [10,11,24,70] combine the two representations. Given a sparse point cloud of a clothed person, IPNet [10] infers an occupancy field with body/clothing layers, registers SMPL to the body layer with inferred body-part segmentation, and captures clothing as offsets from SMPL to the point cloud. Given an RGB image of a clothed person, ARCH [24] and ARCH++ [23] reconstruct 3D human shape in a canonical space by warping query points from the canonical to the posed space, and projecting them onto the 2D image space. However, to train these models, one needs to unpose scans into the canonical pose with an accurately fitted body model; inaccurate poses cause artifacts. Moreover, unposing clothed scans using the "undressed" model's skinning weights alters shape details. For the same RGB input, Zheng et al. [69,70] condition the implicit function on a posed and voxelized SMPL mesh for robustness to pose variation and reconstruct local details from the image pixels, similar to PIFu [54]. However, these methods are sensitive to global pose, due to their 3D convolutional encoder. Thus, for training data with limited pose variation, they struggle with out-of-distribution poses and in-the-wild images.
Positioning ICON w.r.t. related work. ICON combines the statistical body model SMPL with an implicit function, to reconstruct clothed 3D human shape from a single RGB image. SMPL not only guides ICON's estimation, but is also optimized "in the loop" during inference to enhance its pose accuracy. Instead of relying on the global body features, ICON exploits local body features that are agnostic to global pose variations. As a result, even when trained on heavily limited data, ICON achieves state-of-the-art performance and is robust to out-of-distribution poses. This work links monocular 3D clothed human reconstruction to scan/depth based avatar modeling algorithms [16,19,40,42,56,60,62].

Method
ICON is a deep-learning model that infers a 3D clothed human from a color image. Specifically, ICON takes as input an RGB image with a segmented clothed human (following the suggestion of PIFuHD's repository [49]), along with an estimated human body shape "under clothing" (SMPL), and outputs a pixel-aligned 3D shape reconstruction of the clothed human. ICON has two main modules (see Fig. 3) for: (1) SMPL-guided clothed-body normal prediction and (2) local-feature based implicit surface reconstruction.

Body-guided normal prediction
Inferring full-360 • 3D normals from a single RGB image of a clothed person is challenging; normals for the occluded parts need to be hallucinated based on the observed parts. This is an ill-posed task and is challenging for deep networks. Unlike model-free methods [26,55,59], ICON takes into account a SMPL [38] "body-under-clothing" mesh to reduce ambiguities and guide front and (especially) back clothed-body normal prediction. To estimate the SMPL mesh M(β, θ) ∈ R N ×3 from image I, we use PyMAF [68] due to its better mesh-to-image alignment compared to other methods. SMPL is parameterized by shape, β ∈ R 10 , and pose, θ ∈ R 3×K , where N = 6, 890 vertices and K = 24 joints. ICON is also compatible with SMPL-X [48].
Under a weak-perspective camera model, with scale s ∈ R and translation t ∈ R 3 , we use the PyTorch3D [51] dif- ferentiable renderer, denoted as DR, to render M from two opposite views, obtaining "front" (i.e., observable side) and "back" (i.e., occluded side) SMPL-body normal maps We train the normal networks, G N , with the following loss: where , is a loss (L1) between ground-truth and predicted normals (the two G N in Fig. 3 have different parameters), and L VGG is a perceptual loss [28] weighted by λ VGG . With only L pixel , the inferred normals are blurry, but adding L VGG helps recover details.
Refining SMPL. Intuitively, a more accurate SMPL body fit provides a better prior that helps infer better clothed-body normals. However, in practice, human pose and shape (HPS) regressors do not give pixel-aligned SMPL fits. To account for this, during inference, the SMPL fits are optimized based on the difference between the rendered SMPL-body normal maps, N b , and the predicted clothed-body normal maps, N c , as shown in Fig. 4. Specifically we optimize over SMPL's shape, β, pose, θ, and translation, t, parameters to minimize: where L N diff is a normal-map loss (L1), weighted by λ N diff ; L S diff is a loss (L1) between the silhouettes of the  SMPL body normal-map S b and the human mask S c segmented [52] from I. We ablate L N diff , L S diff in Appx Refining normals. The normal maps rendered from the refined SMPL mesh, N b , are fed to the G N networks. The improved SMPL-mesh-to-image alignment guides G N to infer more reliable and detailed normals N c . Refinement loop. During inference, ICON alternates between: (1) refining the SMPL mesh using the inferred N c normals and (2) re-inferring N c using the refined SMPL. Experiments show that this feedback loop leads to more reliable clothed-body normal maps for both (front/back) sides.

Local-feature based implicit 3D reconstruction
Given the predicted clothed-body normal maps, N c , and the SMPL-body mesh, M, we regress the implicit 3D surface of a clothed human based on local features F P : where F s is the signed distance from a query point P to the closest body point P b ∈ M, and F b n is the barycentric surface normal of P b ; both provide strong regularization against self occlusions. Finally, F c n is a normal vector extracted from N c front or N c back depending on the visibility of P b : where π(P) denotes the 2D projection of the 3D point P. Please note that F P is independent of global body pose. Experiments show that this is key for robustness to out-of-distribution poses and efficacy w.r.t. training data.
We feed F P into an implicit function, IF, parameterized by a Multi-Layer Perceptron (MLP) to estimate the occupancy at point P, denoted as o(P). A mean squared error loss is used to train IF with ground-truth occupancy, o(P). Then the fast surface localization algorithm [36,39] is used to extract meshes from the 3D occupancy inferred by IF.

Baseline models
We compare ICON primarily with PIFu [54] and PaMIR [70]. These methods differ from ICON and from each other w.r.t. the training data, the loss functions, the network structure, the use of the SMPL body prior, etc. To isolate and evaluate each factor, we re-implement PIFu and PaMIR by "simulating" them based on ICON's architecture. This provides a unified benchmarking framework, and enables us to easily train each baseline with the exact same data and training hyper-parameters for a fair comparison. Since there might be small differences w.r.t. the original models, we denote the "simulated" models with a "star" as: where f 2D denotes the 2D image encoder, f 3D denotes the 3D voxel encoder, V denotes the voxelized SMPL, O denotes the entire predicted occupancy field, and γ is the mesh-based local feature extractor described in Sec. 3.2. The results are summarized in Tab. 2-A, and discussed in Sec. 4.3-A. For reference, we also report the performance of the original PIFu [54], PIFuHD [55], and PaMIR [70]; our "simulated" models perform well, and even outperform the original ones.

Datasets
Several public or commercial 3D clothed-human datasets are used in the literature, but each method uses different subsets and combinations of these, as shown in Tab. 1.
Training data. To compare models fairly, we factor out differences in training data as explained in Sec. 4.1. Following previous work [54,55], we retrain all baselines on the same 450 Renderpeople scans (subset of AGORA). Methods that require the 3D body prior (i.e., PaMIR, ICON) use the SMPL-X meshes provided by AGORA. ICON's G N and IF modules are trained on the same data.
Testing data. We evaluate primarily on CAPE [41], which no method uses for training, to test their generelizability. Specifically, we divide the CAPE dataset into the "CAPE-FP" and "CAPE-NFP" sets that have "fashion" and "non-fashion" poses, respectively, to better analyze the generalization to complex body poses; for details on data splitting please see Appx To evaluate performance without a domain gap between train/test data, we also test all models on "AGORA-50" [54,55], which contains 50 samples from AGORA that are different from the 450 used for training.
Generating synthetic data. We use the OpenGL scripts of MonoPort [36] to render photo-realistic images with dynamic lighting. We render each clothed-human 3D scan (I and N c ) and their SMPL-X fits (N b ) from multiple views by using a weak perspective camera and rotating the scan in front of it. In this way we generate 138, 924 samples, each containing a 3D clothed-human scan, its SMPL-X fit, an RGB image, camera parameters, 2D normal maps for the scan and the SMPL-X mesh (from two opposite views) and SMPL-X triangle visibility information w.r.t. the camera.

Evaluation
We use 3 evaluation metrics, described in the following: "Chamfer" distance. We report the Chamfer distance between ground-truth scans and reconstructed meshes. For this, we sample points uniformly on scans/meshes, to factor out resolution differences, and compute average bi-directional point-to-surface distances. This metric captures large geometric differences, but misses smaller geometric details.   "P2S" distance. CAPE has raw scans as ground truth, which can contain large holes. To factor holes out, we additionally report the average point-to-surface (P2S) distance from scan points to the closest reconstructed surface points. This metric can be viewed as a 1-directional version of the above metric. "Normals" difference. We render normal images for reconstructed and ground-truth surfaces from fixed viewpoints (Sec. 4.2, "generating synthetic data"), and calculate the L2 error between them. This captures errors for high-frequency geometric details, when Chamfer and P2S errors are small.
A. ICON -vs-SOTA. ICON outperforms all original state-of-the-art (SOTA) methods, and is competitive to our "simulated" versions of them, as shown in Tab. 2-A. We use AGORA's SMPL-X [47] ground truth (GT) as a reference. We notice that our re-implemented PaMIR * outperform the SMPL-X GT for images with in-distribution body poses ("AGORA-50" and "CAPE-FP"), However, this is not the case for images with out-of-distribution poses ("CAPE-NFP"). This shows that, although conditioned on GT SMPL-X fits, PaMIR * is still sensitive to global body pose due to its global feature encoder, and fails to generalize to out-of-distribution poses. On the contrary, ICON generalizes well to out-of-distribution poses, because its local features are independent from global pose (see Sec. 3.2).
B. Body-guided normal prediction. We evaluate the conditioning on SMPL-X-body normal maps, N b , for guiding inference of clothed-body normal maps, N c (Sec. 3.1). Table 2-B shows performance with ("ICON") and without ("ICON N † ") conditioning. With no conditioning, errors on "CAPE" increase slightly. Qualitatively, guidance by body normals heavily improves the inferred normals, especially for occluded body regions; see We use a 2-stack hourglass model [25], whose receptive field expands to 46% of the image size. This takes a large image area into account and produces features sensitive to global body pose. This worsens reconstruction performance for out-of-distribution poses, such as in "CAPE-NFP". For an evaluation of PaMIR's receptive field size, see Appx We compare ICON to state-of-the-art (SOTA) models for a varying amount of training data in Fig. 6. The "Dataset scale" axis reports the data size as the ratio w.r.t. the 450 scans of the original PIFu methods [54,55]; the left-most side corresponds to 56 scans and the right-most side corresponds to 3, 709 scans, i.e., all the scans of AGORA [47] and THuman [71]. ICON consistently outperforms all methods. Importantly, ICON achieves SOTA performance even when trained on just a fraction of the data. We attribute this to the local nature of ICON's point features; this helps ICON generalize well in the pose space and be data efficient.  Figure 6. Reconstruction error w.r.t. training-data size. "Dataset size" is defined as the ratio w.r.t. the 450 scans used in [54,55]. The "8x" setting is all 3, 709 scans of AGORA [47] and THuman [71].
D. Robustness to SMPL-X noise. SMPL-X estimated from an image might not be perfectly aligned with body pixels in the image. However, PaMIR and ICON are conditioned on this estimation. Thus, they need to be robust against various noise levels in SMPL-X shape and pose. To evaluate this, we feed PaMIR * and ICON with groundtruth and perturbed SMPL-X, denoted with () and () in Tab. 2-A,D. ICON conditioned on perturbed () SMPL-X produces larger errors w.r.t. conditioning on ground truth (). However, adding the body refinement module ("ICON +BR") of Sec. 3.1, refines SMPL-X and improves performance. As a result, "ICON +BR" conditioned on noisy SMPL-X () performs comparably to PaMIR * conditioned on ground-truth SMPL-X (); it is slightly worse/better for in-/out-of-distribution poses.

Reconstruction from in-the-wild images
We collect 200 in-the-wild images from Pinterest that show people performing parkour, sports, street dance, and kung fu. These images are unseen during training. We show qualitative results for ICON in Fig. 8a and comparisons to SOTA in Fig. 2; for more results see our video and Appx To evaluate the perceived realism of our results, we compare ICON to PIFu * , PaMIR * , and the original PIFuHD [55] in a perceptual study. ICON, PIFu * and PaMIR * are trained on all 3, 709 scans of AGORA [47] and THuman [71] ("8x" setting in Fig. 6). For PIFuHD we use its pre-trained model. In the study, participants were shown an image and either a rendered result of ICON or of another method. Participants were asked to choose the result that best represents the shape of the human in the image. We report the percentage of trails in which participants preferred the baseline methods over ICON in Tab Table 3. Perceptual study. Numbers denote the chance that participants prefer the reconstruction of a competing method over ICON for in-the-wild images. ICON is judged significantly more realistic.

Animatable avatar creation from video
Given a sequence of images with the same subject in various poses, we create an animatable avatar with the help of SCANimate [56]. First, we use ICON to reconstruct a 3D clothed-human mesh per frame. Then, we feed these meshes to SCANimate. ICON's robustness to diverse poses enables us to learn a clothed avatar with pose-dependent clothing deformation. Unlike raw 3D scans, which are taken with multi-view systems, ICON operates on a single image and its reconstructions are more reliable for observed body regions than for occluded ones. Thus, we reformulate the loss of SCANimate to downweight occluded regions depending on camera viewpoint. Results are shown in Fig. 1 and Fig. 8b; for animations see the video on our webpage.

Conclusion
We have presented ICON, which robustly recovers a 3D clothed person from a single image with accuracy and realism that exceeds prior art. There are two keys: (1) Regularizing the solution with a 3D body model while optimizing that body model iteratively. (2) Using local features to eliminate spurious correlations with global pose. Thorough ablation studies validate these choices. The quality of results is sufficient to build a 3D avatar from monocular image sequences. Limitations and future work. Due to the strong body prior exploited by ICON, loose clothing that is far from the body may be difficult to reconstruct; see Fig. 7. Although ICON is robust to small errors of body fits, significant failure of body fits leads to reconstruction failure. Because it is trained on orthographic views, ICON has trouble with strong perspective effects, producing asymmetric limbs or anatomically improbable shapes. A key future application is to use images alone to create a dataset of clothed avatars. Such a dataset could advance research in human shape generation [15], be valuable to fashion industry, and facilitate graphics applications.  Possible negative impact. While the quality of virtual humans created from images is not at the level of facial "deep fakes", as this technology matures, it will open up the possibility for full-body deep fakes, with all the attendant risks. These risks must also be balanced by the positive use cases in entertainment, tele-presence, and future metaverse applications. Clearly regulation will be needed to establish legal boundaries for its use. In lieu of societal guidelines today, we have made our code available with an appropriate license. Disclosure. https://files.is.tue.mpg.de/black/CoI CVPR 2022.txt

Appendices
We provide more details for the method and experiments, as well as more quantitative and qualitative results, as an extension of Sec. 3, Sec. 4 and Sec. 5 of the main paper.
Discussion on simulated data. The wide and loose clothing in CLOTH3D++ [9,43] demonstrates strong dynamics, which would complement commonly used datasets of commercial scans. Yet, the domain gap between CLOTH3D++ and real images is still large. Moreover, it is unclear how to train an implicit function from multi-layer non-watertight meshes. Consequently, we leave it for future research.
To statistically analyze the necessity of L N diff and L S diff in Eq. (4), we do a sanity check on AGORA's validation set. Initialized with different pose noise, s θ (Eq. (8)), we optimize the {θ, β, t} parameters of the perturbed SMPL by minimizing the difference between rendered SMPL-body normal maps and ground-truth clothed-body normal maps for 2K iterations. As Fig. 9 shows, L N diff + L S diff always leads to the smallest error under any noise level, measured by the Chamfer distance between the optimized perturbed SMPL mesh and the ground-truth SMPL mesh.

A.3. Perceptual study (Tab. 3)
Reconstruction on in-the-wild images. We perform a perceptual study to evaluate the perceived realism of the reconstructed clothed 3D humans from in-the-wild images. ICON is compared against 3 methods, PIFu [54], PIFuHD [55], and PaMIR [70]. We create a benchmark of 200 unseen images downloaded from the internet, and apply all the methods on this test set. All the reconstruction results are evaluated on Amazon Mechanical Turk (AMT), where each participant is shown pairs of reconstructions from ICON and one of the baselines, see Fig. 11. Each reconstruction result is rendered in four views: front, right, back and left. Participants are asked to choose the reconstructed 3D shape that better represents the human in the given color image. Each participant is given 100 samples to evaluate. To teach participants, and to filter out the ones that do not understand the task, we set up 1 tutorial sample, followed by 10 warm-up samples, and then the evaluation samples along with catch trial samples inserted every 10 evaluation samples. Each catch trial sample shows a color image along with either (1) the reconstruction of a baseline method for this image and the ground-truth scan that was rendered to create this image, or (2) the reconstruction of a baseline method for this image and the reconstruction for a different image (false positive), see Fig. 11c. Only participants that pass 70% out of 10 catch trials are considered. This leads to 28 valid participants out of 36 ones. Results are reported in Tab. 3. Normal map prediction. To evaluate the effect of the body prior for normal map prediction on in-the-wild images, we conduct a perceptual study against prediction without the body prior. We use AMT, and show participants a color image along with a pair of predicted normal maps from two methods. Participants are asked to pick the normal map that better represents the human in the image. Front-and backside normal maps are evaluated separately. See Fig. 12 for some samples. We set up 2 tutorial samples, 10 warm-up samples, 100 evaluation samples and 10 catch trials for each subject. The catch trials lead to 20 valid subjects out of 24 participants. We report the statistical results in Tab. 5. A chi-squared test is performed with a null hypothesis that the body prior does not have any influence. We show some results in Fig. 13, where all participants unanimously prefer one method over the other. While results of both methods look generally similar on front-side normal maps, using the body prior usually leads to better back-side normal maps.  Network architecture. Our body-guided normal prediction network uses the same architecture as PIFuHD [55], originally proposed in [28], and consisting of residual blocks with 4 down-sampling layers. The image encoder for PIFu * , PaMIR * , and ICON enc is a stacked hourglass [45] with 2 stacks, modified according to [25]. Tab. 6 lists feature dimensions for various methods; "total dims" is the neuron number for the first MLP layer (input).  Test-time details. During inference, to iteratively refine SMPL and the predicted clothed-body normal maps, we perform 50 iterations (each iteration takes ∼ 460 ms on a Quadro RTX 5000 GPU) and set λ N = 2.0 in Eq. (4). We conduct an experiment to show the influence of the number of iterations (#iterations) on accuracy, see Tab. 7. The resolution of the queried occupancy space is 256 3 . We use rembg 1 to segment the humans in in-the-wild images, and use Kaolin 2 to compute per-point the signed distance, F s , and barycentric surface normal, F b n . Discussion on receptive field size. As Tab. 8 shows, simply reducing the size of receptive field of PaMIR does not lead to better performance. This shows that our informative 3D features as in Eq. (6) and normal maps N c also play important roles for robust reconstruction. A more sophisticated design of smaller receptive field may lead to better performance and we would leave it for future research.  Table 9. Reconstruction error (cm) w.r.t. training-data size. "Training set scale" is defined as the ratio w.r.t. the 450 scans used in [54,55]. The "8x" setting is all 3, 709 scans of AGORA [47] and THuman [71]. Results outperform ground-truth SMPL-X, which has 1.158 cm and 1.125 cm for Chamfer and P2S in Tab. 2.

B. More Quantitative Results (Sec. 4.3)
C. More Qualitative Results (Sec. 5) Figures 14 to 16 show reconstructions for in-the-wild images, rendered from four different view points; normals are color coded. Figure 17 shows reconstructions for images with out-of-frame cropping. Figure 18 shows additional representative failures. The video on our website shows animation examples created with ICON and SCANimate.