UnrealStereo: A Synthetic Dataset for Analyzing Stereo Vision

Stereo algorithm is important for robotics applications, such as quadcopter and autonomous driving. It needs to be robust enough to handle images of challenging conditions, such as raining or strong lighting. Textureless and specular regions of these images make feature matching difficult and smoothness assumption invalid. It is important to understand whether an algorithm is robust to these hazardous regions. Many stereo benchmarks have been developed to evaluate the performance and track progress. But it is not easy to quantize the effect of these hazardous regions. In this paper, we develop a synthetic image generation tool and build a benchmark with synthetic images. First, we manually tweak hazardous factors in a virtual world, such as making objects more specular or transparent, to simulate corner cases to test the robustness of stereo algorithms. Second, we use ground truth information, such as object mask, material property, to automatically identify hazardous regions and evaluate the accuracy of these regions. Our tool is based on a popular game engine Unreal Engine 4 and will be open-source. Many publicly available realistic game contents can be used by our tool which can provide an enormous resource for algorithm development and evaluation.

Introduction
Stereo algorithms benefit enormously from benchmarks [1,17]. They provide quantitative evaluations to encourage competition and track progress. Despite the great progress over the past years, many challenges still remain unsolved, such as transparency, specularity and weakly textured regions. Image regions containing these challenges are very likely to cause the failure of an algorithm and are hazardous. These hazardous regions are small and uncommon, but critical in the real world. For example, the street * Indicates equal contributions. light is a thin object and covers a small portion in an image, but missing it will be a disaster for autonomous driving. Raining weather is not common, but on a rainy day, significant portions of the road will be very specular. These hazardous cases are summarized by CV-HAZOP [21]. Understanding the performance of these cases is critical for avoiding real-world failures, such as the accidental crash of an autonomous car. Stereo datasets include these challenges, but these hazardous regions are usually a small portion of an image and not very common in a dataset, so they won't affect the overall number a lot. This means achieving a good performance is not necessary corresponding to solving these challenges. To better understand the robustness of stereo algorithms, we built a benchmark UnrealStereo. We use this benchmark to study the impact of four hazardous factors on state-of-the-art algorithms. These factors are specularity, no texture, disparity jumps and transparency. The severity of hazardous factors is precisely controlled by making the material of objects more specular or transparent. Hazardous regions of an image are also provided to understand their impact on performance, see Fig. 1.
We want to understand the robustness of an algorithm when it is facing extreme hazardous conditions. To achieve this, we need to provide corner cases to throughly test it. Corner cases are situations when parameters are at extreme levels. They are widely used to test the correctness and robustness of an algorithm. The idea of using corner cases in computer vision is attractive. But two challenges make it hard for practice: First, it is difficult to capture corner cases. Strong lighting and rainy weather are rare and we can not control the weather to produce such images. Some factors, such as camera viewpoint and lighting, can be controlled with expensive lab setup [10,2], but other properties such as the transparency and specularity of material are hard to control. Also, these lab setups are oversimplified and expensive. In SYNTHIA [16], synthetic images under different weather conditions are produced. Synthetic images and ground truth are available, but the virtual scenes for ren-  [20] and error. The red color means large error. The first row is from KITTI [13] and the second row is our synthetic image. The error of a stereo algorithm is mainly due to hazardous factors, such as transparency and specularity. (b) Our tool can control the severity of these hazardous factors and automatically identify hazardous regions. The top row is a mask computed from object boundary, the bottom is the transparent and specular regions caused by window and wet ground. These two masks correspond to the image in the second row of (a).
dering are not publicly available. So it is not possible to do further tweaking with the rendering parameters to produce more challenging images. UnrealStereo provides tools to convert publicly available game contents into a dataset. Researchers can use images we generated or generate new corner cases to thoroughly test their algorithms.
Second, the hazardous regions of an image are unknown. When an image causes failure, we want to know which hazardous factors cause the failure. So it is useful to spot the hazardous regions of an image. But this requires information beyond disparity map. In CV-HAZOP [21], experts manually labeled the hazardous factors for demonstration, but this approach is hard to scale up. These regions can be automatically computed from information such as material property or object boundary. This information can be found in some datasets, but are absent from most stereo datasets. Lacking this information also limits the possibility of training algorithms that can utilize semantic information to solve these challenges. Computer graphics is a powerful tool for producing images and ground truth. Ground truth can be computed instead of being manually labeled or captured by special sensors. Our benchmark utilized the rich information from synthetic images to automatically identify hazardous regions of an image, which is hard for real images.
In this paper, we develop a data generation tool called UnrealStereo and use it to generate synthetic images for analyzing stereo algorithms. The contributions of Unreal-Stereo are: 1. Provide precise control over the hazardous factors of a scene, such as specularity and transparency of a material. 2. Provide rich ground truth such as object instance mask and material property to automatically identify hazardous regions of an image. 3. Enable the usage of rich game resources on the market to do virtual exper-iments. Using UnrealStereo, we first manually designed some cases which contain hazardous factors and studied the impact of these factors on algorithm performance. Then using the rich ground truth provided by UnrealStereo, we automatically spot hazardous regions of an image, such as object boundary, specular areas and analyze the performance in these regions. Using our benchmark, we can produce a report for each stereo algorithm, showing its robustness to hazardous factors. Reports for algorithms we evaluated will be included in the supplementary material.

Related Work
Many stereo datasets have been created for training and evaluating stereo algorithms.
Middlebury stereo dataset [17] is a widely used indoor scene dataset, which provides high-resolution stereo pairs with dense disparity ground truth. KITTI stereo dataset [6,13] is a benchmark consisting of urban video sequences while semi-dense disparity ground truth along with semantic labels are available. Due to demand of complex equipment and expensive human labor, these two real-world datasets have relatively small sizes. The larger KITTI dataset has about 400 labeled stereo pairs in total for public use. Another disadvantage of real-world dataset is the limited precision of 3D sensors and LIDAR prohibits high-quality ground truth. Recently, a large synthetic dataset [12] is rendered to leverage training deep convolutional neural networks (DCNN) end-to-end for stereo dense prediction.
Synthetic data attracts a lot of attention recently. The progress of computer graphics makes synthesizing realistic images much easier. The ability to get a large amount of images and ground truth is attractive. Synthetic data have been used in optical flow [3], semantic segmentation [15,16,5], stereo [12]. Images and ground truth are provided in these datasets, but the virtual scenes are not available for various reasons. So it is not possible to render new images or change the properties of these scenes. Instead of constructing virtual scenes from scratch, we use game projects that are publicly available in the marketplace. Our tool enables generating images and ground truth from these game projects. Anyone can use our tool to tweak these virtual scenes and produce more data. Many virtual scenes constructed by visual artists in the marketplace can be used. Different from rendering images from a commercial game binary [5], the ability to access 3D models enables us to modify the scene, generate more ground truth and do various virtual experiments. Different from Sintel [3] and Fly-ingthing3D [12], our approach utilizes more realistic 3D models and designed for real-time rendering. Previously, OVVV [18] attempted to use synthetic data for evaluating algorithms, but there is no easy-to-use platform available.
Understanding the impact of hazardous factors is important for computer vision. CV-HAZOP [21] proposes the idea of analyzing hazardous factors in an image. Their method requires manually annotating risk factors, such as specularity area, from images, which is difficult to perform and hard to scale up. Our synthetic pipeline can automatically identify these hazardous regions, enables large-scale analysis. The ability to control the severity of hazardous factors also helps us better understand the weakness of an algorithm.

Method
In this section, we describe how to construct virtual worlds for synthetic image generation. We tweak the severity of harzadous factors to produce corner cases to throughly test four state of the art algorithms. Hazardous regions are computed to analyze the effect of each hazardous factor.

Construct a Virtual World
A virtual world enables a mobile agent to navigate, take actions and interact. We constructed virtual worlds to analyze the robustness of stereo algorithms to hazardous factors. Our virtual worlds are created using the popular game engine Unreal Engine 4 (UE4). Instead of building them from scratch, we bought high-quality game projects and modified them to fit our requirements. The UE4 plugin UnrealCV [14] is used to capture images and ground truth from games. It is capable of generating images, depth and object instance mask. We built our tool on top of Unre-alCV. Our tool can produce accurate depth ground truth for transparent objects and support the extraction of material properties, such as the transparent or specular regions. This information enables us to provide the hazardous regions of an image in the benchmark. We also added the support for multiple cameras, so that two images can be generated simultaneously for stereo. All the game projects are available on the internet, so others can analyze other hazardous factors using our code and these projects.
To generate our test images, we use the scene editor of Unreal Engine to change the material properties of objects, making them more specular or transparent. We also put in occluder to increase the difficulty. Images are captured by moving the virtual camera according to a trajectory. The trajectory is created by two methods. First, we navigate our player and taking photos where we find interesting. Second, we use the cinematic tool of UE4 which is designed for movie production. Creating a few keyframes and intermediate frames will be interpolated. A stereo dual-camera system in standard form is put into our virtual world to produce rectified stereo pairs. Images and ground truth are generated for each frame in the camera trajectories, which is shown in Fig. 2. Given a rectified image pair, the goal of stereo matching is to compute the disparity d for each pixel in the reference image. The disparity is defined as the difference in horizontal location of a point in the left image and its corresponding one in the right. Then the conversion between depth z and disparity d is shown in the following relation z = f B d . where f is the focal length of the camera and B is the baseline that is the distance between the camera centers. The correctness of disparity is verified by warping the reference image according to its disparity map and comparing it with the target image.

Design Hazardous Cases for Evaluation
The task of stereo matching has several well-known difficulties including occlusion, specular surfaces, textureless regions, semi-transparent objects and regions with many jumps in disparity, i.e. frequent disparity discontinuities. We call these factors hazard as mentioned in [21]. Local stereo methods [7,11] suffer from matching ambiguities in weakly textured, specular and semi-transparent regions. Global methods [9,19,20,4], which impose smoothness constrains on neighboring pixels or superpixels, have the ability to overcome some of the above problems. Special efforts have been made to solve these difficulties. The authors of [8] leverage semantic informantions and 3D CAD models to resolve stereo ambiguities brought by specularity or no texture. An end-to-end trained DCNN based algorithm [12] performs well on specular regions of KITTI stereo 2015 [13] after finetuning on the training set. They exploit a fully convolutional siamese architecture to predict dense disparity map at a single forward pass.  To conduct an evaluation of these algorithms on the mentioned challenges, we manually designed and evaluate popular stereo algorithms on four hazardous factors i.e. specularity, transparency, no texture and rapid disparity jumps. While more than one hazardous factor above often cooccur in real world situations, our method enables to investigate the effect of each factor separately. All the four cases are established in a virtual living room while different objects are present. Fig. 3 shows the outlines of the four factors. These cases are important due to the following reasons. Firstly, these difficult cases are very common in the real world and often encountered by either indoor or outdoor stereo vision applications. Results on available stereo datasets such as KITTI [6,13] and Middlebury [17] show a great deteriorated performance for all the algorithms on these cases. Secondly, it is convenient for researchers to analyze the performance of their methods on simple cases where only one factor is present. Furthermore, choosing the solutions for related applications entails a comparison of stereo algorithms when dealing with these hazardous cases. Specifically, in some applications where specular or semitransparent objects appear frequently, the method with the lower error on such regions would be more likely to be chosen.
In the specularity challenge( Fig. 3(a)), we obtain the specular effect by diminish the roughness of specific materi-als. The major specular object is the screen of a TV towards which test stereo pairs are obtained from various viewpoints and distances. In the no texture challenge (Fig. 3(b)), the wall and the ceiling in the room are made textureless because they are the most common textureless objects in real world. To achieve texturelessness while keep reality, we do not directly remove the material of the walls but increase the smoothness of their material. A set of ten test image pairs are collected covering different orientations to the surrounding textureless walls. In the disparity jumping case (Fig. 3(c)), we place in the scene objects such as bamboos, fences and plants of various sizes and poses, which easily form many disparity discontinuities distributed within a small region. In the transparency case (Fig. 3(d)), we placed a transparent sliding wall in a room.
Not only can we construct ideal cases to evaluate stereo algorithm under single difficult condition, for specularity, semi-transparency and no texture, it is also possible to investigate how their performance degrades as the extent of challenge increases. Unlike in real world, adjusting the attribute of materials in virtual world is convenient and costless. This kind of analysis enables us to compare the robustness of stereo matching algorithms, thus making better choice on method selection depending on specific application. Fig. 4 shows the result of making the TV monitor more specular. It is shown that the state-of-art method MC-CNN [20] failed to understand the concept of TV and made a wrong estimation. Consequently, if a quadcopter mistakenly considers this TV as a window and tries to fly through it, this will be a failure. Details and results for evaluation on these cases are presented in Section 4.1.

Automatic Hazardous Region Discovery
It is important to pay more attention to hazardous regions of an image, because these regions are most likely to cause the failure of an algorithm. Manually designed hazardous cases are important for understanding an algorithm, but the size of manually designed data is not large enough.
On the other hand, due to the popularity of virtual reality, there are a lot of high quality virtual environments. These virtual environments can be purchased with a fair price (less than $100) or even free. These virtual environments can produce large amount of images. Moreover, we can also access other information, such as object instance mask, material property. We use these information to compute hazardous regions and to evaluate stereo algorithms on these hazardous regions.
Our rendering process produces extra information beyond depth information. These information includes: object instance mask, specular region, transparent region. Using these extra information, we can locate these hazardous regions automatically. Specifically, we add new focus to stereo method evaluation by providing binary masks for the these regions: 1. object boundary, 2. weakly textured material, 3. specular material, 4. transparent material. These regions are related to what we mention in Section 3.2 respectively. To evaluate disparity jumps hazard, boundary regions are exploited because theoretically frequent disparity discontinuities add difficulty to the computation of initial estimation, thus methods within traditional framework [19,4,20] fail on these regions especially at the boundary. Fig. 5 shows an example of these masks. All of above masks are generated automatically from the object segmentation ground truth. The object boundary mask is obtained by first computing a binary edge map from the object segmentation and dilating it with a radius of r d to include the vicinity of edges. For each object, we annotate its material information only once, before rendering process, then no more human effort is required to obtain corresponding masks. Figure 5. Mask that we compute from object mask and material property. From (a) to (d) are: mask for non-occluded region, boundary region, textureless region and specular region.

Experiment
We use three publicly available game scenes in our experiment. Screenshot of these games can be seen in Fig. 6. The comparison with other stereo datasets is shown in Table. 1. Besides depth and disparity ground truth, we provide extra information, such as: object segmentation mask and material properties. Another unique feature of our dataset is the properties of our virtual worlds can be tweaked with our tool and more challenging images can be produced. Instead of just providing an image dataset with fixed number of images, we provide a synthetic image generation tool. This tool can be used to design new hazardous cases, generate more images. More game scenes from the marketplace can be used in experiment.

Density
Scene Type Extra Info.
Virtual KITTI [5] dense outdoor seg. FlyingThings3D [12] dense toy, outdoor seg., material Ours dense indoor, outdoor seg., material √ Table 1. Comparison of stereo benchmarks and datasets. Our dataset and data generation tool can provide more information, such as: object segmentation mask, material properties. It can also produce new challenging images from a virtual world We test local method CoR [4], global methods on pixellevel MC-CNN [20] and superpixel-level SPS-St [19] as well as DCNN based methods DispNet [12]. Implementation from the authors of these methods are adopted. For completeness, we use two versions of [20], one is fast, the other is accurate. We also include a variant of [12] Disp-NetC which performs better on KITTI benchmark [6]. For the weights of the MC-CNN [20], we adopt the models used in their submission to KITTI. While for the end-to-end model [12], weights trained on the authors' original synthetic dataset are used. We also tried the fine-tuned model, but the performance is lower. In our experiments, we choose two commonly adopted error metrics, i.e., the end-point error which takes average of the absolute difference from ground truth and the 3 pixel error which is the percentage of pixels which deviate from ground truth more than 3 pixel. The 3 pixel error metric is proposed to cover calibration and laser measurement errors in real-world datasets and we use it here to compare results with them. The following evaluations are mainly focused on end-point error.

Hazardous Cases Evaluation
We choose 10 different viewpoint for each of the hazardous cases we designed, i.e. specular, semi-tranparent, textureless, and disparity jumps, covering both frontoparallel and slanted surfaces. At each viewpoint of hazardous scenes except disparity jumps case, we start from the easiest parameter settings that are roughest, opaque or well-textured and adjust the corresponding parameter step by step to increase the extent of hazard, resulting in 8 different levels of corresponding hazard per viewpoint and a total od 80 stereo image pairs. In specularity, semi-transparent and textureless cases, only regions labeled as these types are evaluated. For all the cases, we evaluate non-occluded regions because occlusion will introduce new problems to the task while we focus on analyzing one factor at a time. Table. 2 shows the performance on extreme condition. Fig. 7 displays the comparison of degradation of state-of-the-art stereo algorithms. Type of Hazard: As shown in Table 2, the slanted plane model based method SPS-St [19] performs best in specular case, closely followed by CoR [4] which is not purely local and reasons with overlapping multi-scale set of regions. As for high transparency, tremendously high test errors are shown, which means the little cue from the blur and specularity on transparent surfaces is insufficient for these method to spot the existence of a front surface. The success of DCNN models DispNet and DispNetC [12] on no texture and disparity jumping cases is evident. It is because they have large receptive fields to incorporate context information and do not explicitly impose smoothness constraints. The state-of-the-art MC-CNN [20] has good overall performance, but is less robust in extremely hazardous scenes -it easily suffers from ambiguity brought by no texture, specularity and transparency. Levels of Difficulty: A qualitative result of specularity is shown in Fig. 4. Both the qualitative and quantitative results in Fig. 7 demonstrate a degradation of stereo algorithms as the level of difficulty increases. In no texture case, end-toend DCNN based algorithms [12] have higher endurance for textureless objects, which is conceivably attributed to the ability of DCNN to utilize larger context information. MC-CNN [20] produces accurate result when texture is not so weak. However, as the texture become increasingly weak, when the matching network fails to produce a unambiguous data term, its local smoothness term, which encourages fronto-parallel surfaces, would leads to huge error in these regions. For specular regions, both SPS-St [19] and CoR [4] show higher robustness over other methods, indicating that the large support regions they use to some extent reduces their vulnerability to matching ambiguities. Results on transparency hazard are similar to those on specularity, because they bring the same problem for matching.

Automatic Hazardous Region Discovery
We utilize the game scenes to generate four binocular video sequences with disparity ground truth and hazardous regions, each of which has approximately 500 frames and includes several kinds of hazardous cases. These sequences are described as follows: RealisticRendering: Indoor environment of a living room with common furniture, plants as well as specular surface of a TV screen. UnrealMiddlebury: Indoor scene similar to Middlebury [17]. Objects of various shapes are placed together, which increases the difficulty with more occlusion and disparity discontinuity. SunTemple: Large indoor environment with few objects and no particular hazard being present. UrbanCity: A KITTI like urban scene sequence. The water on the roads simulates the driving condition after a heavy rain.
Stereo algorithms are tested on each video sequence and errors averaged within each sequence are reported. Table 3 shows the evaluation based on harzardous regions. The re-  sults indicate that the boundary and specular regions obtained automatically are indeed hazardous to stereo algorithms.
For overall performance in full and non-occluded regions, results on UrbanCity is consistent with the ranking on KITTI [6] both in terms of end-point and 3 pixel error metric, which indicates the authenticity of our data rendered from virtual scenes. Moreover, for specular regions, results on RealisticRendering and UrbanCity also agree with those on KITTI, except for the performance of DispNetC [12]. For Middlebury like scene in UnrealMiddlebury, significant deterioration of performance is observed. Therefore, the frequent disparity jumps and the transformation of shape due to viewpoint change significantly influences the difficulty of stereo matching. In large indoor environment as SunTemple, the lack of hazards results in relatively low errors. The top ranking method on KITTI MC-CNN [20] outperforms other method as expected.
On the boundary, errors higher than non-occluded regions are observed for all algorithms, which indicates that boundary region is a hazardous factor. This suggest that current stereo methods have difficulty predicting edgepreserved disparity maps, probably due to unsuccessful matching or the failure of segmentation to align well with the boundaries.
Although generated in much more complex environment, most of the results on hazardous regions still coin-cide with the manually designed cases, i.e. robustness of stereo algorithms varies under different levels of hazardous factors. For specularity, SPS-St [19] and CoR [4] outperform other methods on RealisticRendering and UrbanCity, while MC-CNN [20] performs best on SunTemple. From Fig. 7 we find that they coincide respectively with high and low specularity conditions. For textureless materials, results on UrbanCity match the weakest textured case, where endto-end DCNN based models perform best, while the other three could be interpreted as cases with evident texture. [19] and CoR [4] have similar performance both in designed case and rendered natural scenes. The latter uses the former to compute an initial semi-dense estimation of disparity and a multi-scale aggregation mechanism to further refine the result. In scenes with rich hazardous factors, e.g. RealisticRendering, UnrealMiddlebury and UrbanCity, adopting this mechanism results in better accuracy, while in simple scenes, e.g. SunTemple, no such gain in performance is observed.

SPS-St
The performance of DCNN based models [12] drops significantly on these rendered video sequences except for SunTemple. Since DCNN models highly hinges on training data, whether training data includes hazardous cases could determine the respective performance. Therefore, for DCNN models, finetuning becomes important when transfer from one dataset to another.
The state-of-the-art pixel-level global method MC-  Table 3. Performance of state-of-the-art stereo algorithms on rendered sequences. This table shows the disparity errors averaged over all test images in each video sequence. Errors in total image (Full), non-occluded regions, boundaries, textureless materials and specular regions in terms of end-point error (EPE) and ≥ 3 pixel error are evaluated by applying the masks proposed in Section 3.3.
CNN [20] receives best overall performance. However, it easily suffer from ambiguities brought by specularity and transparency. The convolutional neural network (CNN) for patch matching provide better initial disparity estimation on unambiguous regions, meanwhile, make the whole algorithm more vulnerable to specularity and transparency.
The relation of the two error metrics is worth noticing. End-point error (EPE) measures the error in average, while 3 pixel error measures the percentage of incorrect pixels. They are not in agreement in some cases, which could reveal characteristics of some algorithms. Specifically, for DCNN based method, significantly higher 3 pixel errors is observed. The L1 loss function used in the training process of DCNN models encourages lower EPE, which could account for the discrepancy.

Conclusion
In this paper, we present a data generation tool Unreal-Stereo to generate synthetic images to create a stereo bench-mark. We analyzed the effect of four hazardous factors on state-of-the-art algorithms. They are specularity, no texture, disparity jumps and transparency. Each factor is precisely controlled to see its impact. They are tuned to an extreme level to produce corner cases. We also tested these algorithms on three realistic virtual scenes. Hazardous regions of each image are automatically labeled from ground truth such as object mask and material properties. We found that the state-of-the-art method MC-CNN [20] outperforms others in general, but lack robustness in hazardous cases especially specular regions. DCNN based method [12] exhibits interesting properties due to the awareness of larger context.
Our data generation tool can be used to produce more challenging images and compatible with publicly available high-quality game models. This makes our tool capable for many applications. In our future work, we will extend our platform to include more hazardous factors such as the ratio of occlusion and analyze more computer vision problems. It is also interesting to explore the rich ground truth we gen-erate, such as object mask and material properties. This semantic information enables the development of stereo algorithms that utilizes high-level knowledge.


Detailed Garment Recovery from a Single-View Image

Most recent garment capturing techniques rely on acquiring multiple views of clothing, which may not always be readily available, especially in the case of pre-existing photographs from the web. As an alternative, we pro- pose a method that is able to compute a rich and realistic 3D model of a human body and its outfits from a single photograph with little human in- teraction. Our algorithm is not only able to capture the global shape and geometry of the clothing, it can also extract small but important details of cloth, such as occluded wrinkles and folds. Unlike previous methods using full 3D information (i.e. depth, multi-view images, or sampled 3D geom- etry), our approach achieves detailed garment recovery from a single-view image by using statistical, geometric, and physical priors and a combina- tion of parameter estimation, semantic parsing, shape recovery, and physics- based cloth simulation. We demonstrate the effectiveness of our algorithm by re-purposing the reconstructed garments for virtual try-on and garment transfer applications, as well as cloth animation for digital characters.

INTRODUCTION
Retail is a multi-trillion dollar business worldwide, with the global fashion industry valued at $3 trillion [FashionUnited 2016]. Approximately $1.6 trillion of retail purchasing in 2015 was done via online e-commerce sales, with growth rates in the double digits [Lindner 2015]. Thus, enabling better online apparel shopping experiences has the potential for enormous economic impact. Given the worldwide demand for fashion and the obvious impact of this demand on the apparel industry, technology-based solutions have recently been proposed, a few of which are already in use by commercial vendors. For example, there are several computer-aided design (CAD) software systems developed specifically for the apparel industry. The apparel CAD industry has focused predominantly on sized cloth-pattern development, pattern-design CAD software, 3D draping preview, and automatic generation of 2D clothing patterns from 3D body scanners or other measurement devices. Some of the leading apparel CAD companies include Gerber Technology, Lectra, Optitex, Assyst, StyleCAD, Marvelous Designer, Clo-3D and Avametric, etc. Unfortunately, developing these systems requires careful and lengthy design by an apparel expert.
More recently, there have been efforts to develop virtual try-on systems, such as triMirror, that allow users to visualize what a garment might look like on themselves before purchasing. These methods enable 3D visualization of various garments, fast animation of dynamic cloth, and a quick preview of how the cloth drapes on avatars as they move around. However, the capabilities of these new systems are limited, especially in terms of ease-of-use and applicability. Many of the virtual try-on systems use simple, fast imagebased or texture-based techniques for a fixed number of avatar poses. They do not typically account for the effects on fabric materials of different conditions (e.g. changing weather, varying poses, weight fluctuation, etc). Furthermore, almost all of these virtual tryon systems assume either that the user selects one of a pre-defined set of avatars or that accurate measurements of their own bodies have been captured via 3D scans.
In this work, we consider the problem of recovering detailed models of garments from a single-view image. Such a capability enables users to virtually try on garments given only a single photograph of themselves wearing clothing. Instead of representing the clothed human as a single mesh Li et al. 2012], we define a separate mesh for a person's clothing, allowing us to model the rich physical interactions between clothing and the human body. This approach also helps capture occluded wrinkles in clothing that are caused by various sources, including garment design that incorporates pleats, cloth material properties that influence the drape of the fabric, and the underlying human body pose and shape. Figure 1 illustrates some results generated by our system. In addition to virtual try-on applications, broader impacts in graphics include improving the accuracy of clothing models for animated characters, with the potential to further increase the visual realism of digital human models that already incorporate bodydependent priors for hair [Chai et al. 2012], face [Cao et al. 2013], skin [Nagano et al. 2015], and eyeballs [Bérard et al. 2014].
With limited input from a single-view image, we constrain the problem's solution space by exploiting three important priors. The first prior is a statistical human body distribution model constructed from a (naked) human body data set. This statistical model is used for extracting and matching the human body shape and pose in a given input image. The second prior is a collection of all sewing patterns of various common garment types, such as skirts, pants, shorts, t-shirts, tank tops, and dresses, from a database of all garment templates. Finally, the third prior is a set of all possible configurations and dynamical states of garments governed by their respective constitutive laws and modeled by a physically-based cloth simulation. Simulation helps provide additional 3D physical constraints lacking in a 2D image.  ] c , a human-body dataset, and a garment-template database as input. We preprocess the input data by performing garment parsing, sizing and features estimation, and human-body reconstruction. Next, we recover an estimated garment described by the set of garment parameters, including fabric material, design pattern parameters, sizing and wrinkle density, as well as the registered garment dressed on the reconstructed body.
Our method proceeds as follows. To construct an accurate body model, the user indicates 14 joint positions on the image and provides a rough sketch outlining the human body silhouette. (This step can also be automated using image processing and body templates for standard unoccluded poses.) From this information, we use a statistical human model to automatically generate a human body mesh for the image. To estimate the clothing model, we first compute a semantic parse of the garments in the image to identify and localize depicted clothing items. This semantic segmentation is computed automatically using a data-driven method for clothing recognition [Yamaguchi et al. 2013]. We then use the semantic parsing to extract garment sizing information, such as waist girth, skirt length and so on, which are then used to map the depicted garments onto the existing garment templates and adjust the sewing patterns based on the extracted parameters. We also analyze the segmented garments to identify the location and density of wrin-kles and folds in the recovered garments, which are necessary for estimating material properties of the garments for virtual try-on.
Once we have obtained both the body and clothing models, we perform an image-guided parameter identification process, which optimizes the garment template parameters based on the reconstructed human body and image information. We fit our 3D garment template's surface mesh onto the human body to obtain the initial 3D garment, then jointly optimize the material parameters, the body shape, and the pose to obtain the final result. The flow chart of the overall process is shown in Fig. 2. Our main contributions include: -An image-guided garment parameter selection method that makes the generation of virtual garments with diverse styles and sizes a simple and natural task (Section 5); -A joint material-pose optimization framework that can reconstruct both body and cloth models with material properties from a single image (Section 6); -Application to virtual try-on and character animation (Section 7).

RELATED WORK
Our work is built on previous efforts in cloth modeling, human pose/shape recovery, garment capture from single-view images, and semantic parsing.
Cloth Modeling: Cloth simulation is a traditional research problem in computer graphics. Early work on cloth simulation includes [Weil 1986;Ng and Grimsdale 1996;Baraff and Witkin 1998;House and Breen 2000]. More recently, a number of methods were proposed to solve the complicated problems presented in cloth simulation, including collision detection [Govindaraju et al. 2007;Tang et al. 2009;Curtis et al. 2008], collision handling, friction handling [Bridson et al. 2002], strain limiting [Goldenthal et al. 2007;English and Bridson 2008;Thomaszewski et al. 2009;Wang et al. 2010] and remeshing [Narain et al. 2012].
Realistic wrinkle simulation is an important problem in realistic cloth modeling. Volino and Magnenat-Thalmann [1999] introduced a geometry-based wrinkle synthesis. Rohmer et al. [2010] presented a method to augment a coarse cloth mesh with wrinkles. Physically based cloth wrinkle simulation depends on an accurate model of the underlying constitutive law; different bending and stretching energy models for wrinkle simulation have been proposed [Bridson et al. 2003].
Garment modeling is built upon cloth simulation. It also needs to take into consideration the design and sewing pattern of the garment. Some methods start from the 2D design pattern [Protopsaltou et al. 2002;Decaudin et al. 2006;Berthouzoz et al. 2013] or 2D sketches [Turquin et al. 2007;Robson et al. 2011]. Other methods explore garment resizing and transfer from 3D template garments [Wang et al. 2005;Meng et al. 2012;Sumner and Popović 2004]. In contrast, our work synthesizes different ideas and extends these methods to process 2D input image and fluidly transfer the results to the simulation of 3D garments. We can also edit the 2D sewing patterns with information extracted from a single-view image, which can be used to guide the generation of garments of various sizes and styles.
Human Pose and Shape Recovering: Human pose and shape recovery from a single-view image has been extensively studied in computer vision and computer graphics. Taylor [2000] presented an articulated-body skeleton recovery algorithm from a single-view image with limited user input. Agarwal et al. [2006] proposed a learning-based method to recover human body poses from monocular images. Ye et al. [2014] applied a template-based method for real-time single RGBD image human pose and shape estimation. We refer readers to this survey on human motion recovery and analysis [Moeslund et al. 2006].
Human pose and shape recovery in computer graphics focus primarily on reconstructing muscle accurately and on watertight 3D human body meshes. A realistic 3D human body mesh is the basis for character animation. A human body mesh is required for the recovery of clothing with rich details. For human body mesh generation, we follow the previous data-driven methods, most of which are PCA based. These techniques use a set of bases to generate a variety of human bodies of different poses and shapes. Seo and Thalmann [2003] presented a method to construct human body meshes of different shapes. Following this work, Anguelov et al. [2005] introduced the SCAPE model, which can produce human body meshes of different poses and shapes. Using the SCAPE model, Balan et al. [2007] presented a method to recover detailed human shape and pose from images. Hasler et al. [2009] encode both human body shapes and poses using PCA and semantic parameters. Building upon these previous models, Zhou et al. [2010] proposed a method to recover the human body pose and shape from a singleview image.
Clothing Capturing: In the last decade, many methods have been proposed for capturing clothing from images or videos. Methods can be divided into two categories: marker-based and markerless. Most marker-based clothing capture methods require the markers to have been pre-printed on the surface of the cloth. Different kinds of markers have been used for the capture [Scholz and Magnor 2006;Hasler et al. 2006;Tanie et al. 2005;Scholz et al. 2005;White et al. 2007]. Markerless methods, which do not require pre-printed clothing markers, can be characterized into several categories of methods: single-view Jeong et al. 2015], depth camera based [Chen et al. 2015]; and multi-view methods [Popa et al. 2009]. Another method based on cloth contact friction design was proposed by Casati et. al. [2016].
These methods have some limitations, however, including inability to capture fine garment details and material properties, the loss of the original garment design, and complexity of the capturing process. In contrast, our method can retrieve the 2D design pattern with the individual measurements obtained directly from a single image. Using a joint human pose and clothing optimization method, our algorithm recovers realistic garment models with details (e.g. wrinkles and folds) and material properties.
Semantic Parsing: Semantic parsing is a well-studied problem in computer vision, where the goal is to assign a semantic label to every pixel in an image. Most prior work has focused on parsing general scene images [Long et al. 2015;Farabet et al. 2013;Pinheiro and Collobert 2014]. We work on the somewhat more constrained problem of parsing clothing in an image. To obtain a semantic parse of the clothing depicted in an image, we make use of the data-driven approach by Yamaguchi et al. [2013]. This method automatically estimates the human body pose from a 2D image, extracts a visual representation of the clothing the person is wearing, and then visually matches the outfit to a large database of clothing items to compute a clothing parse of the query image.

PROBLEM STATEMENT AND ASSUMPTIONS
In this section, we give the formal definition of the problem. The input to our system is an RGB image Ω. We assume the image is comprised of three parts: the background region Ω b , the foreground naked human body parts Ω h and the foreground garment Ω g , where Ω = Ω b ∪ Ω h ∪ Ω g . In addition, we assume that both the human body and the garment are in a statically stable physical state. Although this assumption precludes images capturing a fast moving human, it provides a crucial assumption for our joint optimization algorithm.

Problem:
Given Ω g , Ω h , how to recover -the garment described by a set of parameters < C, G, U, V >, -along with a set of parameters < θ, z > that encode human body pose and shape obtained from the image.
Garment: For the clothing parameters, C is the set of material parameters including stretching stiffness and bending stiffness coefficients; U is the 2D triangle mesh representing the garment's pattern pieces; and V is the 3D triangle mesh representation of the garment. For each triangle of the 3D garment mesh V, there is a corresponding one in the 2D space U. For each mesh vertex • x ∈ V, such as those lying on a stitching seam in the garment, there might be multiple corresponding 2D vertices u ∈ U. The parameter G is the set of parameters that defines the dimensions of the 2D pattern pieces. We adopt the garment sizing parameters based on classic sewing patterns [Barnfield 2012] shown in Fig. 3a, 3c and 3e, with the corresponding parameters defined in Fig. 3b, 3d, and 3f, respectively.
For example, we define the parameter G pants = < w 1 , w 2 , w 3 , w 4 , h 1 , h 2 , h 3 > for pants, where the first four parameters define the waist, bottom, knee and ankle girth and the last three parameters indicate the total length, back upper, and front upper length. For each basic garment category, we can manually define this set of parameters G. By manipulating the values of the parameters G, garments of different styles and sizes can be modeled: capri pants vs. full-length pants, or tight-fitting vs. loose and flowy silhouettes. We use the material model developed by Wang et al. [2011]. The material parameters C are the 18 bending and 24 stretching parameters.
Human Body: For the human body parameters, θ is the set of joint angles that together parameterize the body pose, and z is the set of semantic parameters that describe the body shape. We follow the PCA encoding of the human body shape presented in [Hasler et al. 2009]. The semantic parameters include gender, height, weight, muscle percentage, breast girth, waist girth, hip girth, thigh girth, calf girth, shoulder height, and leg length. Table I provides a list of formal definitions for the notation used in this paper. Table I. Notation and definition of our method.

NOTATION DEFINITION
C material property parameters of the garment G garment (sizing) parameters 2D triangle mesh representing garment pattern u vertex of the 2D garment pattern mesh V 3D triangle surface mesh of the garment V simulated 3D triangle surface mesh of the garment B body skeleton of 3D human body mesh B garment skeleton of 3D garment mesh x vertex of the 3D triangle mesh P average wrinkle density of the 2D segmented garment in the image k bending stiffness parameters w stretching stiffness parameters F deformation gradient of the deforming garment K garment 3D mesh curvature measurement S 2D garment silhouette Ψ bending energy of the garment Φ stretching energy of the garment υ ij rigging weights of the 3D garment mesh β joint angles of the skeleton of the 3D garment mesh θ joint angles of the human body mesh z semantic parameters of the human body mesh shape p vertex of the 3D human body mesh P 3D human body mesh Z i PCA basis of human body shape ω i rigging weights of the human body mesh D c garment database D h human body database

DATA PREPARATION
This section describes the data preprocessing step. We begin with the data representations for the garment and the human body, followed by a brief description of each preprocessing module.

Data Representations
The garment template database can be represented as a set where N is the number of garment templates. Each garment template consists of a 2D triangle mesh U representing the sewing pattern, a 3D mesh V, a set of dimension parameters G for each pattern piece, the skeleton B garment , and a set of material parameters C. The human body template database D h = {< θ j , z j , B body,j > |j ∈ 1, · · · , M } consists of M naked human body meshes with point to point correspondence. We use several human body datasets, including the SCAPE dataset [Anguelov et al. 2005], the SPRING dataset , the TOSCA dataset [Bronstein et al. 2008;Young et al. 2007;Bronstein et al. 2006], and the dataset from [Hasler et al. 2009]. Our garment template is defined in the same metric system as the human template to scale the garments during the initial registration. Each garment template and human template is rigged on a common skeleton with the same set of joints. Parameterized Garment Model: Given the garment template database D c , each vertex u of the 2D garment pattern mesh is computed as with g i as the i th 2D pattern sizing parameter in the set G, ν i is the weight associated with the vertex u and u 0 is the vertex position of the 2D garment pattern template. Parameterized Human Model: Given the body database D h , we extract a statistical shape model for human bodies. Under this model, each world space vertex position p on the human body is parameterized as which is a composition of a linear blend skinning model [Kavan et al. 2010] and an active shape model [Zhao et al. 2003]. Here ω i and B i are the set of weights and bones associated with the vertex p. T B i is the transformation matrix of bone B i . p 0 and Z i are the mean shape and active shape basis at the rest pose, respectively. The basis Z i is calculated by running PCA [Hasler et al. 2009] on D h .

Preprocessing
Our preprocessing step consists of: a) human body reconstruction to recover the human body shape and pose from the input image, b) garment parsing to estimate the locations and types of garments depicted in the image, and c) parameter estimation to compute the sizing and fine features of the parsed garments. Human Body Reconstruction: Our human body recovery relies on limited user input. The user helps us identify the 14 human body joints and the human body silhouette. With the identified joints, a human body skeleton is recovered using the method presented in [Taylor 2000]: the semantic parameters z are optimized to match the silhouette. In this step, we ignore the camera scaling factor.
Template sewing pattern and parameter space of a skirt, pants, and t-shirt. (a) The classic circle sewing pattern of a skirt. (b) Our parametric skirt template showing dashed lines for seams and the four parameters < l 1 , r 1 , r 2 , α >, in which parameter l 1 is related to the waist girth and parameter r 2 is related to the length of the skirt. (c) The classic pants sewing pattern. (d) Our parametric pants template with seven parameters < w 1 , w 2 , w 3 , w 4 , h 1 , h 2 , h 3 >.
Garment Parsing: We provide two options for garment parsing. The first uses the automatic computer vision technique presented in [Yamaguchi et al. 2013]. This approach combines global pretrained parse models with local models learned from nearest neighbors and transferred parse masks to estimate the types of garments and their locations on the person. The second option requires assistance from the user. Given the image Ω, we extract the clothing regions Ω b,h,g by performing a two-stage image segmentation guided by user sketch. In the first stage, a coarse region boundary is extracted using a graph cut algorithm [Li et al. 2004]. Then, the region is refined via re-clustering [Levin et al. 2008].
Image Information Extraction: Given the segmentation of the garment Ω g , the next step is to convert it to pixel-level garment silhouette S and compute the average wrinkle density P. Instead of using the wrinkle density for each part of the garment, the average wrinkle density encodes the overall material properties of it. We extract the average wrinkle density P from the garment images using an improved implementation of [Popa et al. 2009]. We first detect edges using Holistically-Nested edge detection [Xie and Tu 2015] and then smooth the edges by fitting them to low-curvature quadrics. We smooth edges split during the detection phase by merging those that have nearby endpoints and similar orientations. Finally, we form 2D folds by matching parallel edges. Edges not part of a pair are unlikely to contribute to a fold and are discarded. The average number of wrinkles per area is the average wrinkle density P.

Initial Garment Registration
Our initial clothing registration step aims to dress our template garment onto a human body mesh of any pose or shape. We optimize the vertex positions of the 3D mesh, x, of the template clothing based on the human body mesh parameters < θ, z >. In this step, we ignore the fit of the clothing on the human body (this step is intended to fix the 2D triangle mesh U). We follow the method proposed in [Brouet et al. 2012] for registering a template garment to a human body mesh with a different shape. However, their method is unable to fit the clothing to human meshes with varying poses; we extend their approach by adding two additional steps.
The first step requires the alignment of the joints Q c of the template garment skeleton with the joints Q h of the human body mesh skeleton, as shown in Fig. 4. Each joint q ∈ Q c of the garment has one corresponding joint t ∈ Q h of the human body mesh. We denote the number of joints of the garment as K c . This step is done by applying a rigid body transformation matrix T on the joint of the garment, where T minimizes the objective function Next, we need to fit this transformed 3D template garment mesh onto the human body mesh with pose described by parameter θ, the vector of the angles of the joints. Our template garment is then deformed according to θ. We denote the vector β as the joint angles of the template garment mesh. We set the value of the vector β i to the value of the corresponding joint angle θ i of the human body mesh. Then we compute the 3D garment template mesh such that it matches the pose of the underlying human body mesh according to this set of joint angles β by, where υ ij is the weight of bone B j on vertex x i and T B j is the transformation matrix of the bone B j . An example result is shown in Fig. 4c. The final step is to remove collisions between the garment surface mesh and the human body mesh. This step is similar to the ICP algorithm proposed by Li et al. [2008]. We introduce two constraints: rigidity and non-interception. The deformation of the clothing should be as-rigid-as-possible [Igarashi et al. 2005]. After this step, we have an initial registered garment with a 3D mesĥ V(T, θ) that matches the underlying human pose and is free of interpenetrations with the human body. We show our initial garment registration results in Fig. 5.

IMAGE-GUIDED PARAMETER IDENTIFICATION
In this section, we explain the step-by-step process of extracting garment material and sizing parameters < C, G > from an image.

Overview
Starting from our 2D triangle mesh U of the pattern pieces, we select garment parameters based on the sizing and detailed information < S, P > estimated from the source image. In this step, we adjust the garment material and sizing parameters < C, G > but fix the 3D meshV(T, θ) (computed from Sec. 4.3) to obtain the garment that best matches the one shown in the image. We need two specific pieces of information from the image: the pixel-level garment silhouette S and the average wrinkle density P of the clothing. For example, for a skirt, we need to estimate the waist girth and the length of the skirt from the image.
Using this information, we initialize the garment sizing parameters G. Based on the wrinkle information computed from the image,  we optimize both the fabric material parameters C and the sizing parameters of the garment pattern G.

Garment Types, Patterns, and Parameters
For basic garment types, such as skirts, pants, t-shirts, and tank tops, we use one template pattern for each. We modify the classic sewing pattern according to the parameters G. By adjusting the garment parameters G and fabric material parameters C, we recover basic garments of different styles and sizes. The classic circle skirt sewing pattern is shown in Fig. 3a. Our parametric space, which is morphed from this circle sewing pattern, is shown in Fig. 3b. For the skirt pattern, there are four parameters to optimize: G skirt =< l 1 , r 1 , r 2 , α >. The ratio between the parameter l 1 and r 2 is constrained by the waist girth and skirt length information extracted from the image. The other two parameters, r 1 and α, are constrained by the wrinkle density. With different garment parameters, skirts can vary from long to short, stiff to soft, and can incorporate more or fewer pleats, enabling us to model a wide variety of skirts from a single design template. Similarly for pants, the classic sewing pattern and our template pattern pieces are shown in Fig. 3c and Fig. 3d. There are seven parameters for the dimensions of the pants template: G pants =< w 1 , w 2 , w 3 , w 4 , h 1 , h 2 , h 3 > with the first four parameters describing the waist, bottom, knee, and ankle girth, and the last three parameters representing the total, back-upper and frontupper lengths. The t-shirt sewing pattern is shown in Fig. 3e, and our parametric t-shirt pattern is shown in Fig. 3f with the garment parameters G tshirt =< r, w 1 , w 2 , h 1 , h 2 , l 1 >. Among the parameters G tshirt , parameter r describes the neckline radius, w 1 describes the sleeve width, w 2 describes the shoulder width, h 1 describes the bottom length, h 2 describes the total length, and l 1 describes the length of the sleeve. Different sewing patterns result in very different garments. Traditional sewing techniques form skirt wrinkles by cutting the circular portion of the pattern. To simulate this process but make it generally applicable, we modify the parameter G, which achieves the same effect. In addition to the differences created by sewing patterns, professional garment designers also take advantage of cloth material properties to produce different styles of clothing. We tune the bending stiffness coefficients and stretching stiffness coefficients in C to simulate this cloth selection process.

From Wrinkle Density to Material Property
One of the key insights in this work is the identification of fabric materials based on wrinkles and folds, because different fabric stiffness produce varying wrinkle/folding patterns. We characterize the wrinkles and folds using their local curvatures. The first step is to map the wrinkle density P (computed in Sec. 4.2) to the average local curvature K.
We recover the garment material parameter C by minimizing the average local curvature differences between our recovered garment K(C, G) and the reference garment K(P) target The reference garment average local curvature K(P) target is computed by linear interpolation. We first approximate the average local curvature threshold for the sharp wrinkles and smooth folds. The average local curvature threshold for one sharp wrinkle is up to 10 5 m −1 and that for smooth folds is close to 10 −4 m −1 . Sharp wrinkles or large folds are determined by the density of the extracted 2D wrinkles. The density of the extracted 2D wrinkles ranges from 1 m −2 to 50 m −2 based on our observation. The interpolation process (with the linear interpolation function I) is with the linear interpolation function I(1) = 10 −4 and I(50) = 10 5 . Local curvature estimation of κ at each vertex is computed based on the bending between the two neighboring triangles sharing the same edge. For each vertex x of the two triangles that share an edge e, the local curvature κ is computed following the approach from Wang et al. [2011] and Bridson et al. [2003] κ = sin(α/2)(h 1 + h 2 ) −1 |e|x , where h 1 and h 2 are the heights of the two triangles that share the edge e and α is the supplementary angle to the dihedral angle between the two triangles. The corresponding bending force f bend for each vertex x is computed as where k is the bending stiffness coefficient. Stretching also affects the formation of wrinkles. Each triangle < u 0 , u 1 , u 2 > in the 2D template mesh is represented as , and each triangle in the 3D garment mesh ing forces f stretch are computed by differentiating the stretching energy Ψ, which depends on the stretching stiffness parameter w, the deformation gradient F = d m D −1 m , and the Green strain tensor The sizing and style of the garment described by the parameter G obtained from the parsed garment are matched by minimizing the silhouette which is a 2D polygon differences between our recovered garment S(C, G) and the reference garment silhouette S target The distance between two polygons is computed by summing up the distances between each point in polygon S(C, G) to the other polygon S target . To compute the 2D silhouette S(C, G), we first project the simulated garment 3D mesh V(C, G,V(T, θ)) onto the the 2D image with the projection matrix H. Then compute the 2D polygon enclosing the projected points. The process is expressed as with f as the method that convert the projected points to a 2D polygon. We ignore the camera scaling factor in this step since the input is a single-view image. It is natural to scale the recovered clothing and human body shape as a postprocessing step. Combining these two objectives, the objective (energy) function is expressed as

Optimization-based Parameter Estimation
The optimization is an iterative process (given in Algorithm 1), alternating between updates for the garment sizing and material parameters, G and C. We found that the value of the objective function is more sensitive to the cloth material properties C than to the garment parameter G, so we maximize the iterations when optimizing for C, fixing G. The optimization of parameter C is coupled with the cloth dynamics. The underlying cloth simulator is based on the method proposed in [Narain et al. 2012]. We drape the initial fitted garment onto the human body mesh. The garment is in the dynamic state and subject to gravity. We couple our parameter estimation with this physically based simulation process. Before the simulation, we change the cloth material parameters C so that when in static state the average of the local curvature κ matches the targeting threshold K target . That is to say, our optimizer minimizes K − K target 2 by changing the bending stiffness parameters k and stretching stiffness parameters w.
We apply the L-BFGS [Liu and Nocedal 1989] method for our material parameter optimization. When the clothing reaches a static state, the optimizer switches to optimizing parameter G. The optimizer for the parameter G is not coupled with the garment simulation. The objective function is evaluated when the clothing reaches the static state. We adopt the Particle Swarm Optimization (PSO) method [Kennedy 2010] for the parameter G optimization. The PSO method is known to be able to recover from local minima, making it the ideal method for some of the non-convex optimization problems.
We use 40 particles for the parameter estimation process. The alternating process usually converges after four steps. One example result of the garment parameter process is shown in Fig. 6.
We constrain the cloth material parameter space. We use the "Gray Interlock" presented in [Wang et al. 2011], which is composed of 60% cotton and 40% polyester, as the "softest" material, meaning it bends the easiest. We multiply the bending parameters of this material by 10 2 to give the "stiffest" material based on our experiments. Our solution space is constrained by these two materials, and we initialize our optimization with the "softest" material parameters.

Optimal Parameter Selection
The parameter identification step provides us with the initial recovered garment described by the set of material and sizing parameters < C , G >. Many realistic garment wrinkles and folds, however, are formed due to the underlying pose of the human body, especially wrinkles that are located around human joints. Therefore, in this step, we further refine our results by optimizing both the pose parameters of the human body θ and the material properties of the cloth C . The optimization objective for this step is The optimization process (shown in Algorithm 2) is similar to the garment parameter identification step, alternating between up-• dating the pose parameter θ and the material parameters C . We use Particle Swarm Optimization method [Kennedy 2010].
The objective function (Eqn. 13) is more sensitive to the pose parameter θ than to the material parameters C . We constrain the optimization space of parameter θ by confining the rotation axis to only the three primal axes. An example of our joint material-pose optimization method is shown in Fig. 7. while E joint > do // E joint as defined in Eqn. 13 3: Fix C , Optimize for θ using Particle Swarm Method 4: Fix θ, Optimize for C using Particle Swam Method returnC,θ 7: end procedure

Application to Image-Based Virtual Try-On
This joint material-pose optimization method can be applied directly to image-based virtual try-on. We first recover the pose and shape of the human body < θ, z > from the single-view image. Then we dress the recovered human body with the reconstructed garments <C,G,Ũ,V > from other images. We perform the initial garment registration step (Sec. 4.3) to fit the 3D surface mesĥ V onto the recovered human body < θ, z >.
Existing state-of-the-art virtual try-on rooms require a depth camera for tracking, and overlay the human body with the fitting garment [Ye et al. 2014]. Our algorithm, on the other hand, is able to fit the human body from a single 2D image with an optimized virtual outfit recovered from other images. We provide the optimized design pattern together with a 3D view of the garment fitted to the human body.
The fitting step requires iterative optimization in both the garment parameters and the human-body poses. As in a real fitting process, we vary the sizing of the outfits for human bodies of different sizes and shapes. When editing in parameter space using the methods introduced in the previous section, we ensure that the recovered garment can fit on the human body while minimizing the distortion of the original design. For each basic garment, we use one template pattern and the corresponding set of parameters. To preserve the garment design, we do not change the material properties of the fabric when virtually fitting the recovered garment to a new mannequin.

RESULTS AND DISCUSSION
We have implemented our algorithm in C++ and demonstrated the effectiveness of our approach throughout the paper. In this section, we show example results, performance, and comparisons to other garment recovery methods.

Garment Recovery Results
We show several examples of garment recovery from a single-view image. In Fig. 8 and Fig. 9, we show that our method can recover garments of different styles and materials. Fig. 10 demonstrates the effectiveness of our method for the recovery of partially occluded garments. It also shows that our recovered garment can be applied to human bodies in different poses.
Image-Based Garment Virtual Try-On: We show examples of our image-based garment virtual try-on method (Sec. 6.2) in Fig. 1 and Fig. 11. We can effectively render new outfits onto people from only a single input image.
Evaluation: We evaluate the accuracy of the recovered sizing parameters G and local curvature K using synthetic scenes. Each synthetic scene has two lighting conditions, mid-day, and sunset (shown in Fig. 12). We fix the both the extrinsic and the intrinsic camera parameters for scene rendering, and the garments are in static equilibrium. Through these ten test cases, we can best validate the accuracy and reliability of our method against different body poses and lighting conditions on T-shirts and pants, as the sizing and material parameters are known exactly and do not require noisy measurements and/or data fitting to derive a set of estimated/measured parameters (which are not likely to be 100% accurate) to serve as the ground truth. The evaluation result, after eliminating the camera scaling factor, is shown in Table II.
We found that the lighting conditions mainly affect the body silhouette approximation and the garment folding parsing, while the body skeleton approximation is affected by the pose. Overall, we achieve an accuracy of up to 90.2% for recovering the sizing parameters and 82% for recovering the material parameters for tshirts and pants under different body poses and lighting conditions. The accuracy is computed as the average accuracy for each parameter from the ground truth.
We evaluate the accuracy of the recovered material properties by measuring the difference between the ground truth and that of the recovered garment for both the mean curvature and the material parameters, as the accuracy of mean-curvature recovery also correlates with the accuracy of the material-parameter estimation. As shown in Table III, we are able to achieve an accuracy of up to 86.7% and 80.2%, respectively, for the recovery of mean curvatures and different material parameters for the skirt.

Comparison with Other Related Work
We compare our results with the multi-view reconstruction method CMP-MVS [Jancosek and Pajdla 2011] together with the structurefrom-motion framework [Wu 2011;. For a fair comparison, we apply smoothing [Taubin 1995] to the results of their work. Fig. 13 and Fig. 14 show that the garment recovered using our method is clean and comparable in visual quality to the recovered garments using multi-view methods. In addition, we are able to esti- We further compare the results of our work against two recent methods -one using 3D depth information and an expertdesigned 3D database [Chen et al. 2015], and the other using a large database of manually labeled garment images [Jeong et al. 2015]. Our method, which does not require depth information, an expert-designed 3D database, or a large manually labeled garment image database, achieves a comparable level of high accuracy to Chen et al. [2015] (see Fig. 15) and higher visual quality when  compared with Jeong et al. [2015] (see Fig. 16). In addition, our method is able to recover material and estimate sizing parameters directly from a given image.

Performance
We run our method on a desktop with an Intel(R) Core(TM) i7 CPU, 3.20GHz. For each garment, our pipeline takes on average 4 to 6 hours. The garment parameter identification (Sec. 5) and joint material-pose optimization (Sec. 6.1) takes around 60% − 80% of the entire process. The preprocessing step (Sec. 4.2) takes around 20% − 30%. The performance depends largely on the complexity    of the garment, the image quality, and how much the garment is occluded.

Discussions
Estimation of Material Parameters: Our material recovery method depends on the existence of wrinkles and folds of the garment. In cases where no or very few wrinkles or folds are present, other image features, such as textures and shading, would be required to identify the material properties. In most garments like shirts, skirts, or dresses, wrinkles and folds are common (especially around the joints or from the garment stylization), and can be highly informative with regards to garment material properties. Based on this observation, we are able to estimate material parameters as well as recover garment geometry from single-view images. This capability is one of our main objectives, and it is the key feature differentiating our work from existing techniques. Accuracy of Geometry Reconstruction: In general, it is expected that recovery from single-view images should yield less accurate results than from standard 3D reconstruction and/or the most recent 3D multi-view methods. Our method adopts accurate physics-based cloth simulation to assist in the recovery process and achieves comparable visual quality, with a focus on capturing plausible wrinkles and folds, as well as material parameters required for virtual try-on using only photographs. However, it is important to note that high visual quality does not always guarantee geometric accuracy in 3D garment recovery. At the same time, for some applications such as virtual try-on, rapid design, and prototyping, it is unclear if a high degree of geometric accuracy is required; it is also unknown how much error tolerance is needed for the comfortable fitting of garments. These are important considerations for further investigation in application to fashion design and e-commerce.

Limitations
The current implementation of our approach depends on two databases: a database of commonly available garment templates and a database of human-body models.
The range of garments we can recover is, to some extent, limited by the available garment templates. Our parameter identification method can only generate garments that are "morphable" from the garment template, i.e. homeomorphic to the garment template. For example, since we use only one template for each garment type, we cannot yet model variations in some clothing details, e.g. multi-layered skirts, or collars on shirts. But for those garments that are not morphable from the template, our method can recover whichever version of the garment is closest to the actual garment. With a more extensive set of templates, we can begin to model more variations of styles and cuts, with richer garment details.
Another limitation is the human body shape recovery. Our reduced human body shape is described by a set of semantic parameters z. The representation of this set of semantic parameters is not enormous, though it is sufficient to include most of the common human body shapes, as shown in our images. The known artifacts of linear human shape blending can also affect results. Aside from the human body shape recovery, our method is also limited by the state-of-art 3D human pose recovery methods. Manual intervention is needed when these methods fail to output a reasonably accurate 3D human pose.

CONCLUSION AND FUTURE WORK
In this paper we present an algorithm for highly detailed garment recovery from a single-view image. Our approach recovers a 3D mesh of the garment together with the 2D design pattern, fine wrinkles and folds, and material parameters. The recovered garment can be re-targeted to other human bodies of different shapes, sizes, and poses for virtual try-on and character animation.
In addition to addressing some limitations mentioned above, there are many possible future research directions. First of all, we plan to develop a parallelized implementation of our system on GPU or a many-core CPU for fast garment recovery. Both the underlying cloth simulator and the optimization process can be significantly accelerated. We also plan to extend our approach to enable fabric material transfer from videos for interactive virtual try-on. Furthermore, we hope to explore possible perception metrics, similar in spirit to .


s2fieldsofstudy:
Computer Science
Physics

text:
Acoustic Characterization of Environments (ACE) Challenge Results Technical Report

This document provides the results of the tests of acoustic parameter estimation algorithms on the Acoustic Characterization of Environments (ACE) Challenge Evaluation dataset which were subsequently submitted and written up into papers for the Proceedings of the ACE Challenge. This document is supporting material for a forthcoming journal paper on the ACE Challenge which will provide further analysis of the results.

Room dimensions, source, microphone and fan positions continued .
Talker positions used to produce babble noise .

Introduction
This document provides supplementary information and the results of the Acoustic Characterization of Environments (ACE) Challenge Phases 1 and 2 for all the tasks, Reverberation Time (T 60 ) and Direct-to-Reverberant Ratio (DRR) estimation in both fullband and in frequency bands.

Room recording procedure
The recording procedure in each room involved the following steps:  Table 3 provides each of the talker positions used to produce the babble noise. The z coordinates are not provided since these were not captured. However, the talkers were seated and their mouths were situated at approximately the same height as the microphone arrays which were all at 1.19 m  3.54, 1.19) above the floor. Table 4 provides the source-microphone distances and Direction-of-Arrivals (DoAs) in spherical coordinates, whilst Table 5 provides the fanmicrophone distances and DoAs in spherical coordinates.

Taxonomy of algorithms submitted
There were three main classes of algorithms submitted to the ACE Challenge:   The ABC approaches derive the estimate for the acoustic parameter directly from the signal without requiring any prior information. Bias compensation may be performed in order to account for noise or specific aspects of the source material. An example of this is the maximum likelihood    16.4 method [6] which directly produces the T 60 estimate.
The SFM approaches estimate a parameter from a signal that is correlated with the acoustic parameter to be estimated, and then apply a mapping function to give the acoustic parameter estimate. An example of this is the Spectral Decay Distributions (SDD) method which determines Negative- Side Variance (NSV) from STFT bins and then applies a mapping to obtain the T 60 .
The MLMF approaches typically use many features of the source material to train a neural network which then estimates the acoustic parameter from the features of a test signal. An example of this is the Non-Intrusive Room Acoustics (NIRA) [7] algorithm.
There were no hybrid approaches submitted to the ACE Challenge although several participants applied noise reduction to the source signals before performing parameter estimation.
Algorithms are further classified as being either providing an estimate in Fullband (FB), in frequency bands, or Subbands (SBs).

Results
The participating institutions in the ACE challenge along with their respective algorithms are listed in Table 6 in order of appearance of their algorithms in the results tables. For the fullband tasks the results are presented as box plots where there is a box shown for each algorithm. Both single and multi-channel algorithms are shown in the same figures and tables. On each box in the box plot, the central notch is the median, the edges of the box are the 25 th and 75 th percentiles, the whiskers extend to the most extreme data points not considered outliers. Boxes are colour-coded according to algorithm class: ABC: yellow; MLMF: cyan; SFM: green.
Outliers are plotted individually. The algorithms are identified on the box plot by a single character which corresponds to the character in the table after the figure. The results are sorted by the research group which achieved the highest correlation coefficient in the results across all noises and Signal-to-Noise Ratios (SNRs) in fullband. For the T 60 fullband task, the last three algorithms are those compared in Gaubitch et al. [8] and are included as baselines to enable the progress made in blind T 60 estimation since 2012 to be assessed. Similarly, for the DRR fullband task, Jeub et al. [9] is included as the last algorithm since this was a freely available estimator prior to the ACE Challenge. The correlation coefficient for each algorithm is plotted as a black cross in the same column as the algorithm. The value is provided on the right hand y-axis.
A table of numerical results is also provided following each figure which also provides the legend for the algorithm identifiers, A, B, C, etc. The columns in the table are as follows: (1) 6. MSE, the mean squared error in the estimation results defined as 7. ρ, the Pearson correlation coefficient between the estimated and the ground truth results defined as where Et¨u is the mathematical expectation; 8. RTF, the real-time factor, the total computation time divided by the total duration of all processed speech files. All implementations were in Matlab except for those marked with a : which used Matlab for feature extraction and C++ for the machine learning-based mapping, and those marked with a ; which were implemented entirely in C++.
By considering the bias, MSE, and ρ, it is possible to determine how well the estimator works. For example, an estimator with a low bias and MSE might simply be giving an estimate close to the median for every speech file. However, by examining the ρ, it will be possible to distinguish between such an algorithm, which will have a low correlation, and a better algorithm which is more accurately estimating the parameter concerned which will have a higher correlation. The RTF is useful for determining whether the algorithm has practical applications requiring low computational complexity such as hearing aids and mobile devices.
For the frequency-dependent tasks, a box plot is provided per algorithm with each box representing the performance in a particular frequency band. Frequency dependent algorithms have also been included in the fullband plots. Where those algorithms themselves produce a fullband estimate, this has been used directly as in the case of the DRR Estimation using a Null-Steered Beamformer (DENBE) [5] and Particle Velocity [4] algorithms. Where no fullband estimate is produced, a fullband estimate was obtained by taking the mean of the results over the 400 to 1250 Hz frequency bands as in the case of the Model-based subband RTE [3] algorithm as recommended in ISO 3382 [10].
2 Overall results summary 2

.1 Fullband T 60 estimation overall results
The overall results for fullband T 60 estimation are shown in Fig. 2 and Table 7.

Fullband DRR estimation overall results
The overall results for fullband DRR estimation are shown in Fig. 3 and Table 8.             Figure 18: Fullband T 60 estimation error in ambient noise at´1 dB SNR  Figure 19: Fullband T 60 estimation error in babble noise at 18 dB SNR            3.5 Frequency-dependent T 60 estimation results by noise type and SNR 3.5.1 Ambient noise at 18 dB                4.5 Frequency-dependent DRR estimation results by noise type and SNR 4.5.1 Ambient noise at 18 dB Figure 88: Frequency-dependent DRR estimation error in fan noise at´1 dB SNR for algorithm DENBE with filtered subbands [5]