FIELD

The present disclosure relates generally to generating three-dimensional (3D) virtual garment model, and more particularly to system and methods for automatically generating 3D virtual garment model using both text description and image representation of a product.

BACKGROUND

The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

E-commerce has been a significant part of social life, and there are millions of fashion products for sale on online retail platforms such as amazon.com, jd.com and alibaba.com. Although picture of clothes and picture of clothes dressed on models are usually provided by the platforms, 3D representation of the garment on fashion model is not available.

Therefore, a demand exists in the art to address the aforementioned deficiencies and inadequacies.

SUMMARY

In certain aspects, the present disclosure relates to a method for generating a three-dimensional (3D) model of a garment (i.e., 3D garment model). The method includes:providing, by a computing device, description and at least one image of the garment;identifying, by the computing device, type and attributes of the garment from the description;classifying, by the computing device, the at least one image to obtain classification of the at least one image as a garment image having the garment or the model image having the garment dressed on a first model;segmenting, by the computing device, the at least one image based on the type of the garment and the classification of the image to obtain a mask of the garment;determining, by the computing device, landmarks of the garment in the at least one image using the mask, and parsing the garment in the at least one image into a plurality of parts, where the parsing may or may not use the landmarks;fitting, by the computing device, the landmarks to a 3D garment template corresponding to the type of the garment to obtain 3D garment representation;registering, by the computing device, the 3D garment representation to a second model to obtain a preliminary 3D garment model, where the 3D garment model is dressed on the second model, and the second model is a 3D human body model;projecting, by the computing device, the preliminary 3D garment model dressed on the second model to at least one 2D projected image; andcomparing, by the computing device, the at least one 2D projected image with corresponding one of the at least one model image, so as to refine the preliminary 3D garment model to obtain the final 3D garment model.

In certain embodiments, the type of the garment is a category of the garment including sweater, shirt, skirt, T shirt and pant, and the attributes of the garment includes color, material, neckline, and gender of the garment. In certain embodiments, the step of identifying the type and attribute of the garment is performed by retrieving the type and attribute from a product database or by extracting the type and attribute from a shopping webpage of the garment.

In certain embodiments, the step of classifying the at least one image is performed by a neural network, the neural network includes concatenated convolution blocks to extract semantic features from the at least one image, and each of the convolution blocks includes a 2D convolution layer, a maxpooling layer, and a batch norm layer.

In certain embodiments, the step of segmenting the at least one image includes, when the at least one image is classified as the garment image: segmenting the at least one image using graph cut.

In certain embodiments, the step of segmenting the at least one image includes, when the at least one image is classified as the model image: locating a bounding box enclosing garment region of the at least one image using Faster region convolutional neural network (Faster R-CNN), and segmenting the at least one image using Fully convolutional neural network (FCN).

In certain embodiments, the step of determining landmarks is performed using a neural network.

In certain embodiments, the step of determining the landmarks and parsing the garment includes deforming a 2D garment template to the mask, where the 2D garment template is retrieved based on the type of the garment, and includes landmark points and parsing of the 2D garment template.

In certain embodiments, the method further includes:after classifying the at least one image as a model image: estimating pose of the first model in the model image; andbefore the step of projecting: positioning the second model in the estimated pose of the first model in the model image.

In certain embodiments, the method further includes rendering texture on the preliminary 3D garment model, the final 3D garment model, and intermediate 3D garment models intermediate between the preliminary 3D garment model and the final 3D garment model.

In certain aspects, the present disclosure relates to a method for generating a three-dimensional (3D) garment model of a garment. In certain embodiments, the method includes:providing, by a computing device, at least one image of the garment;generating, by the computing device, a 3D garment representation based on the at least one image;registering, by the computing device, the 3D garment representation to a 3D model to obtain a preliminary 3D garment model dressed on the 3D model;projecting the preliminary 3D garment model dressed on the 3D model to at least one 2D projected image; andcomparing the at least one 2D projected image with corresponding one of the at least one image, so as to refine the preliminary 3D garment model to obtain the final 3D garment model of the garment.

In certain embodiments, the method further includes classifying the at least one image of the garment as a garment image having the garment only and a model image having the garment dressed on a first model. In certain embodiments, the method further includes estimating a pose of the first model in the model image to obtain an estimated pose; and positioning the 3D model in the estimated pose. In certain embodiments, the first model is a 2D model, and the 3D model is a 3D human body model.

In certain embodiments, the method further includes, before generating the 3D garment representation: determining landmarks of the garment in the at least one image, and fitting the landmarks to a 3D garment template to obtain the 3D garment representation.

In certain embodiments, the method further includes identifying type and attributes of the garment from description of the garment.

In certain embodiments, the steps of generating, registering, projecting and comparing are iteratively performed until the difference between the 2D projected image and the model image is less than a predetermined value.

In certain aspects, the present disclosure relates to a system for generating a 3D garment model. The system includes a computing device. The computing device has a processor and a storage device storing computer executable code. The computer executable code, when executed at the processor, is configured to perform the methods described above.

In certain aspects, the present disclosure relates to a non-transitory computer readable medium storing computer executable code. The computer executable code, when executed at a processor of a computing device, is configured to perform the method as described above.

These and other aspects of the present disclosure will become apparent from following description of the preferred embodiment taken in conjunction with the following drawings and their captions, although variations and modifications therein may be affected without departing from the spirit and scope of the novel concepts of the disclosure.

