Imagining the overall shape of persons based on a text description describing a scene is an easy task for humans even when it involves multiple humans interacting with each other such as in a football match. However, this task is not as simple for machines since the human shape is highly articulated and is usually in interaction with other individuals and objects in the environment, which creates a huge space of versatility. In this work, we propose a model that can generate 3D human shapes matching a high-level text description. Motivations for creating the 3D human mesh include animating multiple 3D avatars based on text and guiding image-synthesis frameworks towards synthesizing more realistic human shapes. In a similar work, Zhang et al.[1]} proposes a convolutional neural network (CNN) for generating a single person pose from text, which generates only the 2D joints but not the full shape and 3D pose. The model also has a limited capacity such that it can only generate one pose for any given text, but cannot handle scenarios that contain multiple poses. A 2D pose may be useful for guiding a pose transfer task as in [2]}, [3]}, [4]}, [5]} where an image of the person is given, but is far too sparse to constrain a network to generate a realistic-looking unseen person from text. Additionally, synthesizing multi-person shapes adds much more complexity to the task, since in typical scenarios the persons in the scene occlude or overlap with each other.
In this work, we address the problem of generating a variable number of 3D human shapes based on the SMPL model [6]}. Figure REF  describes the overall task. In our scenario, the model should have the capability to deal with a variable number of shapes that match the text description, as well as infer the interactions between the different individuals in the scene. Since a non-recurrent GAN cannot handle content generation with a variable number of outputs, we propose a recurrent generative adversarial network (GAN). Our proposed recurrent GAN consists of a recurrent generator and recurrent discriminator (critic) such that the generator generates a single 3D body shape at each of its iterations, and the critic iterates over a set of shapes and outputs a final score as a many-to-one recurrent network. To help the generator focus on the relevant words in the current iteration, we additionally learn words weights that indicate how much information each word is contributing to generating current shape. The hidden state of the recurrent component maintains information that helps it remember for which words semantically consistent shapes have already been generated and for words more shapes have yet to be generated. The critic has to fulfill two roles, first it has to assess the quality of the individual shapes and their semantic consistency with the given text. Additionally it has to assess the matching between the generated shape and the text in a collective way, so that for example it can tell if certain words are still not satisfied, i.e. not all words have a corresponding human shape.
When a text describes multiple people in the scene or even a single person, image generation methods synthesize images with good-looking scenes but with a disfigured human shape [7]}, [8]}, [9]}, [10]}. Using 2D keypoints helps slightly as shown in [11]} (figure REF ), however, it is too sparse to help guide these frameworks in generating a realistic-looking human.
To demonstrate the benefit of the proposed task, we integrate our model with an image synthesis framework while using COCO [12]} as the training dataset and show how it substantially improves the person's appearance in the synthesized images.
<FIGURE>