Motivated to alleviate these shortcomings and make generative modeling more compositional, interpretable and controllable, we propose GANformer2, a structured object-oriented transformer that decouples the visual synthesis task into two stages: planning and execution. We begin by sampling a set of latent variables, corresponding to the objects and entities that compose the scene. Then, at the planning stage, we transform the latents into a schematic layout – a scene representation that depicts the object segments, reflecting their shapes, positions, semantic classes and relative ordering. Its construction occurs in an iterative manner, to capture the dependencies and interactions between different objects. Next, at the execution stage, the layout is translated into the final image, with the latents cooperatively guiding the content and style of their respective segments through bipartite attention [1]}. This approach marks a shift towards a more natural process of image generation, in which objects and the relations among them are explicitly modeled, while the initial sketch is successively tweaked and refined to ultimately produce a rich photo-realistic scene.
