Generative Adversarial Transformers

We introduce the GANformer, a novel and efficient type of transformer, and explore it for the task of visual generative modeling. The network employs a bipartite structure that enables long-range interactions across the image, while maintaining computation of linear efficiency, that can readily scale to high-resolution synthesis. It iteratively propagates information from a set of latent variables to the evolving visual features and vice versa, to support the refinement of each in light of the other and encourage the emergence of compositional representations of objects and scenes. In contrast to the classic transformer architecture, it utilizes multiplicative integration that allows flexible region-based modulation, and can thus be seen as a generalization of the successful StyleGAN network. We demonstrate the model's strength and robustness through a careful evaluation over a range of datasets, from simulated multi-object environments to rich real-world indoor and outdoor scenes, showing it achieves state-of-the-art results in terms of image quality and diversity, while enjoying fast learning and better data-efficiency. Further qualitative and quantitative experiments offer us an insight into the model's inner workings, revealing improved interpretability and stronger disentanglement, and illustrating the benefits and efficacy of our approach. An implementation of the model is available at https://github.com/dorarad/gansformer.


Introduction
The cognitive science literature speaks of two reciprocal mechanisms that underlie human perception: the bottom-up processing, proceeding from the retina up to the cortex, as local elements and salient stimuli hierarchically group together to form the whole [27], and the top-down processing, where surrounding global context, selective attention and prior knowledge inform the interpretation of the particular [32]. While their respective roles and dynamics are being  actively studied, researchers agree that it is the interplay between these two complementary processes that enables the formation of our rich internal representations, allowing us to perceive the world around in its fullest and create vivid imageries in our mind's eye [13,17,39,52].
Nevertheless, the very mainstay and foundation of computer vision over the last decade -the Convolutional Neural Network, surprisingly, does not reflect this bidirectional nature that so characterizes the human visual system, and rather displays a one-way feed-forward progression from raw sensory signals to higher representations. Unfortunately, the local receptive field and rigid computation of CNNs reduce their ability to model long-range dependencies or develop holistic understanding of global shapes and structures that goes beyond the brittle reliance on texture [26], and in the generative domain especially, they are linked to considerable optimization and stability issues [70] due to their fundamental difficulty in coordinating between fine details across the generated scene. These concerns, along with the inevitable comparison to cognitive visual processes, beg the question of whether convolution alone provides a complete solution, or some key ingredients are still missing. § I wish to thank Christopher D. Manning for the fruitful discussions and constructive feedback in developing the bipartite transformer, especially when explored within the language representation area, as well as for the kind financial support that allowed this work to happen. arXiv:2103.01209v4 [cs.CV] 29 Mar 2022 Figure 2. Bipartite Attention. We introduce the GANformer network, that leverages a bipartite structure to support long-range interactions while evading the quadratic complexity standard transformers suffer from. We present two novel attention operations over the bipartite graph: simplex and duplex, the former permits communication in one direction, in the generative context -from the latents to the image features, while the latter enables both top-down and bottom-up connections between these two dual representations.
Meanwhile, the NLP community has witnessed a major revolution with the advent of the Transformer network [65], a highly-adaptive architecture centered around relational attention and dynamic interaction. In response, several attempts have been made to integrate the transformer into computer vision models, but so far they have met only limited success due to scalabillity limitations stemming from its quadratic mode of operation.
Motivated to address these shortcomings and unlock the full potential of this promising network for the field of computer vision, we introduce the Generative Adversarial Transformer, or GANformer for short, a simple yet effective generalization of the vanilla transformer, explored here for the task of visual synthesis. The model utilizes a bipartite structure for computing soft attention, that iteratively aggregates and disseminates information between the generated image features and a compact set of latent variables that functions as a bottleneck, to enable bidirectional interaction between these dual representations. This design achieves a favorable balance, being capable of flexibly modeling global phenomena and long-range interactions on the one hand, while featuring an efficient setup that still scales linearly with the input size on the other. As such, the GANformer can sidestep the computational costs and applicability constraints incurred by prior works, caused by the dense and potentially excessive pairwise connectivity of the standard transformer [5,70], and successfully advance the generative modeling of compositional images and scenes.
We study the model's quantitative and qualitative behavior through a series of experiments, where it achieves stateof-the-art performance for a wide selection of datasets, of both simulated as well as real-world kinds, obtaining particularly impressive gains in generating highly-structured multi-object scenes. As indicated by our analysis, the GANformer requires less training steps and fewer samples than competing approaches to successfully synthesize images of high quality and diversity. Further evaluation provides ro-bust evidence for the network's enhanced transparency and compositionality, while ablation studies empirically validate the value and effectiveness of our approach. We then present visualizations of the model's produced attention maps, to shed more light upon its internal representations and synthesis process. All in all, as we will see through the rest of the paper, by bringing the renowned GANs and Transformer architectures together under one roof, we can integrate their complementary strengths, to create a strong, compositional and efficient network for visual generative modeling.

Related Work
Generative Adversarial Networks (GANs) [28], originally introduced in 2014, have made remarkable progress over the past years, with significant advances in training stability and dramatic improvements in image quality and diversity. that turned them to be nowadays one of the leading paradigms in visual synthesis [5,44,58]. In turn, GANs have been widely adopted for a rich variety of tasks, including imageto-image translation [40,72], super-resolution [47], style transfer [12], and representation learning [18], to name a few. But while generated images for faces, single objects or natural scenery have reached astonishing fidelity, becoming nearly indistinguishable from real samples, the unconditional synthesis of more structured or compositional scenes is still lagging behind, suffering from inferior coherence, reduced geometric consistency and, at times, a lack of global coordination [9,43,70]. As of now, faithful generation of structured scenes is thus yet to be reached.
Concurrently, the last years saw impressive progress in the field of NLP, driven by the innovative architecture called Transformer [65], which has attained substantial gains within the language domain and consequently sparked considerable interest across the deep learning community [16,65]. In response, several attempts have been made to incorporate self-attention constructions into vision models, most commonly for image recognition, but also in segmentation [25], detection [8], and synthesis [70]. From structural perspective, these can be roughly divided into two streams: those that apply local attention operations, failing to capture global interactions [14,37,56,57,71], and others that borrow the original transformer structure as-is and perform attention globally across the entire image, resulting in prohibitive computation due to the quadratic complexity, which fundamentally hinders its applicability to low-resolution layers only [3,5,19,24,41,66,70]. Few other works proposed sparse, discrete or approximated variations of self-attention, either within the adversarial or autoregressive contexts, but they still fall short of reducing memory footprint and computational costs to a sufficient degree [11,24,36,38,62].
Compared to these prior works, the GANformer stands out as it manages to avoid the high costs ensued by self attention, employing instead bipartite attention between the image features and a small collection of latent variables. Its design fits naturally with the generative objective of transforming source latents into an output image, facilitating long-range interaction without sacrificing computational efficiency. Rather, the network maintains a scalable linear computation across all layers, realizing the transformer's full potential. In doing so, we seek to take a step forward in tackling the challenging task of scene generation. Intuitively, and as is later corroborated by our findings, allocating multiple latents to interact through attention with the generated image serves as a structural prior of a bottleneck that promotes the formation of compact and compositional scene representations, as the different latents may specialize to certain objects or semantic regions of interest. Indeed, as demonstrated in section 4, the Generative Adversarial Transformer achieves state-of-the-art performance in synthesizing varied real-world indoor and outdoor scenes, while showing indications for semantic disentanglement along the way. In designing our model, we draw inspiration from multiple lines of research on generative modeling, compositionality and scene understanding, including techniques for scene decomposition, object discovery and representation learning. Several variational approaches [7,22,23,31] perform iterative inference to encode scenes into multiple slots, but are mostly applied in the contexts of synthetic and oftentimes fairly rudimentary 2D settings. Works such as Capsule networks [29,61] leverage ideas from psychology about Gestalt principles [34,63], perceptual grouping [6] or analysis-bysynthesis [4], and like us, introduce ways to piece together visual elements to discover compound entities and, in the cases of Set Transformers [48] or A 2 -Nets [10], group local information into global aggregators, which proves useful for a broad spectrum of tasks, spanning unsupervised segmentation [30,50], clustering [48], image recognition [2], NLP [59] and viewpoint generalization [46]. However, our work stands out incorporating new ways to integrate information across the network through novel forms of attention: (Simplex and Duplex), that iteratively update and refine the assignments between image features and latents, and is the first to explore these techniques in the context of high-resolution generative modeling.
Most related to our work are certain GAN models for conditional and unconditional visual synthesis: A few methods [21,33,54,64] utilize multiple replicas of a generator to produce a set of image layers, that are then combined through alpha-composition. As a result, these models make quite strong assumptions about the independence between the components depicted by each layer. In contrast, our model generates one unified image through a cooperative process, coordinating between the different latents through the use of soft attention. Other works, such as SPADE [55,73], employ region-based feature modulation for the task of layout-to-image translation, but, contrary to us, use fixed segmentation maps and static class embeddings to control the visual features. Of particular relevance is the prominent StyleGAN model [44,45], which utilizes a single global style vector to consistently modulate the features of each layer. The GANformer generalizes this design, as multiple style vectors impact different regions in the image concurrently, allowing for spatially finer control over the generation process. Finally, while StyleGAN broadcasts information in one direction from the single global latent to the local image features, our model propagates information both from latents to features and vice versa, enabling topdown and bottom-up reasoning to occur simultaneously 1 .

The Generative Adversarial Transformer
The Generative Adversarial Transformer (GANformer) is a type of Generative Adversarial Network, which involves a generator network (G) that maps random samples from the latent space to the output space (e.g. an image), and a discriminator network (D) which seeks to discern between real and fake samples [28]. The two networks compete with each other through a minimax game until reaching an equilibrium. Typically, each of these networks consists of multiple layers of convolution, but in the GANformer case, we instead construct them using a novel architecture, called Bipartite Transformer, formally defined below.
The section is structured as follows: we first present a formulation of the Bipartite Transformer, a domain-agnostic generalization of the Transformer 2 (section 3.1). Then, we provide an overview of how the transformer is incorporated into the generative adversarial framework (section 3.2). We conclude by discussing the merits and distinctive properties of the GANformer, that set it apart from the traditional GAN and transformer networks (section 3.3).

The Bipartite Transformer
The standard transformer network is composed of alternating multi-head self-attention and feed-forward layers. We refer to each pair of self-attention and feed-forward operations as a transformer layer, such that a transformer is considered to be a stack of several such layers. The Self-Attention layer considers all pairwise relations among the input elements, updating each one by attending to all the others. The Bipartite Transformer generalizes this formulation, featuring instead a bipartite graph between two groups of variables -in the GAN case, latents and image features. In as a biologically-accurate reflection of cognitive top-down processing. Rather, this analogy plays as a conceptual source of inspiration that aided us through the idea development. 2 By transformer, we precisely mean a multi-layer bidirectional transformer encoder, as described in [16], which interleaves selfattention and feed-forward layers. the following, we consider two forms of attention that could be computed over the bipartite graph -Simplex attention and Duplex attention, depending on the direction in which information propagates 3 -either in one way only, from the latents to the image, or both in top-down and bottom-up ways. While for clarity purposes, we present the technique here in its one-head version, in practice we make use of a multi-head variant, in accordance with prior work [65].

SIMPLEX ATTENTION
We begin by introducing the simplex attention, which distributes information in a single direction over the bipartite transformer graph. Formally, let X n×d denote an input set of n vectors of dimension d (where, for the image case, n = W ×H), and Y m×d denote a set of m aggregator variables (the latents, in the generative case). We can then compute attention over the derived bipartite graph between these two groups of elements. Specifically, we define: Where q(·), k(·), v(·) are functions that respectively map elements into queries, keys, and values, all maintaining dimensionality d. We also provide the mappings with positional encodings, to reflect the distinct spatial position of each element e.g. in the image (see section 3.2 for details). Note that this bipratite attention is a generalization of self attention, where Y = X.
We can then integrate the attended information with the input elements X, but whereas the standard transformer implements an additive update rule of the form: we instead use the retrieved information to control both the scale as well as the bias of the elements in X, in line with the practice promoted by the StyleGAN model [44]. As our experiments indicate, such multiplicative integration enables significant gains in the model's performance. Formally: Where γ(·), β(·) are mappings that compute multiplicative and additive factors (scale and bias), both maintaining a dimension of d, and ω(X) = X−µ(X) normalizes the features of X 4 . By normalizing X (the image features), and then letting Y (the latents) control X's statistical tendencies, we essentially enable information propagation from Y to X, intuitively, allowing the latents to control the visual generation of spatial attended regions within the image, so as to guide the synthesis of objects and entities.

DUPLEX ATTENTION
We can go further and consider the variables Y to posses a key-value structure of their own [53]: where the values V store the content of the Y variables as before (i.e. the randomly sampled latent vectors) while the keys K track the centroids of the attention-based assignment between X and Y , which can be computed by K = a(Y, X) -namely, the weighted averages over X elements, using the attention distribution derived by comparing them to Y elements. Intuitively, each centroid tracks the region in the image X that interacts with the respective latent in Y . Consequently, we can define a new update rule: This update compounds together two attention operations: first (1) computing attention assignments between X and Y , by K = a(Y, X), and then (2) refining the soft assignments by considering their centroids, through A(Q, K, V ), where Q = q(X), which computes attention between the elements X and their centoroids K. This is analogous to the Expectation-Maximization or k-means algorithms, [49,50], where we iteratively refine the assignments of elements X to clusters Y based on their distance to their respective centroids K = a(Y, X). As is empirically shown later, this works more effectively than the update u s defined above.
Finally, to support bidirectional interaction between X and Y (the image and the latents), we can chain two reciprocal Figure 6. Sample images and attention maps of lower and upper GANformer layers, for the CLEVR, LSUN-Bedrooms, FFHQ and Cityscapes datasets. The colors in the attention maps correspond to the assignment between the image regions and the latent variables that control them. For the CLEVR dataset, we can see multiple attention maps produced by different layers of the model, revealing how the role of the latent variables changes at different stages of the generation -while they correspond to an instance segmentation as the layout of the scene is being formed in the early low-resolution layers, they behave similarly to a surface normal in the upper high-resolution layers of the generator. We see similar progression from a coarser to finer pattern of attention for the FFHQ dataset.

The Generator and Discriminator Networks
Our generator and discriminator networks follow the general design of prior work [44,45], with the key difference of incorporating the novel bipartite attention layers instead of the single-latent modulation that characterizes earlier models: Commonly, a generator network consists of a multilayer CNN that receives a randomly sampled vector z and transforms it into an image. The popular StyleGAN approach departs from this design and, instead, introduces a feed-forward mapping network that outputs an intermediate vector w, which in turn interacts directly with each convolution through the synthesis network, globally modulating the feature maps' statistics at every layer.
Effectively, this approach attains layer-wise decomposition of visual properties, allowing StyleGAN to control global aspects of the picture such as the pose, lighting conditions or color scheme, in a coherent manner over the entire image. But while StyleGAN successfully disentangles global attributes, it is more limited in its ability to perform spatial decomposition, as it provides no direct means to control the style of localized regions within the generated image.
The bipartite transformer offers a solution to accomplish this objective. Instead of modulating the style of all features globally, we use instead our new attention layer to perform adaptive region-wise modulation. As shown in figure 3 (right), we split the latent vector z into k components, z = [z 1 , ...., z k ] and, as in StyleGAN, pass each of them through a shared mapping network, obtaining a corresponding set of intermediate latent variables Y = [y 1 , ..., y k ]. Then, during synthesis, after each CNN layer of the generator, we let the feature map X and latents Y play the roles of the two element groups, mediating their interaction through our new attention layer -either simplex or duplex. This setting thus allows for a flexible and dynamic style modulation at the level of the region. Since soft attention tends to group elements based on their proximity and content similarity, we see how the transformer architecture naturally fits into the generative task and proves useful in the visual domain, allowing the model to exercise finer control in modulating local semantic regions. As we see in section 4, this capability turns out to be especially useful in modeling highly-structured scenes.
As to the loss function, optimization and training configurations, we adopt the settings and techniques used by Style-GAN2 [45], including in particular style mixing, stochastic variation, exponential moving average for weights, and a non-saturating logistic loss with lazy R1 regularization 5 . 5 In the prior version of the paper and in earlier stages of the model development, we explored incorporating the bipartite attention to both the generator and the discriminator, in order to allow both components make use of long-range interactions. However, in ablation experiments we observed that applying attention to the

Summary
To recapitulate the discussion above, the GANformer successfully unifies the GAN and Transformer architectures for the task of scene generation. Compared to traditional GANs and transformers, it introduces multiple key innovations: • Compositional Latent Space with multiple variables that coordinate through attention to produce the image cooperatively, in a manner that matches the inherent compositionality of natural scenes.
• Bipartite Structure that balances between expressiveness and efficiency, modeling long-range dependencies while maintaining linear computational costs.
• Bidirectional Interaction between the latents and the visual features, which allows the refinement and interpretation of each in light of the other.
• Multiplicative Integration rule to impact the features' visual style more flexibly, akin to StyleGAN but in contrast to the classic transformer network.
As we see in the following section, the combination of these design choices yields a strong architecture that demonstrates high efficiency, improved latent space disentanglement, and enhanced transparency of the generative process.

Experiments
We investigate the GANformer through a suite of experiments that study its quantitative performance and qualitative behavior. As we will see below, the GANformer achieves state-of-the-art results, successfully producing high-quality images for a varied assortment of datasets: FFHQ for human faces [44], the CLEVR dataset for multi-object scenes [42], and the LSUN-Bedrooms [69] and Cityscapes [15] datasets for challenging indoor and outdoor scenes. Notably, it even attains state-of-the-art FID scores for the challenging and highly-structured COCO dataset.
Further analysis we conduct in sections 4.1, 4.2 and 4.3 provides evidence for multiple favorable properties the GANformer posses, including better data-efficiency, enhanced transparency, and stronger disentanglement than prior approaches. Section 4.4 then quantitatively assesses the network's semantic coverage of the natural image distribution for the CLEVR dataset, while ablation and variation studies at section 4.5 empirically validate the necessity of each of the model's design choices. Taken altogether, our evaluation offers solid evidence for the GANformer's effectiveness and efficacy in modeling compsitional images and scenes.
generator only allows for stronger results, and so we have updated the paper accordingly. We compare our network with several approaches, including both baselines and leading models for image synthesis: (1) A baseline GAN [28] that follows the typical convolutional architecture 6 ; (2) StyleGAN2 [45], where a single global latent interacts with the evolving image by modulating its global style; (3) SAGAN [70], which performs self attention across all feature pairs in low-resolution layers of the generator and the discriminator; and (4) k-GAN [64] that produces k separated images, which are then blended through alpha-composition.
To evaluate all models under comparable training conditions, model size, and optimization scheme, we implement them all within our public codebase, which extends the official StyleGAN repository. All models have been trained with images of 256 × 256 resolution and for the same number of training steps, roughly spanning a week on 2 NVIDIA V100 GPUs per model (or equivalently 3-4 days using 4 GPUs). For the GANformer, we select k -the number of latent variables, from the range of 8-32. Note that increasing the value of k does not translate to an increased overall latent dimension, and we rather keep it equal across models. See section A for further implementation details, hyperparameter settings and training configurations.
As shown in table 1, our model matches or outperforms prior work, achieving substantial gains in terms of FID score, which correlates with image quality and diversity [35], as well as other commonly used metrics such as Precision and Recall (P&R) 7 . As could be expected, we obtain 6 In the baseline GAN, we input the noise through the network's stem instead of through weight modulation. 7 Note that while the StyleGAN paper [45] reports lower FID scores for FFHQ and LSUN-Bedrooms, they are obtained by training for 5-7 times longer than our experiments (specifically, they train for up to 17.5 million steps, producing 70M samples and demanding over 90 GPU-days). To comply with a reasonable compute budget, we equally reduced the training duration for all models in our evaluation, maintaining the same number of steps. the least gains for the FFHQ human faces dataset, where naturally there is relatively lower diversity in image layout. On the flip side, most notable are the significant improvements in performance for CLEVR, where our approach successfully lowers FID scores from 16.05 to 9.17, as well as LSUN-Bedrooms, where the GANformer nearly halves the FID score from 11.53 to 6.51, being trained for equal number of steps. These findings suggest that the GANformer is particularly adept at modeling scenes of high compositionality (CLEVR) or layout diversity (LSUN-Bedrooms). Comparing between the Simplex and Duplex Attentions further reveals the strong benefits of integrating the reciprocal bottom-up and top-down processes together.

Data and Learning Efficiency
We examine the learning curves of our and competing models (figure 7, (3)) and inspect samples of generated images at different stages of the training (figure 12). These results both indicate that our model learns significantly faster than competing approaches. In the case of CLEVR, it produces high-quality images in approximately 3-times less training steps than the second-best approach. To further explore the GANformer's learning aptitude, we perform experiments where we reduce the size of the dataset each model (and specifically, its discriminator) is exposed to during training to varying degrees ( figure 7, (4)). These results similarly validate the model's superior data-efficiency, especially where as few as 1k images are provided for training.

Transparency & Compositionality
To gain more insight into the model's internal representation and its underlying generative process, we visualize the attention distributions produced by the GANformer as it synthesizes new images. Recall that at each layer of the generator, it casts attention between the k latent variables and the evolving spatial features of the generated image.  As illustrated by figures 4 and 6, the latent variables tend to attend to coherent visual regions in terms of proximity and content similarity. Figure 6 provides additional attention maps computed by the model in various layers, showing how it behaves distinctively in different stages of the generation process. The visualizations imply that the latents carry a semantic sense, capturing objects, visual entities or other constituent components of the synthesized scenes. These findings can thereby attest to an enhanced compositionality that our model acquires through its multi-latent structure. Whereas prior work uses a single monolithic latent vector to account for the whole scene and modulate features at a global scale only, our design lets the GANformer exercise finer control that impacts features at the object granularity, while leveraging the use of attention to make its internal representations more structured and transparent.
To quantify the compositionality exhibited by the model, we use a pre-trained detector [67] to produce segmentations for a set of generated scenes, in order to measure the correlation between the attention cast by the latents with various semantic classes. Figure 8shows the classes that have the highest correlation with respect to the latent variables, indicating that different latents indeed coherently attend to semantic concepts such as windows, pillows, sidewalks or cars, as well as background regions like carpets, ceiling, and walls. This illustrates how the multiple latents are effectively used to semantically decompose the scene generation task.

Disentanglement
We consider the DCI and Modularity metrics commonly used in the disentanglement literature [20,60] to provide more evidence for the beneficial impact our architecture has on the model's internal representation. These metrics asses the Disentanglement, Completeness, Informativeness and Modularity of a given representation, essentially evaluating the degree to which there is a 1-to-1 correspondence between latent factors and global image attributes. To obtain the attributes, we consider the area size of each semantic class (e.g. cubes, spheres, floor), predicted by a pre-trained segmentor, and use them as the output response features for measuring the latent space disentanglement, computed over 1k images. We follow the protocol proposed by Wu et al. [68] and present the results in table 3. This analysis confirms that the GANformer's latent representations enjoy higher disentanglement compared to competing approaches.

Image Diversity
A major advantage of compositional representations is that they can support combinatorial generalization -a key foundation of human intelligence [1]. Inspired by this observation, we measure this property in the context of visual synthesis of multi-object scenes. We use a pre-trained object detector on generated CLEVR scenes, to extract the objects and properties within each sample. We then compute Chi-Square statistics on the sample set to determine the degree to which each model manages to cover the natural uniform distribution of CLEVR images. Table 2 summarizes the results, where we can see that our model obtains better scores across almost all the semantic properties of the scenes distribution. These metrics complement the common FID and PR scores as they emphasize structure over texture, or semantics over perceptual appearance, focusing on object existence, arrangement and local properties, and thereby substantiating further the model's compositionality.

Ablation Studies
To validate the usefulness of bipartite attention, we conduct ablation studies, where we vary the index of the earliest and latest layers of the generator network to which attention is incorporated. As indicated by figure 7 (1-2), the earlier (or lower resolution) attention begins being applied, the better the model's performance and the faster it learns. The same goes for the latest layer to apply attention to -as attention can especially contribute in high-resolutions, which benefit the most from long-range interactions. These studies provide a validation for the effectiveness of our approach in enhancing generative scene modeling.

Conclusion
We have introduced the GANformer, a novel and efficient bipartite transformer that combines top-down and bottom-up interactions, and explored it for the task of generative modeling, achieving strong quantitative and qualitative results that attest to the model robustness and efficacy. The GANformer fits within the general philosophy that aims to incorporate stronger inductive biases into neural networks to encourage desirable properties such as transparency, data-efficiency and compositionality -properties which are at the core of human intelligence, serving as the basis for our capacity to plan, reason, learn, and imagine. While our work focuses on visual synthesis, we note that the bipartite transformer is a general-purpose model, and expect it may be found useful for other tasks in both vision and language. Overall, we hope that our work will help progressing further in our collective search to bridge the gap between the intelligence of humans and machines.

Acknowledgments
We are grateful to Stanford HAI for the generous computational resources provided through Amazon AWS cloud credits. I also wish to thank Christopher D. Manning for the fruitful discussions and constructive feedback in developing the bipartite transformer, especially when we explored it for language representation, as well as for the kind financial support he provided that allowed this work to happen.
In the following, we provide additional experiments and visualizations for the GANformer model. First, we present in figures 12 and 9 a comparison of sample images produced by the GANformer and a set of baseline models, over the course of the training and after convergence respectively. Section A specifies the implementation details, optimization scheme and training configuration of the model. In section B and figure 8, we evaluate the spatial compositionality of the GANformer's attention mechanism, shedding light upon the roles of the different latent variables.

A. Implementation and Training Details
To evaluate all models under comparable conditions of training configuration, model size, and optimization details, we implement them all within the TensorFlow codebase introduced by the StyleGAN authors [44]. See table 4 for particular settings of the GANformer and table 5 for comparison of model sizes.
In terms of the loss function, optimization and training configuration, we adopt the settings and techniques used in the StyleGAN2 model [45], including in particular style mixing, Xavier Initialization, stochastic variation, exponential moving average for weights, and a non-saturating logistic loss with lazy a R1 regularization. We use Adam optimizer with batch size of 32 (4 × 8 using gradient accumulation), equalized learning rate of 0.001, β 1 = 0.0 and β 2 = 0.99 as well as leaky ReLU activations with α = 0.2, bilinear filtering in all up/downsampling layers and minibatch standard deviation layer at the end of the discriminator. The mapping layer of the generator consists of 8 layers, and ResNet connections are used throughout the model, for the mapping network, synthesis network and discriminator.
We train all models on images of 256 × 256 resolution, padded as necessary. The CLEVR dataset consists of 100k images, the FFHQ has 70k images, Cityscapes has overall about 25k images and LSUN-Bedrooms has 3M images. The images in the Cityscapes and FFHQ datasets are mirroraugmented to increase the effective training set size. All models have been trained for the same number of training steps, roughly spanning a week on 2 NVIDIA V100 GPUs per model.

B. Spatial Compositionality
To quantify the compositionality level exhibited by the model, we employ a pre-trained segmentor to produce semantic segmentations for the synthesized scenes, and use them to measure the correlation between the attention cast by the latent variables and the various semantic classes. We derive the correlation by computing the maxi- mum intersection-over-union between a class segment and the attention segments produced by the model in the different layers. The mean of these scores is then taken over a set of 1k images. Results presented in figure 8 for the LSUN-Bedrooms and Cityscapes datasets, showing semantic classes which have high correlation with the model attention, indicating it decomposes the image into semanticallymeaningful segments of objects and entities.  GAN StyleGAN2 k-GAN Figure 9. State-of-the-art comparison. A comparison between models' sample images for the CLEVR, LSUN-Bedrooms and Cityscapes datasets. All models have been trained for the same number of steps, which ranges between 5k to 15k kimg training samples. Note that the original StyleGAN2 model has been trained by its authors for up to 70k kimg samples, which is expected to take over 90 GPU-days for a single model. See next pages for comparison with further models. These images show that given the same training length the GANformer model's sample images enjoy higher quality and diversity compared to prior works, demonstrating the efficacy of our approach.
SAGAN VQGAN GANformer s Figure 10. A comparison of models' sample images for the CLEVR, LSUN-Bedrooms and Cityscapes datasets. See figure 9 for further description. Figure 11. A comparison between models' sample images for the CLEVR, LSUN-Bedrooms and Cityscapes datasets. See figure 9 for further description.

GAN
StyleGAN k-GAN Figure 12. State-of-the-art comparison over training. A comparison between models' sample images for the CLEVR, LSUN-Bedrooms and Cityscapes datasets, generated at different stages throughout the training. Sample images from different points in training are based on the same sampled latent vectors, thereby showing how the image evolves during the training. For CLEVR and Cityscapes, we present results after training to generate 100k, 200k, 500k, 1m, and 2m samples. For the Bedroom case, we present results after 500k, 1m, 2m, 5m and 10m generated samples during training. These results show how the GANformer, especially when using duplex attention, manages to learn a lot faster than competing approaches, generating impressive images early in the training.