Diffusion models with large-scale pre-training have achieved significant success in the field of visual content generation, particularly exemplified by Diffusion Transformers (DiT) . However, DiT models have faced challenges with scalability and quadratic complexity efficiency.
In this paper, we aim to leverage the long sequence modeling capability of Gated Linear Attention (GLA)  Transformers, expanding its applicability to diffusion models.
We introduce Diffusion Gated Linear Attention Transformers (DiG) , a simple, adoptable solution with minimal parameter overhead, following the DiT design, but offering superior efficiency and effectiveness.
In addition to better performance than DiT, DiG-S/2 exhibitshigher training speed than DiT-S/2 and savesGPU memory at a resolution of.
Moreover, we analyze the scalability of DiG across a variety of computational complexity. DiG models, with increased depth/width or augmentation of input tokens, consistently exhibit decreasing FID.
We further compare DiG with other subquadratic-time diffusion models.
With the same model size, DiG-XL/2 isfaster than the recent Mamba-based diffusion model at aresolution, and isfaster than DiT with CUDA-optimized FlashAttention-2 under theresolution.
All these results demonstrate its superior efficiency among the latest diffusion models.