\(\bullet\)_FlashAttention_. Different from most existing approximate attention methods that trade-off model quality to improve the computing efficiency, FlashAttention [283] proposes to optimize the speed and memory consumption of attention modules on GPUs from an IO-aware perspective. There exist different levels of memory on modern GPUs, _e.g._, SRAM with a fast IO and HBM with a relatively slow IO. FlashAttention organizes the input into blocks and introduces necessary recomputation, both to make better use of the fast memory SRAM. Implemented as a fused kernel in CUDA, FlashAttention has been integrated into PyTorch [197], DeepSpeed [74], and Megatron-LM [75]. The updated version FlashAttention-2 [284] further optimizes the work partitioning of GPU thread blocks and warps, leading to around 2\(\times\) speedup when compared to the original FlashAttention.