Let's consider LLaMA 13B (\(d=128\)) with a sequence length of 1024 (\(s=1024\)) as an example. The calculated arithmetic intensity is \(1.97\). However, the A100 80G GPU can perform \(312\) TFLOPs and transfer 2 TB of data in one second, _i.e._, its ideal arithmetic intensity is \(156\). This indicates that the bottleneck in attention calculation lies in the process of data transfer (_i.e._, excessive I/O loading).