To address this issue, researchers formally quantify data transfer from GPU memory to GPU caches using the number of bytes in I/O, and they assess weight computation by measuring the number of FLOPs [320]. Specifically, let \(b\), \(s\), \(n\), \(d\), and \(h\) denote the batch size, sequence length, number of attention heads, hidden size of each head, and overall hidden size (\(h=n\cdot d\)), respectively. During the layer-wise multi-head self-attention calculation in causal decoder, the I/O bytes and FLOPs at each decoding step can be expressed as \(8bsn+4bsnd+4bnd\) and \(8bsnd\), respectively [320].