Emerging datacenter accelerators, including TPUs [1]}, [2]} and GPUs, have been using new hardware architectures to keep up with the fast-increasing demand of computing power from ML models. In particular, because matrix-multiplication is the core operation in neural networks, the most special feature of these accelerators is the matrix-multiply-and-accumulate units, called tensor cores [3]} in GPUs and matrix multiply units [1]}, [2]} in TPUs. These new hardware architectures have changed the way ML models execute on the accelerators. For example, recent work [6]} has observed that FLOPs and latency do not correlate on these accelerators. However, with these empirical observations, there is yet no in-depth analysis to reveal the root cause.
