While activation functions have been studied thoroughly for their impact on accuracy [1]}, [2]}, their impact on speed is less well understood. With the high computing capacity on TPUs and GPUs, the FLOPs difference among different activation functions is negligible. However, because of the low operational intensity of all activation functions and the shape of rooflines (Figure REF ) of TPU and GPU, all activation functions are memory-bound [3]} on TPUv3 and GPUv100. These memory-bound activation functions can have large negative performance impact to the end-to-end model speed, because they can drag the overall model into the slope region of the rooflines (where ML model performance is far away from the TPU/GPU peak performance as shown in Figure REF .
