Owing to the space limitation, we evaluate two popular neural networks, ResNet [1]} (CNN) and BERT [2]} (Transformer), which cover tasks from the computer vision and NLP domain.
We evaluate ResNet-50 for image classification on the ImageNet [3]} dataset. ResNet has been widely used as the backbone in many applications, e.g., Mask R-CNN for instance segmentation [4]}. For the state-of-the-art Transformer model family, we use BERT-base with 128 sequence length on GLUE (general language understanding evaluation) [5]} dataset. We only present the results on four datasets (MRPC, CoLA, SST-2, and MNLI).
Finally, we implement the activation functions Mish [6]}, Swish [7]}, and GELU [8]} on the networks respectively.
All the experiments are conducted on the NVIDIA RTX 2080Ti GPU [9]} with 11 GB of off-chip GDDR-based global memory.
