For experimentation we use ResNet50 [1]} with GELU [2]} non-linear activation function and pre-activation's [3]} at every bottleneck layers. With the same given set of hyperparameters and traing the two different models as shown in Figure REF  we could clearly see that loss on the test set is decrease by almost a factor of 1.5 (Also increasing the test accuracy) which could be helpful for predicting the images that are not the same domain of our train set.
<FIGURE>