Combining the results in Section REF , we argue that this stagnation originates from the “cognitive speed” of the network for confidence. If there is no mapping like MLP or Softmax, the network will understand this knowledge (confidence) very slowly, and with the addition of MLP, this knowledge is transmitted to the entire network by parameters, so the network can quickly recognize and absorb this knowledge and accelerate the convergence of the network. On this basis, the Softmax function further accelerates this “cognitive speed”, hence both ResNet-50-S3-S and ResNet-50-S3-SR converge at the beginning and outperform ResNet-50-S3 in the initial convergence and final performance. Moreover, we believe that Softmax and Dropout [1]} can alleviate the fluctuation of accuracy, which is why there is no fluctuation except for ResNet-50-S3 in the 40th to 60th epoch. In addition, the reason why ResNet-50-S3-S is not better than the baseline is over-fitting caused by MLP, so the performance surpasses the baseline after adding the ReLu function [2]}.
