Last, cross-model complexity comparison is a promising direction.
Given several models with different model frameworks and different model sizes, how can we compare their expressive capacity? After these models are trained sufficiently on the same dataset, such as obtaining zero training error, how can we compare their effective complexity and further understand their generalization capability? In these cases, the cross-model complexity comparison is useful. Comparing model complexity crossing different deep models can in general help many problems of deep learning, in particular model selection and design.
However, the exploration of cross-model comparison of expressive capacity or effective complexity of deep learning models is still very limited.
Khrulkov et al. [1]} compare expressive capacity between shallow FCNNs, CNNs, and RNNs by connecting network architectures to tensor decompositions.
However, many more sophisticated models are not involved and need to be further explored.
