Though some progress has been made, effective complexity measures are still a largely under-developed direction in deep learning.
Comparing to expressive capacity of deep learning models, measuring effective model complexity is even more challenging. Effective complexity measures have to be capable of capturing fine granularity differences between two models, such as the same model architecture with two different optimization algorithms.
Several previous studies [1]}, [2]}, [3]}, [4]}, [5]} define and explore effective complexity measures mainly from the perspective of piecewise linear property [1]}, [4]}, [5]}, Fisher-Rao metric [2]}, or the size of trainable samples [3]}. However, the effective complexity measure is still a largely unexplored and valuable direction.
