Khrulkov et al. [1]} propose the rank of tensor decomposition as a measure of neural network complexity, since the rank of decompositions corresponds to the width of networks. Based on the correspondence relationships between neural networks and tensor decompositions, they compare the model complexity of RNNs, CNNs, and shallow FCNNs.
The main conclusions are summarized in Table REF . In particular, given a random \(d\) -dimensional tensor whose TT-decomposition is with rank \(r\)  and mode size \(n\) , this tensor has exponentially large ranks of CP-decomposition and HT-decomposition. That tells, to approximate a recurrent neural network, a shallow FCNN or CNN requires an exponentially larger width.
