@inproceedings{
wu2024separation,
title={Separation and Bias of Deep Equilibrium Models on Expressivity and Learning Dynamics},
author={Zhoutong Wu and Yimu Zhang and Cong Fang and Zhouchen Lin},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=UJ9k3j93MD}
}
The deep equilibrium model (DEQ) generalizes the conventional feedforward neural network by fixing the same weights for each layer block and extending the number of layers to infinity. This novel model directly finds the fixed points of such a forward process as features for prediction.  Despite empirical evidence showcasing its efficacy 
compared to feedforward neural networks, a theoretical understanding for its separation and bias is still limited.  In this paper, we take a step
by proposing some separations and studying the bias of DEQ in its expressive power and learning dynamics.   The results include: (1) A general separation is proposed, showing the existence of a width-$m$ DEQ that any fully connected neural networks (FNNs) with depth $O(m^{\alpha})$ for $\alpha \in (0,1)$ cannot
approximate unless its width is sub-exponential in $m$; (2) DEQ with polynomially bounded size and magnitude can efficiently approximate certain steep functions (which has very large derivatives) in $L^{\infty}$ norm, whereas FNN with bounded depth and exponentially bounded width cannot unless its weights magnitudes are exponentially large; (3) The implicit regularization caused by gradient flow from a diagonal linear DEQ is characterized, with specific examples showing the benefits brought by such regularization. 
From the overall study, a high-level conjecture from our analysis and empirical validations is that DEQ has potential advantages in learning certain high-frequency components.
Separation and Bias of Deep Equilibrium Models on Expressivity and Learning Dynamics
Zhoutong Wu Yimu Zhang Cong Fang Zhouchen Lin