Several related works have explored different aspects of state-space models (SSMs) and their applications in various domains. [181] propose a deep state-space model (DSSM) called Continuous Learning DSSM (CLDSSM). CLDSSM integrates regularization-based continual learning (CL) methods to efficiently update multiple dynamic systems without catastrophic forgetting. [172] focuses on enhancing the robustness of SSMs for long sequence tasks through approximate diagonalization. The authors propose a method that approximates diagonalization to improve the robustness of SSMs. By simplifying the problem and considering pure diagonal structures, the proposed approach achieves computational efficiency and allows channel communication. In [171], the _curse of memory_ in SSMs is addressed through stable re-parameterization. The authors introduce a parameterization technique for SSMs that effectively enhances their memory limits. [164] provide a data-dependent generalization bound for SSMs, highlighting the interplay between the SSM parameters and the temporal dependencies of training sequences. Building upon this generalization bound, they propose a scaling rule for model initialization to improve the robustness of SSMs in accommodating different temporal patterns in the sequence data. Wang et al. [160] propose a sequence routing method based on State Space Model and attempt to pre-train a big model without attention. Bidirectional Gated SSM (BiGS) proposed in the paper combines an SSM layer and a multiplicative gating architecture to effectively simplify the sequence modelling task. It reduces computational resources and time for model training by omitting the computationally intensive attention matrix. This approach successfully reduces resource consumption by about \(30\%\) while maintaining comparable performance to traditional pre-trained models. Although the BiGS model does not consider pairwise interactions, it is able to match BERT's pre-training accuracy on the GLUE benchmark test and can be scaled up to 4096 tokens of long-form pre-training without approximation. This approach provides an effective pre-training solution for NLP tasks in resource-limited environments and has the potential to be applied to a wider range of NLP scenarios.