**Emergent Architectures.** The conventional Transformer architectures typically suffer from quadratic computational complexity. Because of this, efficiency has become an important issue when training and making inference with long inputs. To improve efficiency, some studies aim to devise new architectures for language modeling, including parameterized state space models (_e.g.,_ S4 [247], GSS [248], and H3 [249]), long convolutions like Hyea [250], and Transformer-like architectures that incorporate recursive update mechanisms (_e.g.,_ RWKV [251] and RetNet [252]). The key merits of these new architectures are twofold. First, these models can generate outputs recursively like RNNs, meaning that they only need to refer to the single previous state during decoding. It makes the decoding process more efficient as it eliminates the need to revisit all previous states as in conventional Transformers. Second, these models have the capacity to encode an entire sentence in parallel like Transformers. This contrasts with conventional RNNs which has to encode sentences on a token-by-token basis. Thus, they can benefit from the parallelism of GPUs with techniques such as Parallel Scan [253, 254], FFT [250, 251], and Chunkwise Recurrent [252]. These techniques enable models with these new architectures to be trained in a highly parallel and efficient manner.