**Architecture Choice.** In earlier literature of pre-trained language models, there are lots of discussions on the effects of different architectures [89, 29]. However, most LLMs are developed based on the causal decoder architecture, and there still lacks a theoretical analysis on its advantage over the other alternatives. Next, we briefly summarize existing discussions on this issue.