**Prefix Decoder Architecture.** The prefix decoder architecture (_a.k.a._, non-causal decoder [244]) revises the masking mechanism of causal decoders, to enable performing bidirectional attention over the prefix tokens [245] and unidirectional attention only on generated tokens. In this way, like the encoder-decoder architecture, the prefix decoders can bidirectionally encode the prefix sequence and autoregressively predict the output tokens one by one, where the same parameters are shared during encoding and decoding. Instead of pre-training from scratch, a practical suggestion is to continually train causal decoders and then convert them into prefix decoders for accelerating convergence [29], _e.g.,_ U-PaLM [118] is derived from PaLM [56]. Existing representative LLMs based on prefix decoders include GLM-130B [93] and U-PaLM [118].