The U-PaLM models of 8B, 62B, and 540B scales are continually trained on PaLM with UL2R, a method of continue training LLMs on a few steps with UL2's mixture-of-denoiser objective [73]. An approximately 2x computational savings rate is reported.