<cite>Tu and Gimpel (2018)</cite> pretrained their tag language model (TLM) on a large, automatically-tagged corpus and fixed its parameters when optimizing Θ. We instead do not pretrain the TLM and learn its parameters when training Θ.