Non-autoregressive Transformer is a promising text generation model . However , current non-autoregressive models still fall behind their autoregressive counterparts in translation quality . We attribute this accuracy gap to the lack of dependency modeling among decoder inputs . In this paper , we propose CNAT , which learns implicitly categorical codes as latent variables into the non-autoregressive decoding . The interaction among these categorical codes remedies the missing dependencies and improves the model capacity . Experiment results show that our model achieves comparable or better performance in machine translation tasks than several strong baselines . Non-autoregressive Transformer ( NAT , Gu et al . , 2018 ; Wang et al . , 2019 ; Lee et al . , 2018 ; Ghazvininejad et al . , 2019 ) is a promising text generation model for machine translation . It introduces the conditional independent assumption among the target language outputs and simultaneously generates the whole sentence , bringing in a remarkable efficiency improvement ( more than 10× speed-up ) versus the autoregressive model . However , the NAT models still lay behind the autoregressive models in terms of BLEU ( Papineni et al . , 2002 ) for machine translation . We attribute the low-quality of NAT models to the lack of dependencies modeling for the target outputs , making it harder to model the generation of the target side translation . A promising way is to model the dependencies of the target language by the latent variables . A line of research works ( Kaiser et al . , 2018 ; Roy et al . , 2018 ; Shu et al . , 2019 ; Ma et al . , 2019 ) introduce latent variable modeling to the non-autoregressive Transformer and improves translation quality . The latent variables could be regarded as the springboard to bridge the modeling gap , introducing more informative decoder inputs than the previ-ously copied inputs . More specifically , the latentvariable based model first predicts a latent variable sequence conditioned on the source representation , where each variable represents a chunk of words . The model then simultaneously could generate all the target tokens conditioning on the latent sequence and the source representation since the target dependencies have been modeled into the latent sequence . However , due to the modeling complexity of the chunks , the above approaches always rely on a large number ( more than 2 15 , Kaiser et al . , 2018 ; Roy et al . , 2018 ) of latent codes for discrete latent spaces , which may hurt the translation efficiencythe essential goal of non-autoregressive decoding . Akoury et al . ( 2019 ) introduce syntactic labels as a proxy to the learned discrete latent space and improve the NATs ' performance . The syntactic label greatly reduces the search space of latent codes , leading to a better performance in both quality and speed . However , it needs an external syntactic parser to produce the reference syntactic tree , which may only be effective in limited scenarios . Thus , it is still challenging to model the dependency between latent variables for non-autoregressive decoding efficiently . In this paper , we propose to learn a set of latent codes that can act like the syntactic label , which is learned without using the explicit syntactic trees . To learn these codes in an unsupervised way , we use each latent code to represent a fuzzy target category instead of a chunk as the previous research ( Akoury et al . , 2019 ) . More specifically , we first employ vector quantization ( Roy et al . , 2018 ) to discretize the target language to the latent space with a smaller number ( less than 128 ) of latent variables , which can serve as the fuzzy word-class information each target language word . We then model the latent variables with conditional random fields ( CRF , Lafferty et al . , 2001 ; Sun et al . , 2019 ) . To avoid the mismatch of the training and inference for latent variable modeling , we propose using a gated neural network to form the decoder inputs . Equipping it with scheduled sampling ( Bengio et al . , 2015 ) , the model works more robustly . Experiment results on WMT14 and IWSLT14 show that CNAT achieves the new state-of-theart performance without knowledge distillation . With the sequence-level knowledge distillation and reranking techniques , the CNAT is comparable to the current state-of-the-art iterative-based model while keeping a competitive decoding speedup . We propose CNAT , which implicitly models the categorical codes of the target language , narrowing the performance gap between the nonautoregressive decoding and autoregressive decoding . Specifically , CNAT builds upon the latent Transformer and models the target-side categorical information with vector quantization and conditional random fields ( CRF ) model . We further employ a gated neural network to form the decoder inputs . Equipped with the scheduled sampling , CNAT works more robust . As a result , the CNAT achieves a significant improvement and moves closer to the performance of the Transformer on machine translation . For each latent variable , we list the top 3 frequent pos tags and their corresponding percentages .
Non-autoregressive translation models fall behind their autoregressive counterparts in translation quality due to the lack of dependency modelling for the target outputs.
They propose a non-autoregressive transformer-based model which implicitly learns categorical codes as latent variables into the decoding to complement missing dependencies.
The proposed model achieves state-of-the-art without knowledge distillation and a competitive decoding speedup with an interactive-based model when coupled with knowledge distillation and reranking techniques.