The exposure bias problem refers to the training-inference discrepancy caused by teacher forcing in maximum likelihood estimation (MLE) training for auto-regressive neural network language models (LM). It has been regarded as a central problem for natural language generation (NLG) model training. Although a lot of algorithms have been proposed to avoid teacher forcing and therefore to alleviate exposure bias, there is little work showing how serious the exposure bias problem is. In this work, we first identify the auto-recovery ability of MLE-trained LM, which casts doubt on the seriousness of exposure bias. We then develop a precise, quantifiable definition for exposure bias. However, according to our measurements in controlled experiments, there's only around 3% performance gain when the training-inference discrepancy is completely removed. Our results suggest the exposure bias problem could be much less serious than it is currently assumed to be. Language model (LM) is a central module for natural language generation (NLG) tasks (Young et al., 2017) such as machine translation (Wu et al., 2017) , dialogue response generation , image captioning (Lin et al., 2014) , etc. For decades, maximum likelihood estimation (MLE) has been the the most widely used objective for LM training. However, there is a popular belief in the natural language processing (NLP) community that standard MLE training will cause "exposure bias" and lead to a performance degradation during the test-time language generation. The exposure bias problem (Bengio et al., 2015; Ranzato et al., 2016) refers to the following discrepancy between MLE training and test-time generation for language models: During training, the language model predicts the next word conditioned on history words sampled from the groundtruth data distribution. And during generation, the model generates words conditioned on history sequences generated by the model itself. However, due to the exposure to real data during training, the language model is biased to only perform well on the ground-truth history distribution. As a result, during generation the errors will accumulate along the generated sequence, and the distribution generated by the model will be distorted. The forced exposure to ground-truth data during training is also referred to as "teacher forcing". Given its defintion, the exposure bias problem could rise in the general cases when the model needs to make a sequence of decisions or generations (e.g. music/pixel/speech generation (Lamb et al., 2016) ). In this work, we focus on the task of language generation, because the exposure bias problem is originally proposed in this field (Bengio et al., 2015) , and has since attracted huge research attention. In order to avoid teacher forcing, many training algorithms (Bengio et al., 2015; Lamb et al., 2016; Ranzato et al., 2016; Yu et al., 2016; Zhu et al., 2018; Lu et al., 2018; Lin et al., 2017; Guo et al., 2017; Rajeswar et al., 2017; Wiseman & Rush, 2016; Nie et al., 2019; Shi et al., 2018) have been proposed as alternatives to MLE training. Most of these works utilize techniques from generative adversarial network (GAN) (Goodfellow et al., 2014) or reinforcement learning (RL) (Sutton & Barto, 1998) . In this paper, we refer to these algorithms as non-MLE methods or text GANs. Despite the huge research efforts devoted to alleviate exposure bias, surprisingly, its existence or significance is much less studied. In particular, to the best of our knowledge, no existing work Table 1 : Samples of a MLE-trained STOA transformer LM when fed with different types of length-10 history prefix. To save space, we omitted the first 7 words of the random history. attempts to directly show the seriousness of exposure bias in an empirical or theoretical way. This work is motivated by the belief that a good solution should be built upon a testable and quantifiable problem definition. In this rest of this paper, we first identify the "self-recovery" ability of popular LM models, which casts doubt on the original claim of exposure bias. We then develop a precise and quantifiable definition of exposure bias, and validate its seriousness in controlled experiments. In this section, we focus on answering the following question: "Does the EB-M measurement correctly reflect the significance of exposure bias?" In short, our answer is not really. The problem is that the distortion of the marginal P l+1 M |M is not only affected by the presumably existing exposure bias problem alone, but also by the mismatch between the history distribution P M from P D for W 1:l , which grows with the length of the history. Therefore, even if the measured EB-M is significantly larger than one, we can not conclude that exposure bias causes serious deterioration. We provide an example to illustrate this argument: Example 1. Suppose L = 2, and V = {A, B}. P D and P M are crafted as follows: However, the only problem P M has is the mismatch between the history distributions (P M and P D ) for W 1 . The next set of experiments also suggest that EB-M does not precisely reflect exposure bias. On the EMNLP-news data-set (specified in Appendix B), we compare EB-M measurements for several non-MLE training methods with the baseline MLE model. We include results for Scheduled Sampling (SS) (Bengio et al., 2015) , Cooperative Training (CoT) (Lu et al., 2018) , and Adversarial Ranking (RankGAN) (Lin et al., 2017) . We provide implementation details for non-MLE methods in Appendix C. Intuitively, these methods will cause the model to be biased to behave well with model samples as history, instead of data samples. Therefore, we expect EB-M measurement for non-MLE trained models to be smaller than MLE trained models. However, Figure 1 shows that the measurements for different training frameworks are almost the same. We believe the reason is that the EB-M measurements are only reflecting the trivial mismatch between the history distributions. Is it possible that the original definition of exposure bias (Bengio et al., 2015; Ranzato et al., 2016) exactly refers to this mismatch between the model and data history distributions? However, note that this mismatch is inevitable for any imperfect model, and non-MLE training algorithms can not solve it. We believe a better, more precise definition is needed to discriminate exposure bias from this trivial mismatch. Motivated by this view, we propose a second approach in the section below. Following the discussion in the last section, we wish our measurement to be independent of the quality of the history distribution. In light of that, we design a quantity to measure the model's conditional generation quality. Let P H ∈ {P M , P D } denote the history distribution as in the MGD definition (5). With history length l fixed, we define the conditional generation deviation (CGD) with history distribution P H for P M using metric d as: where we assume that P D (· | W 1:l )) is computable, and use it to measure the quality of the model's conditional distribution. For the choice of the distribution distance d, in addition to d T V and d JS , we introduce greedy decoding divergence (d GD ) defined as: where 1 is the indicator function, and P, Q ∈ P. The distance d GD 2 reflects the model's accuracy during greedy decoding. Similar to MGD, exposure bias should imply a significant gap between CGD(P M |M , l, d) and CGD(P M |D , l, d). We again define rate of exposure bias at history length l with metric d to be: For our definition of EB-C, a natural question is why we only focus on the generation distribution of the very next word. The reason is we want to precisely measure how the error caused by the history part affect the generation part, by keeping them separate. If we measure the deviation of, for example, two sampled tokens, the definition will be confusing: Because the second sampled token will be affected not only by the accumulated error induced by the history (sampled from the model), but also by the first generated token as history. To get a better understanding of the intuition behind the definition of EB-C, we recommend readers to read Appendix A about our NMT experiment. Since CGD requires inference for ground-truth data distribution P D , we first consider experiments in a synthetic setting. In text-GAN literature (Yu et al., 2016; Lin et al., 2017) , a randomly-initialized one-layer LSTM model with hidden dimension of 32 is usually used as P D in synthetic experiments (we denote this setting as M random 32 ). However, the model is small-scale and does not reflect any structure existing in real-world text. To improve upon this approach, we take the MLE baseline model trained on EMNLP-news data (described in Appendix B) as P D in this synthetic setting. We denote the data model (P D ) as M news 512 . We then train two LSTM LM (P M ) with different capacities using samples from the data model, with the standard MLE objective. One is a one-layer LSTM with hidden width of 512 (denoted as LSTM-512), the other one is with hidden width of 32 (denoted as LSTM-32). We train P M for 100 epochs using the Adam optimizer with learning rate 0.001. In each epoch, 250k sentences (same to the size of the original EMNLP-news data) of length L = 50 are sampled from M news-512 as training data to avoid over-fitting. We show perplexity (PPL) results of the trained models in Appendix F. Finally, EB-C is calculated using 100k 3 samples from P M and P D . In Figure 2 , we show EB-C measurements with different metrics d m , and the two models give similar results. It is shown that EB-C has a steady but slow increasing trend as history length increases. This is expected as a consequence of exposure bias, because P M deviates farther from P D as history length increases. However, the average value of EB-C is less than 1.03 (the largest average value is from d JS for the LSTM-512 experiment), meaning that the gap between CGD(P M |M , l, d) and CGD(P M |D , l, d) is not large. Also, note that in most NLG applications (such as machine translation or image captioning), the generated sequence typically has short length (less than 20). In that range of history length, the EB-C measurements that exposure bias only has minimal influence. In Appendix E, we repeat the experiment for a transformer LM (Dai et al., 2019) , and get very similar EB-C measurements. These measurements imply a striking conclusion : (Informal) Even if all the bad effects from exposure bias for MLE LM training are removed, the relative performance gain is at most 3%. If the sequence length is not very long, the gain is less than 1%.. To dive deeper into the cause of the gap in CGD, we experiment with corrupted versions of P M as history distribution. We first specify a corrupt rate c ∈ [0, 1], and randomly substitute words in a history sample from P M to a "noise" word drawn uniformly from the vocabulary with probability c. Consequently, larger c will cause the history distribution to deviate farther from the groundtruth P D . In Figure 3 , we show CGD measurement versus the corrupted history P corrupt M . Large gaps are observed between CGD(P M |M corrupt ) and CGD(P M |D ). Therefore, the small gap between CGD(P M |M ) and CGD(P M |D ) in Figure 2 results from the small deviation between the history distribution P M and P D . In other word, P M has learned a "good enough" distribution that is able to keep it in the well-behaving region during sampling. With these observations, we conclude that, in the synthetic setting considered, exposure bias does exist, but is much less serious than it is presumed to be. Although there exists mismatch between the history distribution P M and P D , the mismatch is still in the model's "comfortable zone". In other words, the LSTM LM is more robust than exposure bias claims it to be. To concretize the this argument, we provide an example LM and show that MLE training is unlikely to generate models with a large EB-C value. Example 2. Again suppose L = 2, and V = {A, B}, the ground-truth data distribution is uniform on {AA, AB, BB, BA}. P M is crafted as follows: . Note that the model behaves bad when W 1 = A, which is of high probability during sampling. In this work, we first identify the self-recovery ability of MLE-trained LM, which casts doubt on the seriousness of exposure bias, which has been regarded as a central problem for MLE training by the LM community. We then explore two intuitive approaches to quantify the significance of exposure bias for LM training. The first quantification EB-M relies on the marginal generation distribution and reveals some vagueness in the original definition of exposure bias. We argue that we should focus on the model's generation performance in terms of its conditional distribution and propose a second quantification EB-C, which we regard as the precise definition for exposure bias. We design a evaluation of EB-C at different history length with real human (turkers from AMT) as the data model, for a SOTA transformer LM. It is shown that removing the training-testing discrepancy only gives around 2% of performance gain. Our synthetic experiments also gives very similar measurements. By analyzing EB-C measurements with perturbed history samples, we hypothesise that although the mismatch between the data and model distribution for history prefix exists, it is still in the model's "comfortable zone". With these results, we claim that on the contrary to the popular belief, exposure bias is only a minor problem in MLE-based LM training. To wrap up, we discuss the fundamental question "Is MLE training really biased?", from the perspective of objective functions. Note that the MLE objective (1) can be re-written as: where D KL denotes the Kullback-Leibler divergence, and θ denotes the trainable parameters in P M . Therefore, MLE training is minizing the divergence from P M , which is exactly the model's sampling distribution, from P D . While it's true that the training is "exposed" to data samples, we can not simply deduce the objective is "biased". We want to end our discussion with two remarks. First, the proposed quantification approaches should not be used as the only metric for NLG. For example, a position-aware uni-gram LM, which generates words independent of previous context, has no exposure bias problem and can pass our test easily. Second, the intention of this work is not to discourage researchers from exploring non-MLE training algorithms for LM. It is completely possible that an training objective different from , can lead to better generation performance (Lu et al., 2018; Huszr, 2015) . However, though non-MLE algorithms avoid teacher forcing, these algorithms (using GAN or RL for example) are usually less stable and more difficult to tune. Given that the quantified measurement of exposure bias is insignificant, we think it should be questioned whether adopting these techniques to avoid exposure bias is a wise trade-off.
We show that exposure bias could be much less serious than it is currently assumed to be for MLE LM training.
{"src": "scitldr-AIC_train", "paper_id": "rJg2fTNtwr"}