Neural sequence generation is commonly approached by using maximum- likelihood (ML) estimation or reinforcement learning (RL). However, it is known that they have their own shortcomings; ML presents training/testing discrepancy, whereas RL suffers from sample inefficiency. We point out that it is difficult to resolve all of the shortcomings simultaneously because of a tradeoff between ML and RL. In order to counteract these problems, we propose an objective function for sequence generation using α-divergence, which leads to an ML-RL integrated method that exploits better parts of ML and RL. We demonstrate that the proposed objective function generalizes ML and RL objective functions because it includes both as its special cases (ML corresponds to α → 0 and RL to α → 1). We provide a proposition stating that the difference between the RL objective function and the proposed one monotonically decreases with increasing α. Experimental results on machine translation tasks show that minimizing the proposed objective function achieves better sequence generation performance than ML-based methods. Neural sequence models have been successfully applied to various types of machine learning tasks, such as neural machine translation Sutskever et al., 2014; , caption generation (Xu et al., 2015; BID6 , conversation (Vinyals & Le, 2015) , and speech recognition BID8 BID2 . Therefore, developing more effective and sophisticated learning algorithms can be beneficial. Popular objective functions for training neural sequence models include the maximum-likelihood (ML) and reinforcement learning (RL) objective functions. However, both have limitations, i.e., training/testing discrepancy and sample inefficiency, respectively. indicated that optimizing the ML objective is not equal to optimizing the evaluation metric. For example, in machine translation, maximizing likelihood is different from optimizing the BLEU score BID19 , which is a popular metric for machine translation tasks. In addition, during training, ground-truth tokens are used for the predicting the next token; however, during testing, no ground-truth tokens are available and the tokens predicted by the model are used instead. On the contrary, although the RL-based approach does not suffer from this training/testing discrepancy, it does suffer from sample inefficiency. Samples generated by the model do not necessarily yield high evaluation scores (i.e., rewards), especially in the early stage of training. Consequently, RL-based methods are not self-contained, i.e., they require pre-training via ML-based methods. As discussed in Section 2, since these problems depend on the sampling distributions, it is difficult to resolve them simultaneously.Our solution to these problems is to integrate these two objective functions. We propose a new objective function α-DM (α-divergence minimization) for a neural sequence generation, and we demonstrate that it generalizes ML-and RL-based objective functions, i.e., α-DM can represent both functions as its special cases (α → 0 and α → 1). We also show that, for α ∈ (0, 1), the gradient of the α-DM objective is a combinations of the ML-and RL-based objective gradients. We apply the same optimization strategy as BID18 , who useed importance sampling, to optimize this proposed objective function. Consequently, we avoid on-policy RL sampling which suffers from sample inefficiency, and optimize the objective function closer to the desired RL-based objective than the ML-based objective.The experimental results for a machine translation task indicate that the proposed α-DM objective outperforms the ML baseline and the reward augmented ML method (RAML; BID18 , upon which we build the proposed method. We compare our results to those reported by BID3 , who proposed an on-policy RL-based method. We also confirm that α-DM can provide a comparable BLEU score without pre-training.The contributions of this paper are summarized as follows.• We propose the α-DM objective function using α-divergence and demonstrate that it can be considered a generalization of the ML-and RL-based objective functions (Section 4).• We prove that the α-DM objective function becomes closer to the desired RL-based objectives as α increases in the sense that the upper bound of the maximum discrepancy between ML-and RL-based objective functions monotonically decreases as α increases.• The results of machine translation experiments demonstrate that the proposed α-DM objective outperforms the ML-baseline and RAML (Section 7). In this study, we have proposed a new objective function as α-divergence minimization for neural sequence model training that unifies ML-and RL-based objective functions. In addition, we proved that the gradient of the objective function is the weighted sum of the gradients of negative loglikelihoods, and that the weights are represented as a mixture of the sampling distributions of the ML-and RL-based objective functions. We demonstrated that the proposed approach outperforms the ML baseline and RAML in the IWSLT'14 machine translation task.In this study, we focus our attention on the neural sequence generation problem, but we expect our framework may be useful to broader area of reinforcement learning. The sample inefficiency is one of major problems in reinforcement learning, and people try to mitigiate this problem by using several type of supervised learning frameworks such as imitation learning or apprenticisip learning. This alternative approaches bring another problem similar to the neural sequence generaton problem that is originated from the fact that the objective function for training is different from the one for testing. Since our framework is general and independent from the task, our approach may be useful to combine these approaches. A GRADIENT OF α-DM OBJECTIVEThe gradient of α-DM can be obtained as follows: DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 where DISPLAYFORM3 In Eq. FORMULA0 , we used the so-called log-trick: ∇ θ p θ (y|x) = p θ (y|x)∇ θ log p θ (y|x).
Propose new objective function for neural sequence generation which integrates ML-based and RL-based objective functions.
{"src": "scitldr-AIC_train", "paper_id": "H1Nyf7W0Z"}