Reinforcement learning (RL) is frequently used to increase performance in text generation tasks,
 including machine translation (MT), 
 notably through the use of Minimum Risk Training (MRT) and Generative Adversarial Networks (GAN). 
 However, little is known about what and how these methods learn in the context of MT. 
 We prove that one of the most common RL methods for MT does not optimize the 
 expected reward, as well as show that other methods take an infeasibly long time to converge.
 In fact, our results suggest that RL practices in MT are likely to improve performance
 only where the pre-trained parameters are already close to yielding the correct translation.
 Our findings further suggest that observed gains may be due to effects unrelated to the training signal, concretely, changes in the shape of the distribution curve. Reinforcement learning (RL) is an appealing path for advancement in Machine Translation (MT), as it allows training systems to optimize non-differentiable score functions, common in MT evaluation, as well as tackling the "exposure bias" (Ranzato et al., 2015) in standard training, namely that the model is not exposed during training to incorrectly generated tokens, and is thus unlikely to recover from generating such tokens at test time. These motivations have led to much interest in RL for text generation in general and MT in particular (see §2). Various policy gradient methods have been used, notably REINFORCE (Williams, 1992) and variants thereof (e.g., Ranzato et al., 2015; Edunov et al., 2018) and Minimum Risk Training (MRT; e.g., Och, 2003; . Another popular use of RL is for training GANs (Yang et al., 2018; Tevet et al., 2018) . Nevertheless, despite increasing interest and strong results, little is known about what accounts for these performance gains, and the training dynamics involved. We present the following contributions. First, our theoretical analysis shows that commonly used approximation methods are theoretically ill-founded, and may converge to parameter values that do not minimize the risk, nor are local minima thereof ( §2.2). Second, using both naturalistic experiments and carefully constructed simulations, we show that performance gains observed in the literature likely stem not from making target tokens the most probable, but from unrelated effects, such as increasing the peakiness of the output distribution (i.e., the probability mass of the most probable tokens). We do so by comparing a setting where the reward is informative, vs. one where it is constant. In §4 we discuss this peakiness effect (PKE). Third, we show that promoting the target token to be the mode is likely to take a prohibitively long time. The only case we find, where improvements are likely, is where the target token is among the first 2-3 most probable tokens according to the pretrained model. These findings suggest that REINFORCE ( §5) and CMRT ( §6) are likely to improve over the pre-trained model only under the best possible conditions, i.e., where the pre-trained model is "nearly" correct. We conclude by discussing other RL practices in MT which should be avoided for practical and theoretical reasons, and briefly discuss alternative RL approaches that will allow RL to tackle a larger class of errors in pre-trained models ( §7). Implementing a stochastic gradient ascent, REINFORCE is guaranteed to converge to a stationary point of R under broad conditions. However, not much is known about its convergence rate under the prevailing conditions in NMT. We begin with a qualitative, motivating analysis of these questions. As work on language generation empirically showed, RNNs quickly learn to output very peaky distributions (Press et al., 2017) . This tendency is advantageous for generating fluent sentences with high probability, but may also entail slower convergence rates when using RL to fine-tune the model, because RL methods used in text generation sample from the (pretrained) policy distribution, which means they mostly sample what the pretrained model deems to be likely. Since the pretrained model (or policy) is peaky, exploration of other potentially more rewarding tokens will be limited, hampering convergence. Intuitively, REINFORCE increases the probabilities of successful (positively rewarding) observations, weighing updates by how rewarding they were. When sampling a handful of tokens in each context (source sentence x and generated prefix y <i ), and where the number of epochs is not large, it is unlikely that more than a few unique tokens will be sampled from P θ (·|x, y <i ). (In practice, k is typically between 1 and 20, and the number of epochs between 1 and 100.) It is thus unlikely that anything but the initially most probable candidates will be observed. Consequently, REINFORCE initially raises their probabilities, even if more rewarding tokens can be found down the list. We thus hypothesize the peakiness of the distribution, i.e., the probability mass allocated to the most probable tokens, will increase, at least in the first phase. We call this the peakiness-effect (PKE), and show it occurs both in simulations ( §4.1) and in full-scale NMT experiments ( §4.2). With more iterations, the most-rewarding tokens will be eventually sampled, and gradually gain probability mass. This discussion suggests that training will be extremely sample-inefficient. We assess the rate of convergence empirically in §5, finding this to be indeed the case. A histogram of the update size (x-axis) to the total predicted probability of the 10 most probable tokens (left) or the most probable token (right) in the Constant Reward setting. An update is overwhelmingly more probable to increase this probability than to decrease it. In this paper, we showed that the type of distributions used in NMT entail that promoting the target token to be the mode is likely to take a prohibitively long times for existing RL practices, except under the best conditions (where the pretrained model is "nearly" correct). This leads us to conclude that observed improvements from using RL for NMT are likely due either to fine-tuning the most probable tokens in the pretrained model (an effect which may be more easily achieved using reranking methods, and uses but little of the power of RL methods), or to effects unrelated to the signal carried by the reward, such as PKE. Another contribution of this paper is in showing that CMRT does not optimize the expected reward and is thus theoretically unmotivated. A number of reasons lead us to believe that in our NMT experiments, improvements are not due to the reward function, but to artefacts such as PKE. First, reducing a constant baseline from r, so as to make the expected reward zero, disallows learning. This is surprising, as REINFORCE, generally and in our simulations, converges faster where the reward is centered around zero, and so the fact that this procedure here disallows learning hints that other factors are in play. As PKE can be observed even where the reward is constant (if the expected reward is positive; see §4.1), this suggests PKE may play a role here. Second, we observe more peakiness in the reinforced model and in such cases, we expect improvements in BLEU (Caccia et al., 2018) . Third, we achieve similar results with a constant reward in our NMT experiments ( §5.2). Fourth, our controlled simulations show that asymptotic convergence is not reached in any but the easiest conditions ( §5.1). Our analysis further suggests that gradient clipping, sometimes used in NMT (Zhang et al., 2016; Wieting et al., 2019) , is expected to hinder convergence further. It should be avoided when using REINFORCE as it violates REINFORCE's assumptions. The per-token sampling as done in our experiments is more exploratory than beam search (Wu et al., 2018) , reducing PKE. Furthermore, the latter does not sample from the behavior policy, but does not properly account for being off-policy in the parameter updates. Adding the reference to the sample S, which some implementations allow (Sennrich et al., 2017) may help reduce the problems of never sampling the target tokens. However, as Edunov et al. (2018) point out, this practice may lower results, as it may destabilize training by leading the model to improve over outputs it cannot generalize over, as they are very different from anything the model assigns a high probability to, at the cost of other outputs. The standard MT scenario poses several uncommon challenges for RL. First, the action space in MT problems is a high-dimensional discrete space (generally in the size of the vocabulary of the target language or the product thereof for sentences). This contrasts with the more common scenario studied by contemporary RL methods, which focuses mostly on much smaller discrete action spaces (e.g., video games (Mnih et al., 2015; 2016) ), or continuous action spaces of relatively low dimensions (e.g., simulation of robotic control tasks (Lillicrap et al., 2015) ). Second, reward for MT is naturally very sparse -almost all possible sentences are "wrong" (hence, not rewarding) in a given context. Finally, it is common in MT to use RL for tuning a pretrained model. Using a pretrained model ameliorates the last problem. But then, these pretrained models are in general quite peaky, and because training is done on-policy -that is, actions are being sampled from the same model being optimized -exploration is inherently limited. Here we argued that, taken together, these challenges result in significant weaknesses for current RL practices for NMT, that may ultimately prevent them from being truly useful. At least some of these challenges have been widely studied in the RL literature, with numerous techniques developed to address them, but were not yet adopted in NLP. We turn to discuss some of them. Off-policy methods, in which observations are sampled from a different policy than the one being currently optimized, are prominent in RL (Watkins & Dayan, 1992; Sutton & Barto, 1998) , and were also studied in the context of policy gradient methods (Degris et al., 2012; Silver et al., 2014) . In principle, such methods allow learning from a more "exploratory" policy. Moreover, a key motivation for using α in CMRT is smoothing; off-policy sampling allows smoothing while keeping convergence guarantees. In its basic form, exploration in REINFORCE relies on stochasticity in the action-selection (in MT, this is due to sampling). More sophisticated exploration methods have been extensively studied, for example using measures for the exploratory usefulness of states or actions (Fox et al., 2018) , or relying on parameter-space noise rather than action-space noise (Plappert et al., 2017) . For MT, an additional challenge is that even effective exploration (sampling diverse sets of observations), may not be enough, since the state-action space is too large to be effectively covered, with almost all sentences being not rewarding. Recently, diversity-based and multi-goal methods for RL were proposed to tackle similar challenges (Andrychowicz et al., 2017; Ghosh et al., 2018; Eysenbach et al., 2019) . We believe the adoption of such methods is a promising path forward for the application of RL in NLP. Let θ be a real number in [0, 0.5], and let P θ be a family of distributions over three values a, b, c such that:
Reinforcment practices for machine translation performance gains might not come from better predictions.
{"src": "scitldr-AIC_train", "paper_id": "H1eCw3EKvH"}