Structural planning is important for producing long sentences, which is a missing part in current language generation models. In this work, we add a planning phase in neural machine translation to control the coarse structure of output sentences. The model first generates some planner codes, then predicts real output words conditioned on them. The codes are learned to capture the coarse structure of the target sentence. In order to learn the codes, we design an end-to-end neural network with a discretization bottleneck, which predicts the simplified part-of-speech tags of target sentences. Experiments show that the translation performance are generally improved by planning ahead. We also find that translations with different structures can be obtained by manipulating the planner codes. When human speaks, it is difficult to ensure the grammatical or logical correctness without any form of planning. Linguists have found evidence through speech errors or particular behaviors that indicate speakers are planning ahead BID16 . Such planning can happen in discourse or sentence level, and sometimes we may notice it through inner speech.In contrast to human, a neural machine translation (NMT) model does not have the planning phase when it is asked to generate a sentence. Although we can argue that the planning is done in the hidden layers, however, such structural information remains uncertain in the continuous vectors until the concrete words are sampled. In tasks such as machine translation, a source sentence can have multiple valid translations with different syntactic structures. As a consequence, in each step of generation, the model is unaware of the "big picture" of the sentence to produce, resulting in uncertainty of word prediction. In this research, we try to let the model plan the coarse structure of the output sentence before decoding real words. As illustrated in FIG0 , in our proposed framework, we insert some planner codes into the beginning of the output sentences. The sentence structure of the translation is governed by the codes.An NMT model takes an input sentence X and produce a translation Y . Let S Y denotes the syntactic structure of the translation. Indeed, the input sentence already provides rich information about the target-side structure S Y .For example, given the Spanish sentence in FIG0 , we can easily know that the translation will have a noun, a pronoun and a verb. Such obvious structural information does not have uncertainty, and thus does not require planning. In this example, the uncertain part is the order of the noun and the pronoun. Thus, we want to learn a set of planner codes C Y to disambiguate such uncertain information about the sentence structure. By conditioning on the codes, we can potentially increase the effectiveness of beam search as the search space is properly regulated.In this work, we use simplified POS tags to annotate the structure S Y . We learn the planner codes by putting a discretization bottleneck in an end-to-end network that reconstructs S Y with both X and C Y . The codes are merged with the target sentences in the training data. Thus, no modification to the NMT model is required. Experiments show the translation performance is generally improved with structural planning. More interestingly, we can control the structure of output sentences by manipulating the planner codes. Instead of learning discrete codes, we can also directly predict the structural annotations (e.g. POS tags), then translate based on the predicted structure. However, as the simplified POS tags are also long sequences, the error of predicting the tags will be propagated to word generation. In our experiments, doing so degrades the performance by around 8 BLEU points on IWSLT dataset. In this paper, we add a planning phase in neural machine translation, which generates some planner codes to control the structure of the output sentence. To learn the codes, we design an end-to-end neural network with a discretization bottleneck to predict the simplified POS tags of target sentences. Experiments show that the proposed method generally improves the translation performance. We also confirm the effect of the planner codes, by being able to sample translations with drastically different structures using different planner codes.The planning phase helps the decoding algorithm by removing the uncertainty of the sentence structure. The framework described in this paper can be extended to plan other latent factors, such as the sentiment or topic of the sentence.
Plan the syntactic structural of translation using codes
{"src": "scitldr-AIC_train", "paper_id": "HyeuFOcMyX"}