Sentence Simplification with Transformer XL and Paraphrase Rules
Neural sentence simplification is an emerging technology which aims to automatically simplify a sentence with complex syntax and/or diction while preserving its meaning. Most of the existing work on neural sentence simplification has modeled the task as monolingual machine translation (MT) and has investigated applications of architectures such as RNNor Transformer based seq2seq encoder decoder networks. In this project, we explore a new model by adapting Transformer XL, the cutting edge architecture for language modeling (LM) to sentence simplification. In addition, we propose and investigate two original approaches to incorporate the Simple Paraphrase Database (PPDB) a large database of reduction rules for words and phrases. The character level Transformer XL model achieved performance close to SOTA. Though our experiments using the SimplePPDB variants do not achieve desirable results, they reveal important insight regarding the integration of accurate paraphrase rules into neural models. This project is the first known application of Transformer XL to a non LM task, as well as the first known sentence simplification model that can use character embeddings.