A Dataset and Baselines for Multilingual Reply Suggestion
Reply suggestion models help users process emails and chats faster. Previous work only studies English reply suggestion. Instead, we present MRS, a multilingual reply suggestion dataset with ten languages. MRS can be used to compare two families of models: 1) retrieval models that select the reply from a fixed set and 2) generation models that produce the reply from scratch. Therefore, MRS complements existing cross lingual generalization benchmarks that focus on classification and sequence labeling tasks. We build a generation model and a retrieval model as baselines for MRS. The two models have different strengths in the monolingual setting, and they require different strategies to generalize across languages. MRS is publicly available at https:/github.com/zhangmozhi/mrs. 1 Multilingual Reply Suggestion Automated reply suggestion is a useful feature for email and chat applications. Given an input message, the system suggests several replies, and users may click on them to save typing time (Figure 1) This feature is available in many applications including Gmail, Outlook, LinkedIn, Facebook Messenger, Microsoft Teams, and Uber. Reply suggestion is related to but different from open domain dialog systems or chatbots (Adiwardana et al. 2020; Huang et al. 2020) While both are conversational AI tasks (Gao et al. 2019) the goals are different: reply suggestion systems help the user quickly reply to a message, while chatbots aim to continue the conversation and focus more on multi turn dialogues. Ideally, we want our model to generate replies in any language. However, reply suggestion models require large training sets, so previous work mostly *Work mostly done as an intern at Microsoft Research. +Work done at Microsoft Research. Figure 1: An example of reply suggestion system. User can click on the suggestions for a quick reply. focuses on English (Kannan et al. 2016; Henderson et al. 2017; Deb et al. 2019) To investigate reply suggestion for other languages with possibly limited data, we build a multilingual dataset, dubbed MRS (Multilingual Reply Suggestion) From publicly available Reddit threads, we extract messagereply pairs, response sets, and machine translated examples in ten languages (Table 1) One interesting aspect of the reply suggestion problem is that there are two modeling approaches. Some models follow the retrieval framework and select the reply from a predetermined response set (Henderson et al. 2017) Others follow the generation framework and generate the reply from scratch (Kannan et al. 2016) The two approaches have different advantages. Generation models are more powerful because they are not constrained by the response set. In comparison, retrieval models are easier to train and runs faster, and a curated response set guarantees the coherence and the safety of the model output. The two frameworks make reply suggestion an interesting task for studying cross lingual generalization. Most cross lingual generalization benchmarks use classification and sequence labeling tasks (Tjong Kim Sang, 2002; Nivre et al. 2016; Strassel and Tracey, 2016; Conneau et al. 2018; Schwenk and Li, 2018; Clark et al. 2020; Hu et al. 2020; Lewis et al. 2020b) In contrast, reply suggestion has two formulations that require different cross lingual generalization strategies. While some recent work explores cross lingual transfer ar X iv :2 10 6. 02 01 7v 1 cs .C L 3 J un 2 02 1 Language Code Family Examples Tokens Response Set English EN West Germanic 48,750,948 1,700,066,696 36,997 Spanish ES Romance 2,325,877 195,424,517 45,152 German DE West Germanic 1,864,688 118,711,662 34,747 Portuguese PT Romance 1,822,594 114,642,809 45,225 French FR Romance 1,396,806 133,068,740 32,350 Japanese JA Japonic 727,668 46,124,966 38,817 Swedish SV North Germanic 738,254 47,845,497 32,165 Italian IT Romance 736,296 58,715,043 31,855 Dutch NL West Germanic 638,634 43,847,547 32,293 Russian RU East Slavic 516,739 23,109,295 31,475 Table 1: Dataset statistics for MRS. We collect Reddit message reply pairs for ten language. For each language, we use 80% examples for training, 10% for validation, and 10% for testing. We then create response sets for retrieval models. We also use MT to translate nineteen million English training examples to other languages. learning in generation tasks, the tasks are extractive; i.e. the output often has significant overlap with the input. These tasks include news title generation, text summarization, and question generation (Chi et al. 2020; Liang et al. 2020; Scialom et al. 2020) Reply suggestion is more challenging because the reply often does not overlap with the message (Figure 1) so the model needs to address different cross lingual generalization challenges (Section 5.2) We build two baselines for MRS: a retrieval model and a generation model. We first compare the models in English, where we have abundant training data and human referees. We evaluate the models with both automatic metrics and human judgments. The two models have different strengths. The generation model has higher word overlap scores and is favored by humans on average, but inference is slower, and the output is sometimes contradictory or repetitive (Holtzman et al. 2020) In contrast, the retrieval model is faster and always produces coherent replies, but the replies are sometimes too generic or irrelevant due to the fixed response set. Next, we test models in other languages. We compare different training settings and investigate two cross lingual generalization methods: initializing with pre trained multilingual models (Wu and Dredze, 2019; Conneau et al. 2020; Liang et al. 2020) and training on machine translated data (Banea et al. 2008) Interestingly, the two models prefer different methods: multilingual pretraining works better for the retrieval model, while the generation model prefers machine translation. In summary, we present MRS, a multilingual reply suggestion dataset. We use MRS to provide the first systematic comparison between generation and retrieval models for reply suggestion in both monolingual and multilingual settings. MRS is also a useful benchmark for future research in reply suggestion and cross lingual generalization. The rest of the paper is organized as follows. Section 2 describes the data collection process for MRS. Section 3 introduces task formulations, experiment settings, and evaluation metrics. Section 4 describes the baseline generation and retrieval models. Section 5 presents our experiment results. Section 6 discusses how MRS can help future research. 2 Dataset Construction To study reply suggestion in multiple languages, we build MRS, a dataset with message reply pairs based on Reddit comments. The dataset is available at https:/github.com/zhangmozhi/mrs. We download Reddit comments between January 2010 and December 2019 from the Pushshift Reddit dataset (Baumgartner et al. 2020).1 We extract message reply pairs from each thread by considering the parent comment as an input message and the response to the comment as the reference reply. We remove comments starting with [removed] or [deleted] which are deleted messages. We also skip comments with a rating of less than one, since they are likely to contain inappropriate content. After extracting examples, we identify their languages with fastText language detector (Joulin et al. 2016) For each example, we run the model https:/files.pushshift.io/reddit/
ACL/IJCNLP