Stylized Dialogue Response Generation Using Stylized Unpaired Texts

Generating stylized responses is essential to build intelligent and engaging dialogue systems. However, this task is far from well-explored due to the difficulties of rendering a particular style in coherent responses, especially when the target style is embedded only in unpaired texts that cannot be directly used to train the dialogue model. This paper proposes a stylized dialogue generation method that can capture stylistic features embedded in unpaired texts. Specifically, our method can produce dialogue responses that are both coherent to the given context and conform to the target style. In this study, an inverse dialogue model is first introduced to predict possible posts for the input responses, and then this inverse model is used to generate stylized pseudo dialogue pairs based on these stylized unpaired texts. Further, these pseudo pairs are employed to train the stylized dialogue model with a joint training process, and a style routing approach is proposed to intensify stylistic features in the decoder. Automatic and manual evaluations on two datasets demonstrate that our method outperforms competitive baselines in producing coherent and style-intensive dialogue responses.


Introduction
Building a conversational agent that can produce stylized and coherent responses has been one of the major challenges in dialogue systems . Such an agent can not only yield more vivacious dialogues but also deliver more engaging conversations by taking advantage of the linguistic style matching phenomenon (Niederhoffer and Pennebaker, 2002), which suggests that people tend to adjust their linguistic style during communication to pursue higher engagement. * Corresponding Author: aihuang@tsinghua.edu.cn The pleasure is all mine, my lady.√ The pleasure is all mine, sir.  Figure 1: A pipelined approach to produce formal dialogue responses. For a post x, a response y is first produced using a dialogue model and then it is transferred to a formal response y using a text style transfer model.
Generating stylized dialogue responses has been investigated in various studies, where the definition of styles covers a variety of subtle concepts, such as sentiment (Shen et al., 2017b), emotion , or persona (Li et al., 2016b). Despite the success, previous studies are generally conducted in a fully supervised setting that requires to use dialogue pairs in the target style. However, in most cases, the stylistic features we want to capture are embedded in unpaired texts that can not be directly utilized by these supervised models .
Few studies for dialogue modeling have been proposed to capture the stylistic features embedded in unpaired texts. Specifically, Niu and Bansal (2018) employs a style-aware reinforce loss, and  resorts to a joint continuous latent space. However, despite the reported feasibility, we argue that due to the discrete nature of texts and subtle definition of text styles, it is hard to produce coherent and style-specific responses by relying on sparse reinforce signals or controlling continuous representations.
Note that we can also implement a straightforward stylized dialogue generation pipeline with the help of an unsupervised text style transfer model (Hu et al., 2017), which can be trained using stylized unpaired texts. Specifically, for a post x, a non-stylized dialogue response y is first generated using a regular dialogue model, and then y is transferred to a stylized response y using a text style transfer model. However, this approach may hurt the coherence between x and y since the style transferring process is unaware of x and thus may introduce improper contents. As shown in Figure 1, the style transfer model generates a strong stylistic word "sir" to emphasize the formality of y. However, this makes y incoherent with x since x is most likely to be issued by a female.
In this paper, we propose to build a stylized dialogue generation model that can capture stylistic features embedded in a set of unpaired texts D s . Specifically, in order to tackle the problem of lacking stylized dialogue pairs, an inverse dialogue model is built to predict posts based on the responses, and a set of stylized pseudo dialogue pairs are constructed by producing pseudo posts for texts in D s . Then a stylized dialogue model is trained using these pseudo pairs, and a joint training process is introduced to enhance the coherency between the post and the resulting responses. Moreover, our dialogue models are parameterized using the Transformer-based encoderdecoder framework, and initialized with the pretrained GPT weights (Radford et al., 2018). A style routing approach is devised to fuse a style embedding in each decoder block of the stylized dialogue model to intensify the stylistic features in the decoding process.
We evaluate our method on two datasets with two distinct writing styles: 1) Jinyong novels 1 in Chinese, and 2) formality in English writing. Automatic and human evaluations show that our method significantly outperforms competitive baselines with a large margin in generating coherent dialogue responses while rendering stronger stylistic features.
Our contributions can be summarized as: 1) A novel method is proposed to build a stylized dialogue model that can capture stylistic features embedded in unpaired texts. Specifically, an inverse dialogue model is introduced to generate stylized pseudo dialogue pairs, which are further utilized in a joint training process. An effective style routing approach is devised to further intensify the stylistic features in the decoder. 1 Jinyong is a famous Chinese writer who wrote many Kung Fu novels.
2) Automatic and human evaluations on two datasets show that our method outperforms competitive baselines with a large margin in producing stylized and coherent dialogue responses.

Related Work
Stylized dialogue generation has attracted numerous attentions in recent years Niu and Bansal, 2018). With a rather wide definition of styles, various studies that focus on controllable dialogue generation have been categorized as "stylized" dialogue generation, such as generating personalized (Li et al., 2016b) or emotional  dialogues. However, the training process of these dialogue model usually require dialogue pairs in the target style, whereas our study aims to capture stylistic features embedded in unpaired texts.
Moreover, the styles defined in most previous studies are deeply fused with the text contents (Tikhonov et al., 2019). Enforcing these styles may limit the expressive ability of the dialogue model because there are contradictions between certain semantic contents and style categories. For example, it is hard, if not impossible, for a service agent to yield comforting contents when enforcing a negative sentiment. Unlike most previous works, our study investigates to model the writing styles that are "orthogonal" to the text semantic, so that the contents we want to deliver will not be constrained by the style we intend to render.
Text style transfer is a related but different task compared to our work. Specifically, these text style transfer models aim to preserve the style-agnostic contents of the input text (Fu et al., 2018). In contrast, our study aims to produce coherent responses rather than to preserve the contents of the posts. Early works on this task focus to disentangle the representation of styles and contents (Hu et al., 2017;Shen et al., 2017a;Prabhumoye et al., 2018). However, recent studies argue the effectiveness of such disentanglement (Lample et al., 2019), and propose to revise the latent codes using classifiers (Liu et al., 2020;Wang et al., 2019). Some works are also proposed to render the target styles by replacing stylistic words (Wu et al., 2019a,b).
We have also noticed a recent work that considers a contextual constraint in the text style transferring process (Cheng et al., 2020). However, although being feasible, the training of this model requires style-labelled parallel data. This hinders us from directly employing this model in our study since these parallel data are usually unavailable. Back translation is a popular approach that has been widely employed in various NLP tasks such as machine translation (Sennrich et al., 2016), dialogue data augmentation (Su et al., 2020), and text style transfer Lample et al., 2019;Dai et al., 2019). This approach is similar to the inverse dialogue model introduced in our study. However, different from previous approaches that focus on modeling the one-to-one mapping between the source and target languages, our inverse dialogue model tries to capture the oneto-many mappings between the responses and posts with the help the proposed joint training process. In our study, the diversity of the generated pseudo posts are enhanced using a sampling approach.

Task Definition
In this study, we propose to build a stylized dialogue model without utilizing dialogue pairs in the target style. Specifically, our method takes as input two sets of data in the training stage: 1) M unpaired texts D s = {t 1 , ..., t M } in the writing style S 1 ; 2) N dialogue pairs D p = { x 1 , y 1 , ..., x N , y N } with style S 0 , where x i and y i is the post and response, respectively. Our stylized dialogue model aims to generate a response y that is coherent to a given post x while exhibiting a certain style S i (i = 0, 1): (1)

Model Overview
Our model consists of two mirrored sub-modules ( Figure 2): (1). A stylized dialogue module (i.e., e and d in Figure 2) that can produce a stylized response y based on a given post x and a style label S i (i = 0, 1). A style routing approach is devised  (2). An inverse dialogue module (i.e.,ê andd in Figure 2) that aims to produce pseudo posts x based on an input response y. Note that the inverse dialogue model is introduced to tackle the problem of lacking dialogue pairs in style S 1 , i.e., we can regard the texts in D s as possible dialogue responses and use the predicted pseudo posts to construct pseudo dialogue pairs in style S 1 . Therefore, we omit the style label in the inverse decoderd to encourage it to focus more on the semantic aspect of the dialogue.
The dialogue modules in our study are parameterized using the Transformer-based encoder and decoder architecture (Vaswani et al., 2017) and are initialized using pretrained GPT (Radford et al., 2019) weights. Further, we also follow previous works (Golovanov et al., 2019) to share the weights of the encoder and decoder from the same submodule to save memories. Particularly, the weights of e and d are shared, and the weights ofê andd are shared.
Moreover, to better capture the one-to-many phenomenon and alleviate the problem of producing trivial posts in the inverse dialogue model, a top-k sampling scheme is employed to sample multiple pseudo posts for each stylized text in D s , and all these posts are utilized in the training process. Further, a joint training process is also introduced to train these two sub-modules in an iterative fashion to enhance the coherency of the response.

Style Routing
There exist various approaches to condition the decoder d on the style label. For example, employing a special style token as the start token (Lample et al., 2019), or adding a style embedding to each word embedding . However, these approaches only incorporate the style representation in the input layer of the decoder, whereas the higher layers are not affected.
In this study, a style routing approach is devised to enhance existing approaches to stylize d in the stylized dialogue model (see Figure 3). Specifically, in each decoder block, we first fuse the representation of the post x and previously decoded token sequence y p using the attention routing mechanism , i.e., two sequences of representations, R prev , R post ∈ R l×h , are first calculated: where e w (y p ) ∈ R l×h denotes the embedding of y p and it is used as the query in MMHA and MHA, which represent the masked and un-masked multi-head attention operation, respectively. l is the length of y p , and h is the hidden size. e w (x) is the output of the encoder. A sequence of fused representations R avg is obtained as: Then for a given style S i , a style embedding e s (S i ) ∈ R 1×h is allocated and e s (S i ) is routed into R avg by adding it to each time step of the sequence: Also note that the fusion operation in Eq. 4 and 5 is similar to some previous studies that try to incorporate additional contexts in a transformerbased decoder (Golovanov et al., 2019). However, different from these approaches that focus to model sequential contexts, the styles modeled in our study are categorical, and more priority is allocated to the style representation in our model. Moreover, we are the first to use such style routing approach in the stylized dialogue generation task.

Joint Training
The training of our model involves the following losses: 1) standard maximum log likelihood losses Algorithm 1 Joint training process Step > N f then 7: Dpp ← empty set. 8: Sample ns stylized texts D b s = {ti} ns i=1 ⊂ Ds 9: for each ti ∈ D b s do 10: Decode m posts {x ij } m j=1 from pd(x|ê(ti)) 11: The loss L p2r and L r2p is used to train the stylized dialogue model and inverse dialogue model, respectively; 2) an inverse dialogue loss evaluated on texts from D s : in which x is the pseudo post sampled from the inverse dialogue model.
Note that the gradient back-propagation through the loss L inv is intractable due to the indifferentiable sampling process in Eq. 8. In this study, we approximate the ideal back-propagation process through L inv by truncating the gradients associated with the sampling operation. Specifically, when optimizing L inv , the parameters of the inverse dialogue model are fixed, and the stylized dialogue model is trained with pseudo posts x that are sampled from the inverse dialogue model. Similar approaches have been proven to be effective in other NLP tasks (Lample et al., 2018;He et al., 2020). However, unlike previous works that use the greedy decoding scheme, our study employs the top-k sampling scheme with beam search to produce x since the mapping between dialogue responses and posts is not unique. The greedy decoding scheme may limit the diversity of the decoded pseudo posts and lead to sub-optimal performance.  To facilitate the learning with the above gradient approximation approach, a joint training process is introduced to train the model literately. Specifically, in each training iteration, we first update the stylized and inverse dialogue model by optimizing the losses L p2r and L r2p using a batch of dialogue pairs sampled in D p . Further, a batch of stylized sentences D b s are sampled from D s . For each sentence t i ∈ D b s , m pseudo posts x i1 , ..., x im are sampled from the inverse dialogue model, and m pseudo dialogue pairs x ij , t i , (j = 1, ..., m) in the style S 1 are constructed. These pseudo pairs are used to train the stylized dialogue model with the loss L inv . Moreover, to avoid corrupted pseudo posts at the beginning of the training process, we pre-train the inverse dialogue model on L r2p for N f steps before using it to decode pseudo posts. The detailed training process is summarized in Algorithm 1.

Dataset
Our method is evaluated on two datasets with two distinct styles (see statistics in Table 1).
1) WDJN: We collect 300K Weibo Dialogues (style S 0 ) as D p and sampled 95.1K stylized texts from Jinyong's Novels (style S 1 ) as D s . Moreover, we also extracted 2K dialogue pairs from Jinyong's novels with hand-designed rules. These dialogues are used as the test set D t together with 2K additional Weibo dialogues. Note that all the Weibo dialogues in our WDJN dataset (both training and testing) are manually inspected and filtered by annotators.
Also note that to prevent the model from copying stylistic phrases in the post when producing Jinyong style responses in the testing phase, we erase the stylistic features related to Jinyong's writing from the posts in these 2K Jinyong style dialogues in D t using the back translation approach . Moreover, all the resulting posts are manually checked and revised to ensure the stylis-tic features related to style S 1 are erased. More details about the WDJN dataset can be find the Appendix A. The WDJN dataset will be released for public use.
2) TCFC (Wu et al., 2020): This dataset focuses on the formality in English writing. We sampled 217.2K informal dialogue pairs (style S 0 ) as D p and 500.0K formal texts (style S 1 ) as D s from the original dataset, and used the test data in the original dataset as our test set D t , which contains 1,956 manually-crafted dialogue pairs (978 informal pairs and 978 formal pairs).

Implementation Details
For experiments on the WDJN and TCFC dataset, we used the pre-trained CDial-GPT  and DialoGPT (size 345M)  model to initialize the dialogue modules, respectively. The top-K sampling process in Algorithm 1 employs a K = 20 and beam size of 4 (WDJN) or 2 (TCFC). The value of N f is set to 300. The training of our model stops after 10 iteration epochs on D p (WDJN) or after 8,000 steps of updates (TCFC). See Appendix B for more details of the reproduction guidance.

Baselines
We choose two groups of baselines: The first group contains dialogue models with different style modeling scheme: 1) S2S (Golovanov et al., 2019): a strong Transformer-based dialogue model that is only trained on D p . This baseline can only produce responses in style S 0 ; 2) SLM: the "Fusion" model proposed by Niu and Bansal (2018), in which an independent stylized language model is trained on D s , and the distributions decoded from the S2S baseline and the stylized LM are fused when producing responses in style S 1 ; 3) SRL: the "RL" model proposed by Niu and Bansal (2018), in which a reinforce signal produced by a style classifier is used to enforce the style S 1 ; 4) SFusion : A fused latent space is built using a multi-task training scheme. Specifically, for each post, six responses are sampled, and two classifiers are used to rank these responses for the styles.
The second group of baselines are built using the pipelined approach, i.e., different unsupervised text style transfer models are trained on texts from D s and D p , and responses produced by the S2S baseline (in style S 0 ) are transferred to exhibit the target   Table 3: Automatic and manual evaluation results for responses with style S 0 . All differences between our model and baselines are significant with p-value <0.01 except for the ones marked with *.
style S 1 using these models: 5) S2S+BT: a backtranslation-based text style transfer model (He et al., 2020); 6) S2S+CT: a model that tries to entangle the latent code for styles and contents (Wang et al., 2019); 7) S2S+PTO: a model that renders the target style by replacing stylistic words (Wu et al., 2019a). Note that for baselines SLM, SRL and all the baselines in the second group, the responses generated by the S2S baseline are used as their responses for the S 0 style since they can only produce responses in S 1 once trained. Moreover, for fair comparisons, we implemented baselines 1-3 using the same architecture and hyper-parameters as in our model. For baselines 4-7, we used the official codes released by the authors. Note that it is non-trivial to utilize the pre-trained GPT model in the baseline SFusion since it handles fixed-length latent codes.

Automatic Evaluation
Metrics: We first used automatic metrics to evaluate the response quality of our model: 1). BLEU (Papineni et al., 2002) was used to measure n-gram (n=1, 2) overlap between the generated responses and the reference responses; 2). Distinct (Dist.) (Li et al., 2016a) measures the proportion of unique n-grams in the generated responses (n=2).
To evaluate the style intensity of the each model, we first trained two text style classifiers (i.e., BERT (Devlin et al., 2019) and SVM) and then calculated the style intensity score as the portion of generated responses that conform to the target style based on these classifiers. In our study, the texts from D p and D s were used to train the classifiers for the WDJN experiments, and the formal/informal texts from the GYAFC dataset (Rao and Tetreault, 2018) were used to train the classifiers for the TCFC experiments. The accuracy of the BERT and SVM classifier on the holdout test set was 98.52% and 94.20% respectively for the WDJN experiments, and 93.98% and 89.57% respectively for the TCFC experiments (see Appendix C for more details).   Table 4: Automatic and manual evaluation results of ablation models for responses with style S 0 and S 1 . All differences between our model and ablation models are significant with p-value <0.05 except for the ones marked with *.

Results
in style S 1 (Table 2) and S 0 ( Table 3). Note that the baseline S2S is not included in Table 2 since it can not produce responses in style S 1 . Similarly, only the baselines S2S and SFusion are contained in Table 3. Significance tests are performed between the results of our model and all the baselines using the t-test with bootstrap resampling (Koehn, 2004). As can be seen from the automatic results, our method outperforms all the baselines with large margins when generating dialogue responses in style S 1 (Table 2), and achieves competitive performance when producing responses in style S 0 (Table 3). This indicates that our model can produce high quality responses that are both coherent to the given context and consistent to the target style. We can further observe that: 1). The pipelined approaches achieve lower BLEU scores comparing to our method. This verifies our claim that the response coherency is affected by the style transferring process. Similar results are also observed in manual evaluation. 2). The high diversity (i.e., Dist. scores) of the baselines on the TCFC dataset come along with a dramatic decrease on the BLEU scores. This is because that these baselines overfit to the diverse colloquial phrases in the informal responses, and fail to render responses in style S 1 , which are more formal and less diverse.
Also note that the style intensity scores for human generated responses (last row in Table 2 and 3) do not match the accuracy of our style classifiers. This is because that the train data of these classifiers involve non-conversational texts, which leads to mismatches when testing using conversational responses. To alleviate this mismatch, we performed manual evaluations to concrete our analysis.

Manual Evaluation
Metrics: For a given post, dialogue responses with different styles were generated using our model and all the baselines. Three annotators were re-cruited from the crowd-sourcing platform to evaluate these responses from three aspects: 1) Fluency (Flu.): whether the response is fluent and free from grammar errors; 2) Coherency (Coh.): whether the response is coherent with the dialogue context; 3) Style Intensity (Style): whether the response conforms to the given style. Each metric is rated among {0, 1, 2}, in which 0 means worst and 2 best. Moreover, the Harmonic Average (i.e, HAvg.) of above measures is also reported.
Results: We sampled 300 posts from D t for each of these two datasets. Fleiss's kappa κ (Randolph, 2005) was used to measure the annotation agreement between annotators. Specifically, for Flu., Coh., and Style, the κ value was 0.69, 0.50, 0.86, respectively on the WDJN dataset (indicating substantial, moderate, and substantial agreement), and 0.44, 0.31, 0.42, respectively on the TCFC dataset (indicating moderate, fair, and moderate agreement).
As shown in Table 2, our model surpasses all the baselines significantly on style intensity (except for S2S+CT on the WDJN dataset, which comes with dramatic decreases on the fluency and coherency scores) when producing responses in style S 1 , and it achieves competitive or higher fluency and coherency scores. This verifies the superiority of our method in producing coherent and style intensified dialogue responses. Moreover, results in table 3 also shows that our model achieved competitive performance when generating responses in style S 0 .
Another interesting observation from the results in Table 2 and 3 is the trade-off between the coherency and style intensity when generating stylized dialogue responses, i.e., the high style intensity usually comes at the cost of a low coherency. For example, the model S2S+CT achieves the best style intensity score on the WDJN dataset (1.50) when producing responses in style S 1 , but it obtains the  worst coherency (0.19) score. This phenomenon is also observed in various previous studies (Niu and Bansal, 2018;. Nevertheless, our model achieves a competitive coherency while producing style-intensive responses. Also note that the baselines SFusion, SRL, and S2S+CT generally yield low HAvg. scores on both datasets. This verifies our claim that it is hard to generate stylized and coherent responses relying on the sparse reinforce signals (i.e., SRL) or continuous latent codes (i.e., SFusion and S2S+CT).
The superiority of our method to generate stylized dialogue responses is further demonstrated by analyzing the style intensity scores of responses with different levels of coherency. Specifically, all the annotated responses that are generated with a designated style of S 1 were collected and categorized into three groups based on the coherency scores (i.e., 0, 1, or 2) they received. The averaged style intensity score for each group was calculated and shown in Figure 4. It can be seen that our model achieves the highest style intensity scores in all coherency groups. This further demonstrates that the responses produced by our method are more style-intensive than those by the baselines.

Ablation Study
Ablation studies were performed to verify the effect of each component in our method. Specifically, the following variants were tested: 1) without the style routing approach (w/o Rout.), i.e., the style embedding is not explicitly incorporated in each decoder block as in Eq.5. The decoder d is stylized by employing a style token as the start token and adding a style embedding to each word embedding; 2) Without the joint training process (w/o JointT), i.e., an inverse dialogue model is first trained, and then a fixed set of pseudo pairs are generated and used to train the stylized dialogue model. Note that the same amount of pseudo pairs were used to optimize the loss L inv in this variant as it is used in Algorithm 1; 3) Without using the top-K sampling scheme when producing pseudo posts (w/o Samp.), i.e., pseudo pairs are decoded greedily; 4) Without using the pre-trained GPT weights (w/o PreT).
As shown in Table 4, our model achieves the highest BLEU and Coh. scores among all the ablation models. We can further observe that: 1) Almost all our variants surpass the baselines with a large margin on the style intensity score. This verifies the feasibility of our framework in capturing stylistic features; 2) Removing the joint training process (w/o JointT) or the top-K sampling scheme (w/o Samp.) makes the dialogue models over-fit to render more stylistic features while failing to achieve high BLEU and Coh. scores. However, we argue that since our stylized decoder is already strong in capturing stylistic features, it is critical to utilize the proposed joint training and top-K sampling scheme to improve the response coherency; 3) The pre-training approach significantly improves the diversity and coherency of the generated responses. Table 5 shows some dialogue responses generated by our model and the baselines on the two datasets. We can observe that the models that directly manipulate the continuous latent space (i.e., SFusion and S2S+CT) yield non-fluent responses. This is TCFC dataset Pseudo Post: Are you enjoying the new album? Text in S1: Yes, I am. I am loving her last cd. Pseudo Post: I'm so tired of golf. Text in S1: What is the point of golf? Pseudo Post: Hey, are you going to the game tonight? Text in S1: Hardly, I live up north. Maybe next time.

WDJN dataset
Pseudo Post: I am very, very sad today (今天的我,伤心的 不得了) Text in S1: Are your hurt? (你受伤了么？) Pseudo Post: Did anyone come to see me today?
(大哥，有人来啦。) Pseudo Post: I'm going to kill you today (今天我要杀了你) Text in S1: Dude, you could have killed me, but you didn't.
(老兄，刚才你本可杀我，没有下手。) because that it hard to build a smooth latent space for discrete texts. Moreover, pipelined approaches either fail to convert the inputs to the target style (i.e., S2S+PTO on the WDJN dataset), or hurt the coherency between the response and the post (i.e., S2S+BT, S2S+CT, and S2S+PTO on the TCFC dataset). In addition, we sampled some of these pseudo pairs generated by the inverse dialogue model in the training phase (Table 6). It can be seen that these pseudo pairs are generally of high quality both in fluency and coherency.

Conclusion
In this paper, we present a stylized dialogue generation method that can produce coherent and styleintensive responses by utilizing stylized unpaired texts. An inverse dialogue model is introduced in our method to produce stylized pseudo dialogue pairs, which are used in a joint training process to train the stylized dialogue model. Further, a style routing approach is introduced to intensify stylistic features in the decoding process. We demonstrate our method on two datasets with two different styles: Chinese Jinyong novels and formality in English writing. Automatic and manual evaluation shows that our method outperforms competitive baselines in producing coherent and style-intensive responses. As future works, we will extend this method to other stylized text generation tasks.