Investigation of Sentiment Controllable Chatbot

Conventional seq2seq chatbot models attempt only to find sentences with the highest probabilities conditioned on the input sequences, without considering the sentiment of the output sentences. In this paper, we investigate four models to scale or adjust the sentiment of the chatbot response: a persona-based model, reinforcement learning, a plug and play model, and CycleGAN, all based on the seq2seq model. We also develop machine-evaluated metrics to estimate whether the responses are reasonable given the input. These metrics, together with human evaluation, are used to analyze the performance of the four models in terms of different aspects; reinforcement learning and CycleGAN are shown to be very attractive.


I. INTRODUCTION
In contrast to goal-oriented dialogue systems [1], [2], chatbot chats with human users on any subject domain of daily life [3], [4]. The conventional chatbot is based on the seq2seq model [5], generating meaningful responses given user input. It is usually emotionless, which is a major limitation of modern chatbots as emotion plays a critical role in human social interaction, especially in chatting [6]. Hence we seek to train the chatbot to generate responses with scalable sentiment by setting the chat mode. For example, for the input "How was your day today?", the chatbot may respond, "It is wonderful today" or "It is terrible today" depending on the sentiment set, in addition to simply generating a reasonable response. This mode can either be set by the developer or the user, or determined dynamically based on the dialogue context. The techniques mentioned here may be extended to conversational style adjustment, so the machine may imitate the conversational style of someone the user is familiar with, to make the chatbot more friendly or more personal [7], [8].
Substantial effort has been focused on the conversational fluency and content quality of generated responses, for example, by enriching the content diversity [9]- [11], considering additional information [12], and addressing unknown words [13], [14]. Responses have also been generated with controllable factors. The sentiment of a given sentence can be modified using non-parallel data [15]. A chatbot can change the style of responses by optimizing a given sentiment-related function [16]. However, little work has been reported on scaling the sentiment of a chatbot; it remains difficult to evaluate a chatbot with adjustable sentiment properly [17], [18].
This work was financially supported by the Ministry of Science and Technology of Taiwan.
In this paper, we investigate four approaches to scale the sentiment of chatbot responses and use a set of evaluation metrics and human evaluation with which we analyze the approaches. This journal paper is an extension of a conference paper [19], but with additional results on two more corpora.

II. RELATED WORK
The approaches presented in Sections III-B, III-C, III-D and III-E are related to Sections II-A1, II-A2, II-B1 and II-B2, respectively.

A. Controllable Sentence Generation
Sentence generators based on deep learning show promising text generation capabilities, but cannot easily control the generated text. Hence, a series of research aims at controlling the generated sentences, for example, the writing styles or the topics of the generated sentences.
1) Controlled by Input Factors: Sentence generation models can take some factors as input to influence the generation of its outputs. Conditioned recurrent neural networks (CRNN) are used to control the linguistic style of generated text [20]. Affect-LM customizes the degree of emotional content in generated sentences through an additional design parameter [21]. The conditional transformer language model (CTRL) [22] trains to condition on control codes that govern style, content, and task-specific behavior.
This category of approach has been used in dialogue generation. The persona model [12] encodes personas in distributed embeddings that capture individual characteristics such as background information and speaking style, and the embeddings influence the output of the decoder. Instead of encoding personas, in Section III-B, the persona-based model is used to control sentiment. A conversational model is proposed to generate informative responses with controlled sentence function, for example, interrogative, imperative, declarative, etc [23]. This paper is closely related to the Emotional Chatting Machine (ECM) [24]. ECM is a neural conversational model that can generate corresponding responses based on given emotional categories. The basic idea of ECM is similar to the persona-based model, but with more sophisticated network architectures, including internal and external memories. However, it needs the dialogues involving emotional responses to train the model, which is not always available.
2) Controlled by External Function: In this approach, the sentence generation model is explicitly taught to generate sentences with certain aspects [25]- [28]. A hand-crafted or machine-learned function guide the sentence generator to output sentences considered having the desired aspect (like sentiment or topic) based on the function. This approach has been used to make the dialogue generation model generate emotional responses, but hand-crafted functions are used in the previous work [25]. It has been shown that the attribute models learned from data can successfully guide the sentence generator [26]. Because here we focus on dialogue generation, in Section III-C, besides considering the attribute, or sentiment, of the responses, we further guide the model to generate the responses coherent with the input sentences by coherence models.

B. Text Style Transfer
The text style transfer model transfers the input sentence from one style into another. The text style transfer approaches that do not utilize parallel data are used in Sections III-D and III-E. Below are two main categories of approaches to achieve text style transfer without parallel data. All the related work mentioned below only focuses on text style transfer, not dialogue generation as in Section III.
1) Manipulating Latent Space: The latent representations of auto-encoders can be manipulated to induce a change in the output space to achieve text style transfer [16], [29]. The approach used in Section III-D belongs to this category. One way to manipulate latent space to achieve text style transfer is feature disentangle. By separating the content from the style in the latent space, we can modify the style without changing the content [15], [30]- [32].
2) Direct Modification: Instead of manipulating the latent space, this category of approach directly finds a model that can transform the text from one style to another. The approach used in Section III-E belongs to this category. A simple approach is to delete phrases associated with the sentence's style and retrieve new phrases to replace them [33]. The idea similar to CycleGAN [34] or StarGAN [35], which has widely used in image style transfer, has also been used. This category of approaches uses a discriminator to control the style of the generated content [36], [37], and reconstruction loss to maintain the content [38].

III. SENTIMENT CONTROLLABLE CHATBOT
In Section III-A we briefly review the conventional seq2seq chatbot. The four approaches used here are presented in Sections III-B to III-E. All use the seq2seq chatbot as the basic model. The persona-based approach (Section III-B) and reinforcement learning (Section III-C) modify the training algorithm of the conventional seq2seq chatbot. Plug and play (Section III-D) and CycleGAN (Section III-E) modify the response of an off-the-shelf seq2seq chatbot. Below we assume that the chatbot response is to be positive conditioned on the input, although it is simple to generalize the approaches to scalable sentiment.

A. Seq2seq Model
Here we use the attention-based seq2seq model [39] shown in Figure 1 to train a simple chatbot using a corpus of dialogue pairs. In all discussions here, x is the input sentence to the seq2seq chatbot, and y is the output of the seq2seq model.ŷ is the reference response in the training corpus. In the training phase, we input the sentence x (a sequence of one-hot vectors) to the encoder, and the seq2seq model learns to maximize the probability of generating the sentenceŷ given x.  The persona-based model was originally proposed to generate sentences that mimic the responses of specific speakers [12]. It is very similar to the seq2seq model, except that extra information is added to the input of the decoder at each time step. In the original work [12], this extra information is the trained speaker embedding. Here we replace the speaker embedding with a sentiment score (a scalar between 0 and 1) from a sentiment classifier, as shown in Figure 2. This sentiment classifier [40] is trained on a corpus of sentences with labeled sentiments to determine whether a sentence is positive or not. The input of the classifier is a sentence z, and the output is a score SC(z) between 0 and 1 indicating how positive the input is. The input of the decoder at every time step is then the concatenation of the word embedding and a sentiment score. During training the sentiment score of the reference sentence SC(ŷ) is used, and the decoder learns to generate the reference sentence. For testing given the same input, we scale the sentiment of the output by entering the desired sentiment score.

C. Reinforcement Learning
Here we use exactly the seq2seq chatbot shown in Figure 1; the only modification is a set of reward functions designed to scale the response sentiment using reinforcement learning. The components of the reward functions are developed as follows.
1) Semantic Coherence 1: In addition to being a good sentence, the response y should be semantically relevant to the input x. Hence we pre-train a different seq2seq model on a large dialogue corpus to estimate this semantic coherence with a probability P coh (y|x). The first reward is therefore where x and y denote the input and response of the baseline seq2seq chatbot (not the pre-trained seq2seq model), and N y is the length of y for normalization. 2) Semantic Coherence 2: The semantic coherence mentioned above can be estimated in a completely different way. We use the same dialogue corpus to train a RNN discriminator, in which two RNN encoders are used to represent the input x and its corresponding response y as two embeddings; these two embeddings are concatenated and followed by a fully connected layer to produce a score D RNN (x, y) between 0 and 1 which indicates whether x and y are good dialogue pairs. This score is therefore the second reward: 3) Sentiment Score: The third reward is based on the sentiment classifier mentioned in Section III-B: where y is the seq2seq chatbot response. The total reward is then the linear interpolation of the three rewards mentioned above: where α and β are hyper-parameters ranging from 0 to 1 and α + β < 1. We employ the reinforcement learning algorithm with policy gradient [41]. As shown in Figure 3, to generate dialogue responses here, we borrow the concept of plug and play as used in generating images [42]. Here we pre-train a variational recurrent autoencoder (VRAE) [43] in addition to using the same dialogue corpus. The VRAE encoder on the left transforms a sentence y into a fixed-length latent vector h 0 , while the VRAE decoder on the middle right generates a sentence y based on a vector h . The VRAE encoder and decoder are also jointly learned from the dialogue corpus for the chatbot.

D. Plug and Play Model
The following steps take place on-line, when the user enters a sentence. Given an input x, the seq2seq baseline first generates a response y which is then encoded into a latent code h 0 by the VRAE encoder. Then the latent code h 0 is modified into h , based on the following equation: (5) where SC denotes the sentiment classifier and γ and δ are the weights of the loss function term and the regularization term. The first term on the right-hand side of Eq. (5) indicates that we seek a code h such that when decoded into a sentence Decoder(h) using the VRAE decoder, the resulting sentiment score SC(Decoder(h)) is maximized. The second term of Eq. (5) prevents the code h from drifting too far from h 0 . To solve Eq. (5), we calculate the gradient of the sentiment score with respect to the latent code h and apply gradient ascent to the latent code iteratively, until the sentiment score output reaches a pre-defined value. Because Eq. (5) is solved on-line after the user enters an input sentence, this approach is more time consuming. Since the argmax layer between the decoder and sentiment classifier in SC(Decoder(h)) is nondifferentiable, we use soft argmax [44] to approximate argmax so that the gradient back-propagates throughout the whole network, from the sentiment classifier to the decoder. Here we adopt the very powerful cycle generative adversarial network (CycleGAN), which proved successful in image style transformation even without paired data [34]. As illustrated in Figure 4, we show a way to use CycleGAN to transform the sentiment of sentences from negative to positive. The model is trained on two sets of sentences in a corpus with labeled sentiments: a positive sentiment set P and a negative sentiment set N . The sentences in the two sets are unpaired; that is, for a given sentence in P, it is not known which is the corresponding sentence in N . We train two seq2seq translators: G to transform a negative sentence y n to positive and F for positive y p to negative. We also train discriminators D P and D N . They take a sequence of word embeddings as input and learn to distinguish whether the sequence is from word embeddings of a real sentence, or it was generated by G or F . With the continuous word embeddings as the translator output, the gradient can be back-propagated from the discriminator to the translator. Note that F and G transform sequences of word embeddings to sequences of word embeddings. We pretrain the word embedding model with Word2Vec [45]; here it is fixed during CycleGAN training. To transform the output sequence of word embeddings into a sentence, we simply select those words whose embeddings have the highest cosinesimilarity to each given word embedding in the sequence.

E. CycleGAN
The concept of W-GAN [46] is used to train D P and D N . The loss function of the discriminator D P is where y n is a negative sentence sampled from N , and G(y n ) is the output of translator G taking y n as the input. D P learns to minimize Eq. (6), that is, to give low scores to the translated output (the first term on the right) and high scores to real positive sentences y p (the second term). The loss function of the discriminators D N is parallel to Eq. (6): As in improved W-GAN, gradient penalty is applied here. The loss functions for training translators G and F are − D P (G(y n )).
(9) The first terms on the right-hand side of Eqs. (8) and (9) are the same. Given a positive sentence y p , after being transformed into a negative sentence by F and then transformed back to positive by G, it should be very close to the original sentence y p ; likewise for the second terms. The last terms of Eqs. (8) and (9) are different: F learns to generate output F (y p ) that is considered by D N to be a real negative sentence, whereas G learns to generate output G(y n ) that is considered by D P to be a real positive sentence. In this way translators F and G learn to transform the sentences from one sentiment (positive or negative) to the other. Notice that the discriminators D P and D N are jointly trained with the translators F and G. During testing, for any chatbot output y, we simply use G to transform it into a positive sentence G(y).

IV. EXPERIMENTAL SETUP
We trained and tested all our models, including the seq2seq and the four proposed models, on the following three corpora. The first two are in Chinese, whereas the third is in English. Using the training set, we trained five models, including the seq2seq baseline and the four proposed models; we evaluated these models using the testing set. All the evaluation metrics obtained are the average over the testing data. More dataset details are provided in Appendix A.
1) We used the Chinese Emotional Conversation Generation (CECG) task [47], originally offered by the NII Testbeds and Community for Information Access Research (NT-CIR) Project for the Short Text Conversation Task (STC) competition. CECG contains around 1.7M dialogue pairs; each sentence is labeled by one of the following six kinds of sentiments: like, sad, disgusted, angry, happy, and other. We reclassified five of these six sentiment categories into positive and negative categories: like and happy as positive sentiments, and sad, disgusted, and angry as negative sentiments. Both corpora were split into training and testing sets (the latter included 1k dialogue pairs). This corpus was used in both the sentiment classifier and the other models. 2) We collected data from the PTT Boy-Girl board containing the titles and all the article replies from page 1 to page 4000, and we used the articles as context and the replies as responses. Replies include "like" and "boo", which roughly correspond to positive and negative sentiment. However, as this dataset contains no true dialogue data, and the sentiment is not always precise, we use this dataset only for demonstration and not the main experiment. As with the previous dataset, this corpus was also used in both the sentiment classifier and other models.

3) For English, the Twitter chatting corpus is available on
Marsan-Ma's GitHub repository [48] using TensorFlow. This corpus, which contains 3.7M dialogue pairs, is split into training and testing sets, the latter of which includes 28k dialogue pairs. The sentiment classifier used in this work was trained from the Twitter Sentiment Analysis Corpus [49], which consists of 15M data with labeled sentiment (0 or 1). This corpus was also split into a training and testing set. The trained sentiment classifier achieved an 87% accuracy on the validation set.

A. Evaluation Metrics
Evaluation is always difficult in language generation; this is even more so for chatbots. Here we propose two metrics: semantic coherence 1 and 2 (COH1, COH2) for chatbots, which are scores reflecting the degree to which the output sentence y is a proper response to the input sentence x. These are in fact the semantic coherence 1 and 2 mentioned in Section III-C (Reinforcement Learning) designed for the reward function. However, the seq2seq model and the RNN discriminator used to obtain these two scores are re-trained here and are thus different models.
The third metric is the sentiment classifier score (SCL) used to measure how positive the output sentence is. This is the sentiment classifier score SC(y) used in the persona-based model mentioned in Section III-B. Likewise, the sentiment classifier used here is re-trained and is thus different.
The fourth metric is the language model score (LM) which measures whether the output sentence y is a good sentence in terms of a language model [50]. The language model used here is composed of a two-layer GRU [51] model: which is the language model probability P (y) for a sentence y, normalized with the sentence length N y . Eq. (10) is also known as the negative log perplexity (PPL). Note that SCL and LM, the third and fourth metrics, consider only the output sentence y -not the input x. COH1 and COH2, the first and second metrics, however, consider the output y given the input x. In all the following tables, larger evaluation metrics represent better performances.

B. Metric Models
The coherence score (1 and 2) in Chinese uses the PTT Gossiping board dialogue corpus collected by Justin Yang [52], which contains about 400,000 dialogue pairs. Here we split the data into training and testing data sets; the latter contains 1,000 pairs, whereas the other owns the rest.
For the LM score, in Chinese, we crawled the replies of articles from the top 50 most popular PTT boards, for a total of 25 million tokens of data. In English, the corpus of the One Billion Word Benchmark [53] was used.
The coherence score for English and sentiment score for both Chinese and English share the same data with the experiments mentioned in Section IV. Although the metrics are trained with the same corpus, as mentioned above, the models of the metrics are different because of re-training, which guarantees a certain extent of fairness.

VI. INDIVIDUAL MODELS
Due to the space limitation, we only show the results of the individual models on Chinese CECG dataset. The details of the hyper-parameter setup in the following experiments are shown in Appendix B.

A. Sentiment Classifier
To find a proper sentiment classifier, we evaluated six different ways of segmenting Chinese words. Including word-based and character-based methods, we also evaluated different neural network architectures: a CNN with max pooling (CNN for short), a GRU with the last hidden state output (GRU-last), and a GRU averaging all hidden state outputs for the whole sequence (GRU-avg). The total number of characters used in character-based segmentation was 7,297, and the number of words in word-based segmentation was 50,000. We trained these six models with a batch size of 32 and 50,000 epochs. We evaluated the performance using the accuracy and area under the Receiver Operating Characteristics curve (AUC). For the accuracy score, the output scores of the models greater than 0.5 were taken as predictions of positive sentiment; otherwise, they were negative. The predictions were then compared to the real answers to calculate the accuracy rate. The best results of all architectures shown in Table I, GRU-last with word segmentation yielded the best performance on both accuracy and AUC scores. GRU-last is then applied in the following experiments as the sentiment classifier.

B. Persona-Based Model
With a fixed sentiment score input of 1.0, we first compared the performance between the 64-, 128-, and 256-neuron seq2seq models of the persona-based model during the inference step. As shown in Table II  Further, we applied other sentiment scores as input to generate sentences with different degrees of sentiment. Table III shows the results of 0.0, 0.5, and 1.0 as inputs respectively on the 256-neuron model during inference. As expected, the lower we set the input, the lower the resultant sentiment scores: this suggests the model is already transforming sentences to different sentiments.

C. Reinforcement Learning
In this experiment, different coefficient reward combinations were adopted to test the performance of the model. Below we report the results after 2500 training epochs.
In the first three sets of experiments, we fixed R 2 as 0.0 and adjusted the proportion of R 1 and R 3 to determine how the four metrics were affected. The results in Table V show that the COH1 score increases as R 1 rises from 0.0 to 0.8, and the SCL score falls as R 3 decreases from 1.0 to 0.2, which is not surprising as rewards R 1 , R 2 , and R 3 in Eqs. (1), (2), and (3) were in parallel with the COH1, COH2, and SCL score respectively. Interestingly, COH2 degrades compared to the pretrained MLE baseline, showing that the RL model is highly

D. Plug and Play
VAE is used for sentence generation in the plug and play model, and KL cost annealing and vocabulary truncation are applied to improve VAE performance. In this experiment, we first compared the performance between models with and without KL cost annealing. These two models truncate vocabulary randomly at a probability of 0.3. In Table VI    In Table VIII, we see that the 0.3 output successfully reconstructs the original input. The 0.0 output also seems reasonable, although it fails to reconstruct the original sentence: it replaces "禮貌" ("courtesy") with "動作" ("action"), probably because it depends more on the decoder language model. With 0.7 truncation, the model outputs an incorrect sentence, replacing "提 高" ("improve") with "以" ("by"); this could be because the many unknown words hinder the model from learning correct sentence grammar. The model with KL cost annealing and with a vocabulary truncation of 0.3 outperforms all others, so we will use this setting in the following experiments.  Tables IX apply sentiment gradient ascent and descent respectively. Both positive and negative sentiment transfers generate corresponding sentiment outputs. However, most sentiment transfers replace words with strong sentiment bias. For example, in Table IX, "哈哈" ("haha") is added after a positive transfer (sentiment gradient ascent), and "淘汰" ("eliminate") is added after a negative transfer (sentiment gradient descent); although each of these words is a strong indicator of sentiment style, it leads to incorrect grammar.

E. CycleGAN
When training the CycleGAN model, heterogeneous input styles are preferred; thus the data used here is re-marked and selected by the sentiment classifier, leaving only those sentences with strong style bias. In the CycleGAN model training phase, training alternates between the generator and discriminator. Once one of the agents (generator or discriminator) is trained, the other's parameters are fixed and used for inference. If one is better than the other, a training imbalance occurs, and the process is no longer "adversarial". Since the discriminator is easier to train in most cases, the discriminator is stronger than the generator generally, leading to an awkward generator. We evaluate two kinds of generators: generators G and F, created after the generator and the discriminator are trained alternatively using different training epochs. Furthermore, we also tried to add Identity Loss (short for ID loss), which was used to help to keep the content after transferring, in total loss during training the generator. The result is illustrated in Tables X and XI: the training epoch ratios of the generator and discriminator are 1:1, 1:3, and 3:1 after 10000 iterations and 100000 iterations respectively.
The 1:1 result is most stable. For the 1:3 model, although it achieves good performance at 10000 iterations, the model's generator and discriminator both collapse at 100000 iterations; this may be because the generator is unable to deceive the robust discriminator after several iterations of training; repeated training in this case leads to collapse. Although the 3:1 model is relatively stable, its generator is not as good as that of the 1:1 model. Moreover, the experiment also shows if there is an improvement when the ID loss is added: the result shows no significant difference from when only generator loss is used as the total loss.
The 1:1 model with 100000 training iterations is the best model; there will be a comparison between this model and other trained models later. Tables XII contain examples of  sentences transformed by generators G and F. In the left table  of Table XII, as the un-transformed output is already a positive response, generator G has little effect on the result; generator F, however, produces an identifiable result with its clearly negative sentence. In the right table of Table XII, the original output is a negative response; generator G substitutes "討厭" ("hate") for "喜歡" ("like"), making a more positive sentence. Generator F makes little modification to the sentence.

A. Chinese CECG Dataset
Table XIII evaluates the four models using four metrics. The results better than the baselines are in blue, and the ranking of each method is also in the table. The personabased model (using sentiment score equal to 1.0 as input during inference) yields the highest SCL. The RL model yields the best performance on COH1 and LM. The plug and play and CycleGAN models transform the original output sentences directly, resulting in similar SCL scores, but worse than persona-based model and RL model. CycleGAN has better LM score than plug and play, which shows that CycleGAN does better with respect to grammar.

B. Chinese PTT Dataset
In this section, we switched to the PTT dataset to demonstrate the performance of the above four models. To train a sentiment classifier, we also tried CNN, GRU-last, and GRUavg on different word segmentation methods at first. However, as the dirty PTT dataset resulted in poor sentiment classifier performance, we improved the sentiment classifier using the following three steps. First, we compared the performance between CNN, GRU-last, and GRU-avg, and chose GRU-last because of its out-performance (see Table XIV). Second, we picked the more credible data (higher absolute scores) by GRU-last, shrinking the data from 1,402,303 to 1,149,000. Finally, we trained a new sentiment classifier on the remaining data using GRU-last-2. This final sentiment classifier was then used in the four models. GRU-last-2 here indicates the same architecture as GRU-last but not the same model.
The evaluation results on the PTT dataset are shown in Table XV. SCL scores are all high in these four models. It shows that all the four approaches can successfully generate positive responses, but persona-based and RL models are better than plug and play and Cycle GAN. Although the SCL score of persona-based model is the highest, it obtained the worst COH1 and COH2, which shows that its generated responses are not coherent with respect to the inputs. Similar to that on the CECG dataset, the RL model yields the best results in terms of COH1 and LM.

C. English Corpus
The results are listed in Table XVI. For the persona-based model, its SCL score is the highest; however, its COH1, COH2, and LM are the lowest. The RL model performed better than all other models in three out of the four metrics: COH1, COH2, and LM, but not the SCL score. The SCL scores of plug and play model and CylcleGAN are worse than personabased and RL models.

D. Examples
Tables XVII list outputs of the four models on Chinese CECG dataset. The persona-based and RL models usually output reasonable and positive sentences. From the data we also observed both the persona-based and RL models made complicated changes to output sentences; this was rarely seen in other models. The plug and play model tends to generate collapsed sentences. The CycleGAN model typically adjusts sentences by replacing negative words with positive ones; this is not always a significant transformation.

VIII. HUMAN EVALUATION
We performed a subjective human evaluation against sentences generated by five models each on three corpora (CECG, PTT, and English), for a total of 15 results with 30 subjects, all of whom were graduate students. They were asked to answer three questions about the output sentences: (1) Coherence: Is the output sentence a good response to the input? (2) Sentiment: Is the output sentence positive? (3) Grammar: Is the output sentence grammatically correct? They were asked to give scores ranging from 0 to 5, based on a few reference examples with given scores 1, 3, 5 to scale the scores. The average results (normalized to the range from 0 to 1) of three models are listed in Tables XVIII, XIX, and XX. The results better than the baselines are in blue, and the ranking of each method is also in the tables.
For coherence, all the approaches are worse than the baselines, except RL models on Chinese PTT dataset. Except for the second rank on the English corpus, the RL models worked the best among the four approaches in terms of coherence.
For sentiment, all the models can successfully modify the sentiment of responses, except plug and play models, which performed the worst on every corpus. For grammar, the RL models yielded the best performance on each corpus.

IX. DISCUSSION
The persona-based model and RL model can both generate the responses largely different from the original seq2seq model. For the persona-based model, its sentiment score was the highest on all the datasets in terms of both machine and human evaluation (the only exception is the human evaluation on Chinese CECG). However, its coherence and grammar were both worse than the RL model in terms of both machine and human evaluation. This shows that although personabased model can successfully generate very positive responses, the coherence and the grammar of the responses are poor. It tries to output sentences that carry the correct sentiment, but not necessarily relevant given the input. The RL model is generally the best model of the three corpora from all aspects. This is because the reward R 1 and R 2 in Eqs. (1) and (2) were in parallel with coherence, and R 1 in Eq. (1) also takes into account word ordering which leads to correct grammar. Its sentiment score was also high (although not as high as the persona-based model) because its reward R 3 is also in parallel with the sentiment, which yielded positive output. One may argue that the RL model overfits the machine-evaluation metrics since it learns to optimize those metrics. This issue is addressed in Section VIII through human evaluation. Humans also consider that the RL model has reasonable responses with correct grammar.
The plug and play model and CycleGAN only transform the output responses of an off-the-shelf seq2seq model. The plug and play model attempted to modify the latent code of the sentences. As the sentiment classifier primarily considered sentiment without really encoding sentence grammar, when maximizing the sentiment classifier's output, it sometimes transforms the original output of the seq2seq model into collapsed sentences. For CycleGAN, since the two translators directly outputted word embeddings carrying both sentiment and semantics, the models found mappings between words like "bad" to "good", "sorry" to "thank", "can't" to "can". However, this entailed only changes or deletions of specific words and not complex modifications of whole sentences. Since it only changes a few words of the original responses, its responses are not far from the original ones. This explains why the grammar of plug and play is worse than CycleGAN in terms of both machine and human evaluation in all cases. When it comes to sentiment and coherence, humans always    consider CycleGAN is better than plug and play, whereas their performances on COH1, COH2, and SCL are comparable. Because it is difficult for humans to read the sentences with poor grammar, humans usually consider that the sentiments of the collapsed sentences are incorrect, and they are less coherent with the inputs.

X. CONCLUSION
In this paper, we attempted to adjust the sentiment of the chatbot response given the input. We investigated four different models for the tasks; All of the models are based on the conventional seq2seq model. The performances of the four models in terms of machine-evaluated metrics and human evaluation are reported. The persona-based model and RL models, which alter the original seq2seq model parameters, yield good results. The persona-based model is good at exporting sentences of high sentiment score that might be suitable for cases when a chatbot only needs to reply with simple sentences that carry strong sentiment. The RL

APPENDIX A DATASET STATISTICS
Below we present the details of the datasets used in different tasks. In the sentiment classifiers (Section A), there are three corpora: Chinese, PTT, and English. As described in Section VII-B, the corpus used in the PTT task was refined by a special filtering processing which reduced the size from 1,402K to 1,149K. In the experiments (Section B), each of the three corpora are shared by four tasks -the persona-based model, the reinforcement learning model, the plug and play model, and the CycleGAN model. In the metrics (Sections C and D), there are for metrics for the Chinese, PTT, and English tasks. The Chinese and PTT tasks share the same metrics trained by the corpora described in Section C. The WMT11 corpus used for the LM in Section D used the data preprocessing described in [53]. For word-based segmentation, low-frequency words were eliminated to reduce the word dimensions, which led to the vocabulary size shown in the table, which was smaller than the actual size.