---
abstract: 'Neural controllable text generation is an important area gaining attention due to its plethora of applications. In this work, we provide a new schema of the pipeline of the generation process by classifying it into five modules. We present an overview of the various techniques used to modulate each of these five modules to provide with control of attributes in the generation process. We also provide an analysis on the advantages and disadvantages of these techniques and open paths to develop new architectures based on the combination of the modules described in this paper.'
author:
- |
    Shrimai Prabhumoye, Alan W Black, Ruslan Salakhutdinov\
    School of Computer Science\
    Carnegie Mellon University\
    Pittsburgh, PA, USA\
    `sprabhum, awb, rsalakhu@cs.cmu.edu`\
bibliography:
- 'acl2020.bib'
title: Exploring Controllable Text Generation Techniques
---

Introduction
============

Controllable text generation is the task of generating realistic sentences whose attributes can be controlled. The attributes to control can range from being stylistic such politeness, sentiment, formality, etc.; demographic attributes of the person writing the text such as gender, age, etc.; content such as information, keywords, entities, etc to be generated, ordering of information, events, like plot summaries etc. Controlling various attributes of text generation has manifold applications. For instance in dialogue response generation task, work has been done in controlling persona [@zhang2018personalizing; @li2016persona], controlling various aspects of the response such as politeness [@niu:2018], formality, authority etc, grounding the responses in external source of information [@zhou2018dataset; @dinan2018wizard; @ghazvininejad2018knowledge], and controlling topic sequence [@tang2019target; @prabhumoye2020i]. Another application is story generation where you can control the ending [@peng2018towards], the persona [@chandu2019my], the plot [@yao2019plan], and the topic sequence [@huang2019hierarchically]. Controllable text generation is also used to modulate the formality and politeness of emails [@madaan2020politeness]. Report generation can be controlled by pulling disparate source documents into a coherent unified whole, which can use a shared set of sources such as Wikipedia article generation [@liu2018generating; @prabhumoye-etal-2019-towards].

![image](bg-control-modules.png)

Although there is a large body of prior work in controllable text generation, there is no unifying theme. Each work addresses a specific task in a specific context. In this paper we outline a new schema which connects prior work and provides an insight into various aspects of controllable text generation. The schema contains five modules that cover the overall generation pipeline and provide an understanding of the effect of each component on the generation process. Prior work has focused on specific parts of the schema that we outline here and we provide insights into their similarities. We provide an overview of these modules and also present an exploration of the various techniques used to control and update each of these modules.

Most of the controllable text generation tasks can be framed as conditional language generation tasks. They have an input or a [*source*]{} sequence ${\mathbf{U}}$ and an output or a [*target*]{} sequence ${\mathbf{Y}}$ to be generated. In this case, we model the probability of the [*target*]{} sequence conditioned on the [*source*]{} sequence given by $P({\mathbf{Y}} | {\mathbf{U}}) = \prod^T_t P({\mathbf{y}}_t |{\mathbf{U}}, {\mathbf{y}}_{<t})$. The generation of the target tokens of the sequence ${\mathbf{Y}}$ unfolds as a time series where each token ${\mathbf{y}}_t$ is generated at a time step ${\mathbf{t}}$. At a given time step $t$, a generative model takes in the previous hidden state ${\mathbf{h}}_{t-1}$ and the input ${\mathbf{x}}_t$ at current time step. It performs a set of operations denoted by $\boldsymbol{G}$ to produce the output ${\mathbf{o}}_t$ which is used to predict token ${\mathbf{\hat{x}}}_t$. The ground truth token to be generated is denoted by ${\mathbf{y}}_t$. As shown in Figure \[fig:bg-overview\], we have identified the following five modules for controlling the generation process: (1) [**External Input**]{} module is responsible for the initialization ${\mathbf{h}}_0$, of the generation process. (2) [**Sequential Input**]{} module is the input ${\mathbf{x}}_t$ at each time step of the generation. (3) [**Generator Operations**]{} module performs consistent operations or calculations on all the input at each time step. (4) [**Output**]{} module is the output ${\mathbf{o}}_t$ which is further projected on to the vocabulary space to predict the token ${\mathbf{\hat{x}}}_t$ at each time step. (5) [**Training Objective**]{} module takes care of the loss functions used for training the generator.

This schema provides an insight into the contributions of the various modules for controllable text generation. The main advantage of this schema is that it can be used with any algorithmic paradigm like sequence-to-sequence, probabilistic models, adversarial methods, reinforcement learning, etc. The schema can also be used with non-autoregressive algorithms which may generate text using graphical structures like trees [@welleck2019non; @guo2019non]. In this paper, we focus on how this schema can be used to describe controllable text generation focusing particularly on the use of autoregressive models. This work paves way to designing new architectures based on our schema. This can be done by identifying promising techniques for each module and then combining them. Our schema can also be potentially used for applying these techniques on new tasks of similar nature. It also provides an easy access to appropriate comparison with existing techniques for those new architectures. The prior work on unifying text generation models has mostly focused on building efficient tool-kits and modular views of generation. For instance, [@reiter2000buildNLG] details seven sub-tasks which are conceptually distinct to describe the generation process. These sub-tasks can be modelled separately or in some cases they may interleave. In [@reiter2000buildNLG], these seven sub-tasks are primarily characterized as content or structure tasks. Note that this work is not specific to neural text generation. Our work focuses specifically on controlling attributes in neural text generation process. We don’t divide the generation pipeline into several sub-tasks but we divide the neural text generation process into modules all of which are required for generation. In [@hu2019texar], the focus is on building a toolkit for various text generation tasks based on the three properties of versatility, modularity and extensibility. This work enlists few model architectures and learning paradigms for various text generation tasks. In our work, we focus only on the generation process of controllable text generation tasks. We specifically detail the inputs, outputs and operations of the generation process. We do not provide any specific examples of architectures but provide an overview of the basic underlying modules which can be used with any learning paradigm. @xie2017neural provides a practical guide to the neural generation process describing it in terms of initialization, optimization, regularization and decoding strategies. Our work on the other hand does not delve into the implementation details of the generation pipeline but provides an overall schema for understanding of the various components involved.

In the remainder of the paper, we denote the representation of the control attribute by ${\mathbf{s}}$ and the representation of the input or [*source*]{} sentence returned by the encoder as ${\mathbf{h}}_e$. In what follows, we first describe the possible ways of controlling attributes by modulating the [*external input*]{} in [§\[sec:bg-dec-init\]]{}, the [*sequential input*]{} in [§\[sec:bg-dec-inp\]]{}, the [*generator operations*]{} in [§\[sec:bg-gen\]]{}, the [*output*]{} in [§\[sec:bg-out\]]{} and the [*training objective*]{} in [§\[sec:bg-train-obj\]]{}. At the end of each section, we provide an analysis of each of the techniques described and how they fit together.

External Input {#sec:bg-dec-init}
==============

In this section we discuss the different techniques which can be used to control the generation process by updating the initialization of the generator ${\mathbf{h}}_0$. In the standard generation process, ${\mathbf{h}}_0$ is equal to ${\mathbf{h}}_e$. This is marked as module (1) in Figure \[fig:bg-overview\].

Arithmetic or Linear Transform {#sec:bg-dec-init-elem}
------------------------------

One of the easiest ways to control the generation is to concatenate a control vector ${\mathbf{s}}$ to output of the encoder ${\mathbf{h}}_e$. The external input of the decoder ${\mathbf{h}}_0$ will be $[{\mathbf{h}}_e; {\mathbf{s}}]$, where $[a;b]$ denotes concatenation. Here, the control vector ${\mathbf{s}}$ would provide the generator with a strong signal to guide the generation process.

@fu:2017 use this technique to control the style representation for their generator. The encoder builds representation that is devoid of the style and only retains content. The control vector for style is then concatenated to the encoder representation to initialize the decoder. This technique is commonly used in [@ghazvininejad2018knowledge; @zhou2018dataset; @dinan2018wizard] to concatenate information from external sources to dialogue context to generate dialogue responses. @chandu2019my concatenate personality representation ${\mathcal{P}}$ derived from a separate corpus to generate visual stories. They also experiment with a simple arithmetic operation on ${\mathbf{h}}_e$ given by ${\mathbf{h}}_0 = {\mathbf{h}}_e - {\mathcal{S}} + {\mathcal{P}}$ to get the initialization of the generator (here ${\mathcal{S}}$ denotes the average representation of the story). They observed that while concatenation technique is better at preserving the meaning of the generated story, the arithmetic operation provides a better signal of the personality for the generation process.

@hoang2016incorporating uses both the concatenation technique as well as performs a linear transform of ${\mathbf{s}}$ to obtain ${\mathbf{h}}_0$ for language modelling task. The control vectors in this case represents meta data such as key-words, topics etc. In case of the linear transform ${\mathbf{h}}_0 = {\mathtt{tanh}}({\mathbf{W}}_1 {\mathbf{h}}_e + {\mathbf{W}}_2 {\mathbf{s}} + {\mathbf{b}})$. The paper also explores adding the control vector to the encoder representation (${\mathbf{h}}_0 = {\mathbf{h}}_e + {\mathbf{s}}$).

In case of addition, the resulting ${\mathbf{h}}_0$ would be averaged representation of the input representation ${\mathbf{h}}_e$ and ${\mathbf{s}}$. Information could be lost in this case as control is not explicit. In case of concatenation, if the size of the control vector ${\mathbf{s}}$ is too small compared to the context vector ${\mathbf{h}}_e$, then ${\mathbf{s}}$ is over-shadowed by ${\mathbf{h}}_e$ and the generator may not be able to pay attention to ${\mathbf{s}}$. Hence it is important to choose comparable dimensions for these two vectors. But this increases the size of model considerably and could be quite costly. Linear transform avoids these issues and performs better than the other two techniques for @hoang2016incorporating.

Stochastic Changes {#bg:sec-stochastic}
------------------

@kingma2013auto introduce variational auto-encoder, where you can stochastically draw a continuous latent variable ${\mathbf{z}}$ from a Gaussian distribution. The initialization of the generator ${\mathbf{h}}_0$ is based on this latent variable which is drawn. @bowman-etal-2016-generating use this concept for generating sentences from this continuous latent representation. This process of changing the encoder state ${\mathbf{h}}_e$ is can only be used with Kullback-Leibler (KL) Divergence training objective described in [§\[bg:sec-loss-kl\]]{}.

In [@wang-etal-2019-topic], VAE is used to guide the generation process with topics of a document. A gaussian mixture model is used to incorporate topics into latent variables. In [@Xu2020unsupervisedVAE], VAE is used to control for sentiment attribute in style transfer task by constraining the posterior mean to a learned probability simplex.

Such a design of controllable text generation works when the control attributes can be represented as latent variables for example style, topics, strategies etc. This design is difficult to work for content grounded text generation tasks where specific information, keywords or entities have to guide the generation process.

Decompose {#bg:sec-decompose}
---------

You can decompose the encoder representation ${\mathbf{h}}_e$ into multiple subspaces, each of which signifies a different attribute you would like to control. @liu2018learning split the encoder representation ${\mathbf{h}}_e$ into two components, one which represents the structure in the document and the other represents the semantic information. This formulation was used by [@balachandran2020strucsum] for controlling structure in abstractive summarization. This work performs the split with respect to the dimensions of ${\mathbf{h}}_e$. The method forces the first $n$ dimensions of ${\mathbf{h}}_e$ to capture meaning and the latter to capture structure. @balachandran2020strucsum also show quantitative and qualitative analysis on the types of structures of documents learnt by this technique.

@romanov-etal-2019-adversarial decompose the encoder representation ${\mathbf{h}}_e$ into a form vector ${\mathbf{f}}$ and a meaning vector ${\mathbf{m}}$. During the training phase, a [*discriminator*]{} enforces ${\mathbf{m}}$ to not carry any information about the form using an adversarial loss and a [*motivator*]{} is used for a motivational loss that encourages ${\mathbf{f}}$ to carry the information about the form. The generation process can then be guided to adhere to the desired target form. As opposed to splitting ${\mathbf{h}}_e$ with respect to dimensions, this work learns subspaces ${\mathbf{W_m}}$ and ${\mathbf{W_f}}$ given by ${\mathbf{m}} = {\mathtt{tanh}}({\mathbf{W}}_m{\mathbf{h}}_e + {\mathbf{b}}_m)$ and ${\mathbf{f}} = {\mathtt{tanh}}({\mathbf{W}}_f{\mathbf{h}}_e + {\mathbf{b}}_f)$ respectively. When ${\mathbf{h}}_e$ is projected on ${\mathbf{W}}_m$, we get the meaning vector ${\mathbf{m}}$ and similarly when it is projected on ${\mathbf{W}}_f$ we get the form vector ${\mathbf{f}}$. This work shows qualitatively how ${\mathbf{m}}$ and ${\mathbf{f}}$ are learnt in the subspaces using t-SNE plots. It also shows quantitatively the use of ${\mathbf{m}}$ and ${\mathbf{f}}$ in downstream paraphrase detection tasks. This is an excellent method in building interpretable representations for control attributes. Although, the effectiveness of this technique is not yet proven in the style transfer task or the abstractive summarization task. In both the above mentioned works, the models learns interpretable representations of control attributes but were not able to beat state of the art methods in their respective tasks. It is also worth noting that learning good decomposed vectors is especially hard when no supervision is provided on what the decomposed components are supposed to learn.

This techniques works well when the representation space of the input ${\mathbf{x}}$ can be decomposed into subspaces which can represent the control attributes. This means that the input ${\mathbf{x}}$ needs to contain signal of the control attributes. It is unlikely to work when the control attributes need to be externally provided. For example in case of content grounded generation tasks described in [@prabhumoye-etal-2019-towards; @dinan2018wizard; @zhou2018dataset], the input may not necessarily contain the content that needs to be generated. A separate input of the content to be generated is provided in these cases.

External Feedback {#bg:sec-inp-ext}
-----------------

A regularizer is often used to control the external input ${\mathbf{h}}_0$ to the generator. In many cases, an adversarial loss to manipulate the latent space is used as an external feedback mechanism. This essentially controls the latent space of the encoder which is eventually provided as an initialization to the generator. In [@FuTan], a multi-layer perceptron (MLP) is used for predicting the style labels from ${\mathbf{h}}_0$. Similarly, the adversarial loss is also used in [@wang2019controllable] to control the latent representation ${\mathbf{h}}_0$ for style attributes. In [@romanov-etal-2019-adversarial], an adversarial loss is used to ensure that the meaning representation ${\mathbf{m}}$ does not carry any style signals. The adversarial loss is obtained by training a discriminator which takes as input a representation ${\mathbf{m}}$ and tells if it carries the target style signal. Similarly, this work also employs a motivator loss which is the opposite of the adversarial loss to ensure that the style representation ${\mathbf{f}}$ actually does carry the stylistic information. @john-etal-2019-disentangled use multiple losses to control the style and content information represented in ${\mathbf{h}}_0$.

The discriminator which provides external feedback has to be jointly trained with the generator. This technique can be useful with the decompose technique to ensure that the decomposed sub-spaces represent the desired control attributes.

Sequential Input {#sec:bg-dec-inp}
================

In this section we discuss the different techniques which can be used to manipulate the sequential input ${\mathbf{x}}_t$ to the decoder at each time step. ${\mathbf{x}}_t$ here is used to denote the word embedding of the token at time step $t$. This is marked as position (2) in Figure \[fig:bg-overview\].

Arithmetic or Linear Transform {#arithmetic-or-linear-transform}
------------------------------

Similar to changing the initialization, we can change the input to the decoder by concatenating the information at each time step with some additional control vector ${\mathbf{s}}$. Typically, teacher forcing method [@williams1989learning] is used to train the generator. At time step $t$, the generator takes as input the word embedding ${\mathbf{x}}_t$ of the word that was predicted at step $t-1$ and predicts the word to be generated ${\mathbf{y}}_t$ at the current time step. Note that ${\mathbf{x}}_t = {\mathbf{y}}_{t-1}$. The input ${\mathbf{x}}_t$ can be concatenated with ${\mathbf{s}}$ at each time step to control the generation process. Hence, ${\mathbf{\tilde{x}}}_t = [{\mathbf{x}}_t; {\mathbf{s}}]$.

@noraset2017definition, use this technique in the task of definition modeling. They concatenate word embedding vector ${\mathbf{s}}$ of the word to be defined at each time step of the definition generation process. Unfortunately, for this task, this technique has not proved to be effective compared to other techniques of controlling the generation. @zhou2018dataset concatenate the hidden representation of the external source of information ${\mathbf{s}}$ to each time step of dialogue response generation. Similarly, @prabhumoye-etal-2019-towards also concatenate the hidden representation of the external source of information $s$ to each time step of Wikipedia update generation process. In this work as well, this results of this technique were not as impressive as simple concatenating the control context to the input of the encoder. @harrison2019maximizing concatenate a side constraint $s$ which represents style and personality into the generation process. For this task of generating language from meaning representations with stylistic variation, this method performed better than conditioning the encoder with side constraint in terms of BLEU metric. @chandu2019my also concatenate the personality representation ${\mathcal{P}}$ at each time step of the story generation process. This is used to control the personality of the visual stories. In addition to concatenation, this work proposes to modify the sequential input as ${\mathbf{\tilde{x}}}_t = {\mathbf{x}}_t - {\mathcal{S}} + {\mathcal{P}}$ (here ${\mathcal{S}}$ denotes the average representation of the story and ${\mathcal{P}}$ denotes the representation of the personality). The latter technique is better at generating personality conditioned stories than the concatenation technique. Neither of these techniques prove to be conclusively better than making similar changes to the external input module ([§\[sec:bg-dec-init-elem\]]{}). Note that in this technique, changes are made directly to the input of generation and not the context which is the case with external input. Also, most of the prior work has focused on recurrent neural network and its variants for making such changes. It could be interesting to see such changes made to transformers [@vaswani2017attention].

Generator Operations {#sec:bg-gen}
====================

This module takes in the external input ${\mathbf{h}}_0$, the sequential input ${\mathbf{x}}_t$ at time step $t$ and performs computation to return an output ${\mathbf{o}}_t$. The same set of computations ($\boldsymbol{G}$) are performed at each time step. Different set of operations can be performed to compute ${\mathbf{o}}_t$ which are enlisted below. You can also decide to change the operations based on the control vector ${\mathbf{s}}$ to compute ${\mathbf{o}}_t$. This is shown as position (3) in Figure \[fig:bg-overview\].

Recurrent Neural Networks {#sec:bg-gen-rnn}
-------------------------

Recurrent Neural Networks (RNNs) are designed to model sequential information. RNNs perform the same operations for every element of a sequence, with the output depending on previous computations. This recurrence serves as a form of memory. It allows contextual information to flow through the network so that relevant outputs from previous time steps can be applied to network operations at the current time step. Theoretically, RNNs can make use of information in arbitrarily long sequences, but empirically, they are limited to looking back only a few steps.

The Long Short-Term Memory (LSTM) [@hochreiter1997long] units are a type of RNNs that have additional ‘memory cell’ apart from standard units of basic RNNs. The memory cell can maintain information in memory for long periods of time. A set of gates is used to control when information enters the memory, when it’s output, and when it’s forgotten. This architecture lets them learn longer-term dependencies. The vanishing gradient problem of RNNs is resolved here. Gated Recurrent Units (GRUs) [@cho2014learning] are similar to LSTMs, but use a simplified structure designed to adaptively capture dependencies of different time scales. They also use a set of gates to control the flow of information, but they don’t use separate memory cells, and they use fewer gates.

The computations of the RNN or its variants can be modified to account for the control attribute. Additional gates can be added or the control attribute can be provided as an additional input to the standard gates of RNNS.

@gan2017stylenet propose a variant of the LSTM model, named factored LSTM, which controls style representation in image caption task. The parameters of the LSTM module which are responsible to transform the input ${\mathbf{x}}_t$ are factored into three components ${\mathbf{U}}$, ${\mathbf{S}}$ and ${\mathbf{V}}$. The operations of the input (${\mathbf{i}}_t$), forget (${\mathbf{f}}_t$) and output gate (${\mathbf{o}}_t$) are given by: $$\begin{aligned}
    {\mathbf{i}}_t &=& {\mathtt{sigmoid}}({\mathbf{U}}_{ix} {\mathbf{S}}_{ix} {\mathbf{V}}_{ix} {\mathbf{x}}_t + {\mathbf{W}}_{ih} {\mathbf{h}}_{t-1}) \\
    {\mathbf{f}}_t &=& {\mathtt{sigmoid}}({\mathbf{U}}_{fx} {\mathbf{S}}_{fx} {\mathbf{V}}_{fx} {\mathbf{x}}_t + {\mathbf{W}}_{fh} {\mathbf{h}}_{t-1}) \\
    {\mathbf{o}}_t &=& {\mathtt{sigmoid}}({\mathbf{U}}_{ox} {\mathbf{S}}_{ox} {\mathbf{V}}_{ox} {\mathbf{x}}_t + {\mathbf{W}}_{oh} {\mathbf{h}}_{t-1}) \\
    {\mathbf{\tilde{c}}}_t &=& {\mathtt{tanh}}({\mathbf{U}}_{cx} {\mathbf{S}}_{cx} {\mathbf{V}}_{cx} {\mathbf{x}}_t + {\mathbf{W}}_{ch} {\mathbf{h}}_{t-1})\end{aligned}$$

Particularly, the matrix set $\{{\mathbf{S}}\}$ is specific to each style in the task and is responsible to capture the underlying style features in the data.

In [@kiddon2016globally], the GRU unit is modified to accommodate extra inputs - goal ${\mathbf{g}}$ and agenda items $E^{new}_t$ in the recipe generation task. The operation of the new component ${\mathbf{\tilde{h}}}_t$ is given by: $$\begin{aligned}
    {\mathbf{\tilde{h}}}_t = {\mathtt{tanh}}({\mathbf{W}}_h {\mathbf{x}}_t + {\mathbf{r}}_t \odot {\mathbf{U}}_h {\mathbf{h}}_{t-1} + {\mathbf{s}}_t \odot {\mathbf{Y}}{\mathbf{g}} + \\
    {\mathbf{q}}_t \odot ({\mathbf{1}}^{T}_{L} {\mathbf{Z}} {\mathbf{E}}^{new}_{t})^{T})\end{aligned}$$ where ${\mathbf{s}}_t$ is a goal select gate and ${\mathbf{q}}_t$ is a item select gate. With this modification, the generation process is controlled for the items to be generation in the recipe and the goal.

@sem_cond_lstm adapt the LSTM to control the dialogue act information in the generation process. The operation to compute the cell value ${\mathbf{c}}_t$ is given by: $${\mathbf{c}}_t = {\mathbf{f}}_t \odot {\mathbf{c}}_{t-1} + {\mathbf{i}}_t \odot {\mathbf{\tilde{c}}}_t + {\mathtt{tanh}}({\mathbf{W}}_d {\mathbf{d}}_t)$$ The dialogue act representation ${\mathbf{d}}_t$ is build using another LSTM cell.

RNNs, LSTMs and GRUs are commonly used to model controllable text generation tasks [@prabhumoye-etal-2019-towards; @rao2018dear; @see2017get; @zhou2018dataset; @fu:2017]. Most of these variants still have trouble remembering long sequences and are hence commonly used with attention mechanism ([§\[sec:bg-out-att\]]{}) on the source sequence.

Transformer
-----------

Transformers are proposed by [@vaswani2017attention] and they rely on attention mechanism to draw global dependencies between input and output. The Transformer uses stacked self-attention and point-wise, fully connected layers for both the encoder and decoder. The encoder stacks $N$ identical layers, each of which has two sub-layers. The first sub-layer is a multi-head self-attention mechanism ([§\[sec:bg-out-att\]]{}), and the second sub-layer is a positionwise fully connected feed-forward network. Each sub-layer uses residual connections around each of the sub-layers, followed by layer normalization. The decoder has an additional third sub-layer, which performs multi-head attention over the output of the encoder stack.

Since, attention mechanism is at the core of this generator, the decoder can attend over all positions of input sequence. Computations over a sequence can be parallelized in this case and hence it is faster in performance. The modifications made to the computing units of RNN mentioned in [§\[sec:bg-gen-rnn\]]{} which use parameters specific to control attributes such as style, dialog act etc have not been explored with the transformers architecture.

Pre-trained models
------------------

Recently pre-trained conditional language models are used for text generation like GPT [@radford2018improving], GPT2 [@radford2019language], XLNet [@yang2019xlnet], etc. Several works have fine-tuned the pre-trained models for downstream controllable text generation tasks [@sudhakar2019transforming; @dinan2018wizard; @urbanek2019learning]. The language modeling aspects of generation like fluency and grammaticality are already learnt if pre-trained models are used.

These models are hard to fine-tune for sequence-to-sequence tasks such as machine translation, abstractive summarization etc. BART [@lewis2019bart] is a denoising autoencoder built with a sequence-to-sequence model and is particularly effective when fine tuned for text generation. Alternatively, T5 [@raffel2019exploring] treats every NLP problem as a “text-to-text" problem, i.e. taking text as input and producing new text as output. Hence, it can be adapted to controllable text generation tasks. @dathathri2019plug propose a Plug and Play Language Model (PPLM) for controllable language generation. It combines a pretrained LM with one or more simple attribute classifiers that guide text generation without any further training of the LM. This is similar to the classifier feedback technique described in [§\[bg:sec-loss-class\]]{}. Some of the other techniques described in this paper such as stochastic changes [§\[bg:sec-stochastic\]]{} , external feedback [§\[bg:sec-inp-ext\]]{} and [§\[bg:sec-out-ext\]]{}, decompose [§\[bg:sec-decompose\]]{} etc would be hard to incorporate into pre-trained language models without modifying the model architecture or fine-tuning entailing the significant cost of retraining.

Output {#sec:bg-out}
======

In the standard generation process, ${\mathbf{o}}_t$ is the output of the generator module which is projected to the vocabulary space to predict the token ${\mathbf{\hat{x}}}_t$. Here, we discuss the various techniques used to modulate the sequential output ${\mathbf{o}}_t$ at each time step $t$, before projecting it to the vocabulary space. This is marked as position (4) in Figure \[fig:bg-overview\].

Attention {#sec:bg-out-att}
---------

Attention is the most popular way of guiding the generation process. It is typically used to guide the generation process to focus on the source sequence [@bahdanau2015neural]. The attention calculating module takes as input the current hidden state ${\mathbf{h}}_t$ of the generator at each time step ${\mathbf{t}}$. The aim of this module is to determine a context vector ${\mathbf{c}}_t$ that captures relevant source-side information to help predict the token ${\mathbf{\hat{x}}}_t$. In case of [*global attention*]{}, all the hidden states of the encoder are considered to calculate the context vector ${\mathbf{c}}_t$ [@luong2015effective]. This faces the the downside of expensive calculation especially for longer source sequences like documents. To overcome this challenge, [*local attention*]{} only chooses to focus only on a small subset of the source positions per target word. In this case, ${\mathbf{c}}_t$ is calculated over a window of size $D$ of the source hidden states.

@vaswani2017attention view attention as a mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key. This work proposes the simultaneous use of [*scaled dot-product*]{} attention which helps in parallelizing computation and a [*multi-headed*]{} attention which allows the model to jointly attend to information from different representation subspaces at different positions.

@sudhakar2019transforming use self-attention to control for style by simply adding a special target style token in the source sequence. @dinan2018wizard also use transformers to attend over information from external document for guided dialogue response generation. [@zhang2018personalizing] uses the encoded representation of personas to compute the attention weights ${\mathbf{a}}_t$ at a given time step of the decoder. The attention is re-weighted according to the persona of the response to be generated in dialogue. So far, work has not been done to modulate the attention weights to control for attributes like style, topic, content etc.

External Feedback {#bg:sec-out-ext}
-----------------

The output latent space of the generator can be controlled by external feedback. Similar to changing the external input ${\mathbf{h}}_0$, the output latent space can also be changed using adversarial loss. In [@logeswaran2018content], an adversarial loss is used which encourages the generation realistic and attribute compatible sentences. The adversarial loss tries to match the distribution of sentence and attribute vector pairs $({\mathbf{x}}, {\mathbf{s}})$ where the sentence can either be a real or generated sentence. @gong-etal-2019-reinforcement also control the output latent space by providing different types of rewards like style reward, semantic reward and fluency reward in the reinforcement learning setup. The discriminator used to obtain the adversarial loss has to be jointly trained with the generator.

Arithmetic or Linear Transform {#arithmetic-or-linear-transform-1}
------------------------------

@hoang2016incorporating demonstrate three simple ways of changing the output ${\mathbf{o}}_t$ of an RNN to control for meta information like topic, keywords etc. They show that you can add the control vector ${\mathbf{s}}$ to ${\mathbf{o}}_t$. Hence the modified output ${\mathbf{\tilde{o}}}_t$ is ${\mathbf{\tilde{o}}}_t = {\mathbf{o}}_t + {\mathbf{s}}$. Similarly, you can create ${\mathbf{\tilde{o}}}_t$ by concatenating ${\mathbf{s}}$ to ${\mathbf{o}}_t$ (${\mathbf{\tilde{o}}}_t = [{\mathbf{o}}_t; {\mathbf{s}}]$). We can also build ${\mathbf{\tilde{o}}}_t$ using a perceptron layer dependent on ${\mathbf{s}}$ and ${\mathbf{o}}_t$. In this case, ${\mathbf{\tilde{o}}}_t$ is given by ${\mathbf{\tilde{o}}}_t = {\mathtt{tanh}}({\mathbf{W}}_o {\mathbf{o}}_t + {\mathbf{W}}_s {\mathbf{s}} + {\mathbf{b}}_o)$. In each of the three cases, the modified output ${\mathbf{\tilde{o}}}_t$ is then projected to the vocabulary space to predict the token ${\mathbf{\hat{x}}}_t$.

Training Objective {#sec:bg-train-obj}
==================

In this section we describe various methods used to control the generation using objective functions. The output ${\mathbf{o}}_t$ at each time step $t$ of the generation process is projected to the vocabulary space using a linear transform (${\mathbf{\hat{o}}}_t = {\mathbf{W}}_o {\mathbf{o}}_t + {\mathbf{b}}$). A token ${\mathbf{\hat{x}}}_t$ is predicted from the vocabulary by passing ${\mathbf{\hat{o}}}_t$ through a softmax function and taking the max value. The predicted token ${\mathbf{\hat{x}}}_t$ is compared with the reference token ${\mathbf{y}}_t$ using a loss function. This loss function can be tweaked to ensure that the generated text carries the desired control attributes.

General Loss Objective
----------------------

Here, we describe the loss objectives commonly used in natural language generation tasks. These loss objectives do not try to control for any attribute. Instead they try to ensure fluent, grammatical and diverse generations.

#### Cross Entropy Loss:

This is the basic loss used to compare the generated tokens with the reference tokens and is used in all text generation process. At each time step $t$, the generation has to predict a token from the vocabulary. Hence, it could be seen as a classification problem with number of classes being equal to vocabulary size. The categorical cross entropy loss is given by: $$- \Sigma^{M}_{c=1} {\mathbf{y}}_{t, c} {\mathtt{log}} (p_{t, c})$$ where $p_{t, c}$ is the probability of the token $c$ at time step $t$. Note that $p_t = {\mathtt{softmax}} ({\mathbf{\tilde{o}}}_t)$ is the probability distribution over the vocabulary.

#### Unlikelihood loss: 

This maintains a set of negative candidates which is based on repeating tokens or n-grams and frequent tokens [@Welleck2020Neural]. This set is updated at each time step as tokens are generated. This works at both token and sequence level and the objective tries to minimize the repetitions in generations. This is used at train time in augmentation with the maximum likelihood objective and can be used for any task.

#### Diversity-Promoting objective:

This is used to generate a varied set of sentences given similar inputs. Particularly, @li:2015 use Maximum Mutual Information (MMI) as an objective function for the dialogue response generation task. Most generation systems use maximum likelihood objective but this objective additionally tries to reduce the proportion of generic responses. It is given by: $${\mathbf{\hat{T}}} = {\mathtt{argmax}}_{T} \{ {\mathtt{log}} p({\mathbf{T}} | {\mathbf{S}}) - \lambda {\mathtt{log}} p({\mathbf{T}})\}$$ where ${\mathbf{\hat{T}}}$ is the generated target sequence, ${\mathbf{T}}$ is the reference target sequence and ${\mathbf{S}}$ is the source sequence. The second term controls the generation of the high frequency or the generic target sequences. Note that this objective is only used during the inference and the generators are trained using cross entropy loss. @zhang2018personalizing, also use a diversity encouraging objective for dialogue response generation. They train a discriminator to calculate similarity between the source ${\mathbf{S}}$ and target ${\mathbf{T}}$ ($D_{\psi}({\mathbf{T}}, {\mathbf{S}})$) , as well as between the source ${\mathbf{S}}$ and the generated target ${\mathbf{\hat{T}}}$ ($D_{\psi}({\mathbf{\hat{T}}}, {\mathbf{S}})$). They finally try to minimize the difference between $D_{\psi}({\mathbf{T}}, {\mathbf{S}})$ and $D_{\psi}({\mathbf{\hat{T}}}, {\mathbf{S}})$.

Apart from these, many other objectives rely on post-hoc decoding strategies such as stochastic decoding which include Top $k$-sampling [@fan:2018], nucleus sampling [@Holtzman2020The], or beam search variants [@paulus2018a; @kulikov-etal-2019-importance; @vijayakumar2018diverse; @holtzman2018learning].

KL Divergence {#bg:sec-loss-kl}
-------------

The Kullback-Leibler (KL) Divergence score, quantifies how much one probability distribution differs from another probability distribution. The KL divergence between two distributions ${\mathbf{{\mathcal{Q}}}}$ and ${\mathbf{{\mathcal{P}}}}$ is often stated using the following notation: $${\mathtt{KL}} ({\mathbf{{\mathcal{P}}}} \parallel {\mathbf{{\mathcal{Q}}}})$$ where the operator “$\parallel$” indicates [*divergence*]{} or ${\mathbf{{\mathcal{P}}}}$’s divergence from ${\mathbf{{\mathcal{Q}}}}$. Note that KL Divergence is not symmetric i.e ${\mathtt{KL}} ({\mathbf{{\mathcal{P}}}} \parallel {\mathbf{{\mathcal{Q}}}}) \neq {\mathtt{KL}} ({\mathbf{{\mathcal{Q}}}} \parallel {\mathbf{{\mathcal{P}}}})$. KL divergence can be used to minimize the information loss while approximating a distribution. In text generation, the KL Divergence is combined with the evidence lower bound (ELBO) to approximately maximize the marginal likelihood of data $p({\mathbf{x}})$ which helps in better generations. This objective is used in variational autoencoders and its variants in combination with sampling techniques described in [§\[bg:sec-stochastic\]]{}. This objective fits in the controllable text generation paradigm because it allows you to approximate the posterior distribution of the control variables in the latent ${\mathbf{z}}$-space.

Classifier Loss {#bg:sec-loss-class}
---------------

This loss is specifically used to ensure that the generated tokens ${\mathbf{\hat{x}}}$ comply with the control attributes ${\mathbf{s}}$. Note the difference between this loss and the external feedback loss used for the [*external input*]{} module and the [*output*]{} module is that this loss operates at the token level and the external feedback loss works on the latent hidden representations.

In case of style transfer task, this loss is used to guide the generation process to output the target style tokens. Some works [@prabhumoye:2018; @sudhakar2019transforming; @hu2017toward] use this loss to discriminate between all the styles in their task (one verses all fashion). This type of design will suffer from low accuracy scores when the number of styles increases. To counter this problem, this loss can be setup to calculate if the generated sentence ${\mathbf{\hat{x}}}$ belongs to style ${\mathbf{s_1}}$ or not and similarly to calculate another separate loss term for each style [@chandu2019my]. This type of loss design encounters increasing number of loss terms depending on the number of styles. The third way to motivate this loss term is to discriminating between a sentence ${\mathbf{x}}$ from data which belongs to style ${\mathbf{s_1}}$ and a generated sentence ${\mathbf{\hat{x}}}$ which belongs to the same style ${\mathbf{s_1}}$ [@yang2018unsupervised]. Again, you would need as many loss terms as the number of styles in this case. All of these works use cross entropy loss function to measure their losses.

@hu2019makes use a classifier based loss in the visual storytelling task. The classifier is a pre-trained language model [@devlin2019bert] used to measure the coherence between generated sentences of the story. Particularly, the classifier takes as input two sentences at a time ${\mathbf{\hat{x}}}_1$ and ${\mathbf{\hat{x}}}_2$ and outputs a binary label which indicates if ${\mathbf{\hat{x}}}_2$ follows ${\mathbf{\hat{x}}}_1$. In this case, the control variable is coherence in stories which is used to guide the generator to produce consistent sentences.

Task Specific Loss
------------------

Depending on the end task and the attribute to be controlled, you can design different loss objectives to ensure that generations abide by the target attributes.

#### Strategy Loss:

@Zhou2020Augmenting use a dialogue strategy based objective to generate responses for negotiation tasks. This task has ground truth strategies that lead to better negotiations. This loss captures the probability of a particular strategy occurring for the next utterance given the dialogue history. It guides the generator to align the responses with particular strategies.

#### Coverage Loss:

Generating repeated words or phrases is a common problem for text generation systems, and this becomes especially pronounced for multi-sentence text generation task such as abstractive document summarization. @see2017get introduce a [*coverage loss*]{} which penalizes repeatedly attending to the same locations of the source document.

#### Structure loss:

@li-etal-2018-improving-neural introduce two new loss objectives [*structural compression*]{} and [*structural coverage*]{} based on sentence-level attention. These objectives are specially designed for the task of abstractive document summarization. [*structural compression*]{} is used to generate a sentence by compressing several specific source sentences and [*structural coverage*]{} is used to cover more salient information of the original document. These objectives leverage document structure in document summarization, and explore the effectiveness of capturing structural properties of document summarization by regularization of the generative model to generate more informative and concise summaries.

Conclusion and Future Work
==========================

In this paper we propose a new schema to organize the prior work in controllable text generation. The schema contains five modules, each of which plays an important role in the generation process. We detail the various techniques used to modulate each of the five modules to perform controllable text generation. We also provide theoretical understanding and qualitative analysis of these techniques. This understanding paves way to new architectures based on combinations of these modules. The future work will focus on empirical comparison of these techniques to gain an insight into their usefulness and strength.

Acknowledgments {#acknowledgments .unnumbered}
===============

This work was supported in part by ONR Grant N000141812861, NSF IIS1763562, and Apple. We would also like to acknowledge NVIDIA’s GPU support.
