We have witnessed tremendous progresses in natural language generation (NLG) with large-scale Transformer [1]} based language models, such as the GPT-2 [2]}.
A natural question to raise beyond language modeling is, how can we have more fine-grained control over these powerful models?
Specifically, we would desire a language model to generate text centers on some user-defined conditionsInterchangeable with attributes.
We call this kind of language model a conditioned NLG.
One could easily imagine the commercial benefits of employing a conditioned NLG in our every day products, such as search.
To obtain a conditioned NLG, a naiive approach would be reformulate the original language modeling objectives and train the whole model from scratch [3]}.
However, doing so requires tons of labeled text with conditions, which is not always available in most applications.
Recently, PPLM [4]} proposed to take advantage of the pretrained GPT-2 without any further retraining of the language model itself.
This is done by having another pretrained `guidance model', such as a Bag of Word (BOW) or an attribute classifier (AC), to guide the latent state of GPT-2 toward generating more relevant tokens.
Witnessing the upside of this approach, we further explored simple, flexible, and effective approaches to conditioned NLG, that encompasses these desired qualities:
