Recent work by <cite>Lau et al. (2018)</cite> proposes a quatrain generation method that relies on specific domain knowledge about the dataset to train a classifier for learning the notion of rhyming: that a line ending word always rhymes with exactly one more ending word in the poem. They train the classifier along with a language model in a multi-task setup. In contrast, we find that generators trained using our structured adversary produce samples that satisfy rhyming constraints with much higher frequency.