The common practice for training commonsense models has gonefrom–human–to–corpus–to–machine:
humans author commonsense knowledge graphs in order to train commonsense models. In this work, we investigate an alternative,from–machine–to–corpus–to–machine: general language models
author
these commonsense knowledge graphs to train commonsense models. Our study leads to a new framework,Symbolic Knowledge Distillation.
As with prior art in Knowledge Distillation, our approach uses larger models to teach smaller models. A key difference is that we distill knowledge symbolically–as text–in addition to the resulting neural model. We distill only one aspect–the commonsense of a general language model teacher, allowing the student to be a different type of model, a commonsense model.
Altogether, we show that careful prompt engineering and a separately trained critic model allow us to selectively distill high-quality causal commonsense from GPT-3, a general language model. Empirical results demonstrate that, for the first time, ahuman-authoredcommonsense knowledge graph is surpassed by ourautomatically distilledvariant in all three criteria: quantity, quality, and diversity. In addition, it results in a neural commonsense model that surpasses the teacher model’s commonsense capabilities despite its 100x smaller size. We apply this to theAtomicresource, and will share our new symbolic knowledge graph and commonsense models111We will share this following the anonymity period. We have permission from OpenAI to release GPT-3 generations.