Transformer-based Program Synthesis for Low-Data Environments
Recent advancements in large pre-trained transformer models (GPT2/3, T5) have
found use in program synthesis to generate programs that satisfy a set of
input/output examples. However, these models perform poorly on long-horizon and
low-data tasks, and often don't seem to understand the semantics of the
languages they generate. We investigate an approach that tackles both of these
issues, by using attributed context-free-grammars of programming languages to
generate programs, and then analyzing generated programs so that they can be
annotated with compile and runtime attributes, such as types, so that
information about the program can be remembered during long-horizon generation.
We firstly find that synthesized datasets can be made efficiently and can
provide transformer models with enough data in order to perform well on some
synthesis tasks. We also find that giving models access to program attributes
is especially effective in low-data environments, and tends improve the quality
and reduce errors of transformer-generated programs.