AANG: Automating Auxiliary Learning
When faced with data-starved or highly complex end-tasks, it is commonplace
for machine learning practitioners to introduce auxiliary objectives as
supplementary learning signals. Whilst much work has been done to formulate
useful auxiliary objectives, their construction is still an art which proceeds
by slow and tedious hand-design. Intuitions about how and when these objectives
improve end-task performance have also had limited theoretical backing. In this
work, we present an approach for automatically generating a suite of auxiliary
objectives. We achieve this by deconstructing existing objectives within a
novel unified taxonomy, identifying connections between them, and generating
new ones based on the uncovered structure. Next, we theoretically formalize
widely-held intuitions about how auxiliary learning improves generalization of
the end-task. This leads us to a principled and efficient algorithm for
searching the space of generated objectives to find those most useful to a
specified end-task. With natural language processing (NLP) as our domain of
study, we empirically verify that our automated auxiliary learning pipeline
leads to strong improvements over competitive baselines across continued
training experiments on a pre-trained model on 5 NLP end-tasks.