The Effect of Task Ordering in Continual Learning
We investigate the effect of task ordering on continual learning performance.
We conduct an extensive series of empirical experiments on synthetic and
naturalistic datasets and show that reordering tasks significantly affects the
amount of catastrophic forgetting. Connecting to the field of curriculum
learning, we show that the effect of task ordering can be exploited to modify
continual learning performance, and present a simple approach for doing so. Our
method computes the distance between all pairs of tasks, where distance is
defined as the source task curvature of a gradient step toward the target task.
Using statistically rigorous methods and sound experimental design, we show
that task ordering is an important aspect of continual learning that can be
modified for improved performance.