Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activation
Continual learning - learning new tasks in sequence while maintaining
performance on old tasks - remains particularly challenging for artificial
neural networks. Surprisingly, the amount of forgetting does not increase with
the dissimilarity between the learned tasks, but appears to be worst in an
intermediate similarity regime.
  In this paper we theoretically analyse both a synthetic teacher-student
framework and a real data setup to provide an explanation of this phenomenon
that we name Maslow's hammer hypothesis. Our analysis reveals the presence of a
trade-off between node activation and node re-use that results in worst
forgetting in the intermediate regime. Using this understanding we reinterpret
popular algorithmic interventions for catastrophic interference in terms of
this trade-off, and identify the regimes in which they are most effective.