Incremental Learning with Differentiable Architecture and Forgetting Search
As progress is made on training machine learning models on incrementally
expanding classification tasks (i.e., incremental learning), a next step is to
translate this progress to industry expectations. One technique missing from
incremental learning is automatic architecture design via Neural Architecture
Search (NAS). In this paper, we show that leveraging NAS for incremental
learning results in strong performance gains for classification tasks.
Specifically, we contribute the following: first, we create a strong baseline
approach for incremental learning based on Differentiable Architecture Search
(DARTS) and state-of-the-art incremental learning strategies, outperforming
many existing strategies trained with similar-sized popular architectures;
second, we extend the idea of architecture search to regularize architecture
forgetting, boosting performance past our proposed baseline. We evaluate our
method on both RF signal and image classification tasks, and demonstrate we can
achieve up to a 10% performance increase over state-of-the-art methods. Most
importantly, our contribution enables learning from continuous distributions on
real-world application data for which the complexity of the data distribution
is unknown, or the modality less explored (such as RF signal classification).