A Comprehensive Survey of Continual Learning: Theory, Method and Application

Abstract: To cope with real-world dynamics, an intelligent agent needs to incrementally acquire, update, accumulate, and exploit knowledge throughout its lifetime. This ability, known as continual learning, provides a foundation for AI systems to develop themselves adaptively. In a general sense, continual learning is explicitly limited by catastrophic forgetting, where learning a new task usually results in a dramatic performance degradation of the old tasks. Beyond this, increasingly numerous advances have emerged in recent years that largely extend the understanding and application of continual learning. The growing and widespread interest in this direction demonstrates its realistic significance as well as complexity. In this work, we present a comprehensive survey of continual learning, seeking to bridge the basic settings, theoretical foundations, representative methods, and practical applications. Based on existing theoretical and empirical results, we summarize the general objectives of continual learning as ensuring a proper stability-plasticity trade-off and an adequate intra/inter-task generalizability in the context of resource efficiency. Then we provide a state-of-the-art and elaborated taxonomy, extensively analyzing how representative strategies address continual learning, and how they are adapted to particular challenges in various applications. Through an in-depth discussion of promising directions, we believe that such a holistic perspective can greatly facilitate subsequent exploration in this field and beyond.

\section{Introduction}\label{sec:introduction}}
\else
\section{Introduction}
\label{sec:introduction}
\fi

\IEEEPARstart{T}{his} demo file is intended to serve as a ``starter file''
for IEEE Computer Society journal papers produced under \LaTeX\ using
IEEEtran.cls version 1.8b and later.
I wish you the best of success.

\hfill mds

\hfill August 26, 2015

\subsection{Subsection Heading Here}
Subsection text here.

\subsubsection{Subsubsection Heading Here}
Subsubsection text here.

\section{Conclusion}
The conclusion goes here.

\appendices
\section{Proof of the First Zonklar Equation}
Appendix one text goes here.

\section{}
Appendix two text goes here.

\ifCLASSOPTIONcompsoc

  \section*{Acknowledgments}
\else

  \section*{Acknowledgment}
\fi

The authors would like to thank...

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

\section{Introduction}\label{sec:introduction}}

Learning is the basis for intelligent systems to accommodate environments.
In response to external changes, evolution has empowered human and other organisms with strong adaptability to continually acquire, update, accumulate and exploit knowledge \cite{kudithipudi2022biological,parisi2019continual,hadsell2020embracing}.
Naturally, we expect artificial intelligence (AI) systems to adapt in a similar way.
This motivates the study of \textbf{continual learning}, where a typical setting is to learn a sequence of contents one by one and behave as if they were observed simultaneously (Fig.~\ref{fig:CL_Framework}, a).
Such contents can be new skills, new examples of old skills, different environments, different contexts, etc., with particular realistic challenges incorporated~\cite{parisi2019continual,van2019three}.
As the contents are provided incrementally over a lifetime, continual learning is also referred to as \textbf{incremental learning} or \textbf{lifelong learning} in much of the literature, without a strict distinction~\cite{chen2018lifelong,kudithipudi2022biological}.

Unlike conventional machine learning models built on the premise of capturing a static data distribution, continual learning is characterized by learning from dynamic data distributions.
A major challenge is known as \textbf{catastrophic forgetting} \cite{mccloskey1989catastrophic,mcclelland1995there}, where adaptation to a new distribution generally results in a largely reduced ability to capture the old ones.
This dilemma is a facet of the trade-off between \textbf{learning plasticity} and \textbf{memory stability}: an excess of the former interferes with the latter, and vice versa.
Beyond simply balancing the ``proportions'' of these two aspects, a desirable solution for continual learning should obtain strong \textbf{generalizability} to accommodate distributional differences within and between tasks (Fig.~\ref{fig:CL_Framework}, b).
As a naive baseline, retraining all old training samples (if allowed) makes it easy to address the above challenges, but creates huge computational and storage overheads (as well as potential privacy issues). In fact, continual learning is primarily intended to ensure the \textbf{resource efficiency} of model updates, preferably close to learning only new training samples.

\begin{figure}[t]
	\centering
	\includegraphics[width=0.90\columnwidth]{CL_Framework.jpg}
    \vspace{-.2cm}
	\caption{A conceptual framework of continual learning. \textbf{a}, Continual learning requires adapting to incremental tasks with dynamic data distributions. \textbf{b}, A desirable solution should ensure a proper balance between memory stability (red arrow) and learning plasticity (green arrow), as well as an adequate generalizability to intra-task (blue arrow) and inter-task (orange arrow) distributional differences. \textbf{c}, Representative continual learning strategies have targeted various aspects of machine learning.}
	\label{fig:CL_Framework}
    \vspace{-.5cm}
\end{figure}

Numerous efforts have been devoted to addressing the above challenges, which can be conceptually separated into five groups (Fig.~\ref{fig:CL_Framework}, c):
adding regularization terms with reference to the old model (regularization-based approach); approximating and recovering the old data distributions (replay-based approach); explicitly manipulating the optimization programs (optimization-based approach); learning robust and well-generalized representations (representation-based approach); and constructing task-adaptive parameters with a properly-designed architecture (architecture-based approach).
This taxonomy extends the commonly-used ones with recent advances, and provides refined sub-directions for each category.
We summarize how these methods achieve the proposed general objectives, with an extensive analysis of their theoretical basis and typical implementations.
In particular, these methods are \emph{closely connected}, e.g., regularization and replay ultimately act to rectify the gradient directions in optimization, and \emph{highly synergistic}, e.g., the efficacy of replay can be facilitated by distilling knowledge from the old model.

\newcommand{\tabincell}[2]{\begin{tabular}{@{}#1@{}}#2\end{tabular}}
\begin{table*}[th]
    \renewcommand\arraystretch{1.95}
	\centering
    \vspace{-0.2cm}
	\caption{A formal comparison of typical continual learning scenarios. $\mathcal{D}_{t,b}$: the training samples of task $t$ and batch $b$.
	$[b]$: the size of batch $b$.
	$\mathcal{B}_t$: the space of incremental batches belonging to task $t$. $\mathcal{D}_{t}$: the training set of task $t$ (further specified as $\mathcal{D}_{t}^{pt}$ for pre-training). $\mathcal{T}$: the space of all incremental tasks (further specified as $\mathcal{T}^{pt}$ for pre-training).
	$\mathcal{X}_{t}$: the input data in $\mathcal{D}_{t}$. $p(\mathcal{X}_{t})$: the distribution of $\mathcal{X}_{t}$. $\mathcal{Y}_{t}$: the data label of $\mathcal{X}_{t}$. }

	\smallskip

	\begin{tabular}{c|c|c}
	 \hline
	  Scenario & Training & Testing \\
    \hline
     IIL \cite{lomonaco2017core50}&  $\{\{\mathcal{D}_{t,b}, t\}_{b \in \mathcal{B}_t}\}_{t = j}$ & $\{p(\mathcal{X}_{t})\}_{t = j}$; $t$ is not required  \\
          \hline
     DIL \cite{hsu2018re,van2019three}& $\{\mathcal{D}_{t}, t \}_{t \in \mathcal{T}}$; $p(\mathcal{X}_{i}) \neq p(\mathcal{X}_{j})$ and $\mathcal{Y}_{i} = \mathcal{Y}_{j}$ for $i \neq j$ & $\{p(\mathcal{X}_{t})\}_{t \in \mathcal{T}}$, $t$ is not required  \\
          \hline
     TIL \cite{hsu2018re,van2019three}& $\{\mathcal{D}_{t}, t \}_{t \in \mathcal{T}}$; $p(\mathcal{X}_{i}) \neq p(\mathcal{X}_{j})$ and $\mathcal{Y}_{i} \cap \mathcal{Y}_{j} = \emptyset $  for $i \neq j$ & $\{p(\mathcal{X}_{t})\}_{t \in \mathcal{T}}$; $t$ is available  \\
      \hline
     CIL \cite{hsu2018re,van2019three}& $\{\mathcal{D}_{t}, t \}_{t \in \mathcal{T}}$; $p(\mathcal{X}_{i}) \neq p(\mathcal{X}_{j})$ and $\mathcal{Y}_{i} \cap \mathcal{Y}_{j} = \emptyset $  for $i \neq j$ & $\{p(\mathcal{X}_{t})\}_{t \in \mathcal{T}}$; $t$ is unavailable  \\
      \hline
     TFCL \cite{aljundi2019task}& $\{\{\mathcal{D}_{t,b} \}_{b \in \mathcal{B}_t}\}_{t \in \mathcal{T}}$; $p(\mathcal{X}_{i}) \neq p(\mathcal{X}_{j})$ and $\mathcal{Y}_{i} \cap \mathcal{Y}_{j} = \emptyset $  for $i \neq j$ & $\{p(\mathcal{X}_{t})\}_{t \in \mathcal{T}}$; $t$ is optionally available

     \\
      \hline
     OCL \cite{aljundi2019gradient}& $\{\{\mathcal{D}_{t,b} \}_{b \in \mathcal{B}_t}\}_{t \in \mathcal{T}}$, $[b]=1$; $p(\mathcal{X}_{i}) \neq p(\mathcal{X}_{j})$ and $\mathcal{Y}_{i} \cap \mathcal{Y}_{j} = \emptyset $  for $i \neq j$ & $\{p(\mathcal{X}_{t})\}_{t \in \mathcal{T}}$; $t$ is optionally available  \\
     \hline
     BBCL \cite{buzzega2020dark,bang2021rainbow}& $\{\mathcal{D}_{t}, t\}_{t \in \mathcal{T}}$; $p(\mathcal{X}_{i}) \neq p(\mathcal{X}_{j})$, $\mathcal{Y}_{i} \neq \mathcal{Y}_{j}$ and $\mathcal{Y}_{i} \cap \mathcal{Y}_{j} \neq \emptyset $  for $i \neq j$ & $\{p(\mathcal{X}_{t})\}_{t \in \mathcal{T}}$; $t$ is unavailable  \\
      \hline
     CPT \cite{sun2020ernie}& $\{\mathcal{D}_{t}^{pt}, t \}_{t \in \mathcal{T}^{pt}}$, followed by a downstream task $j$, and  & $\{p(\mathcal{X}_{t})\}_{t=j}$; $t$ is not required\\

       \hline
	\end{tabular}

	\label{table:scenario}

\end{table*}

Realistic applications present particular challenges for continual learning, which can be categorized into \emph{scenario complexity} and \emph{task specificity}.
As for the former, for example, the task oracle (i.e., which task to perform) is probably missing in training and testing, and the training samples might be introduced in tiny batches or even one pass. Due to the expense and scarcity of data labeling, continual learning needs to be effective for few-shot, semi-supervised and even unsupervised scenarios.
As for the latter, although current advances mainly focus on visual classification, other visual domains such as object detection, semantic segmentation and image generation, as well as other related fields, such as reinforcement learning (RL), natural language processing (NLP) and ethic considerations, are receiving increasing attention with their own opportunities and challenges.

Considering the significant growth of interest in continual learning, we believe that such an \emph{up-to-date} and \emph{comprehensive} survey can provide a holistic perspective for subsequent work.
Although there are some early surveys of continual learning with relatively broad coverage \cite{parisi2019continual,chen2018lifelong}, important advances in recent years have not been incorporated.
In contrast, the latest surveys typically collate only a local aspect of continual learning, with respect to its biological underpinnings \cite{hadsell2020embracing,hayes2021replay,kudithipudi2022biological,jedlicka2022contributions}, specialized settings for visual classification \cite{masana2020class,de2021continual_review,qu2021recent,mai2022online}, as well as extensions in NLP \cite{biesialska2020continual_review,ke2022continual_review} or RL \cite{khetarpal2022towards}.
To the best of our knowledge, this is the first survey that systematically summarizes the latest advances in continual learning.
Building on such strengths, we provide an in-depth discussion of continual learning, in terms of current trends, cross-directional prospects such as diffusion models, large-scale pre-training, vision transformer, embodied AI, neural compression, etc., and interdisciplinary connections with neuroscience.

Our main contributions include: (1) We present an up-to-date and comprehensive survey of continual learning, in order to bridge the advances in theory, method, and application;
(2) We summarize the general objectives of continual learning based on existing theoretical and empirical results, with a state-of-the-art and elaborated taxonomy of representative strategies;
(3) We categorize the particular challenges of realistic applications into scenario complexity and task specificity, and extensively analyze how continual learning strategies are adapted to them; and
(4) We provide an in-depth discussion of current trends and promising directions to facilitate subsequent work in related fields.

The paper is organized as follows: In Sec.~\ref{Sec.2_Setup}, we introduce the setups of continual learning, including its basic formulations, typical scenarios and evaluation metrics.
In Sec.~\ref{Sec.3_Theory}, we summarize some theoretical efforts for continual learning in response to its general objectives.
In Sec.~\ref{Sec.4_Method}, we present a state-of-the-art and elaborated taxonomy of representative strategies, analyzing their motivations and typical implementations.
In Sec.~\ref{Sec.5_Scenario} and ~\ref{Sec.6_Task}, we describe how these strategies are adapted to realistic challenges of scenario complexity and task specificity.
In Sec.~\ref{Sec.7_Discussion}, we provide a discussion of current trends, cross-directional prospects and interdisciplinary connections with neuroscience.

\iffalse
\newcommand{\tabincell}[2]{\begin{tabular}{@{}#1@{}}#2\end{tabular}}
\begin{table*}[th]
    \renewcommand\arraystretch{1.7}
	\centering
    \vspace{-0.2cm}
	\caption{Typical scenarios for continual learning. $\mathcal{B}_t$ is the space of all incremental batches for task $t$, $[b]$ denotes the size of $b$. }
     \vspace{-0.1cm}
	\smallskip

	\begin{tabular}{c|c|c}
	 \hline
	  Scenario & Training & Testing \\
    \hline
     IIL &  $\{\{(x_{t,b}, y_{t,b}, t) \}_{b \in \mathcal{B}_t}\}_{t = T}$ & $\{p(\mathcal{X}_{t})\}_{t = T}$; $t$ is not required  \\
          \hline
     DIL & $\{(x_{t}, y_{t}, t) \}_{t \in \mathcal{T}}$; $p(\mathcal{X}_{i}) \neq p(\mathcal{X}_{j})$ and $\mathcal{Y}_{i} = \mathcal{Y}_{j}$ for $i \neq j$ & $\{p(\mathcal{X}_{t})\}_{t \in \mathcal{T}}$, $t$ is not required  \\
          \hline
     TIL & $\{(x_{t}, y_{t}, t) \}_{t \in \mathcal{T}}$; $p(\mathcal{X}_{i}) \neq p(\mathcal{X}_{j})$ and $\mathcal{Y}_{i} \cap \mathcal{Y}_{j} = \emptyset $  for $i \neq j$ & $\{p(\mathcal{X}_{t})\}_{t \in \mathcal{T}}$; $t$ is available  \\
      \hline
     CIL & $\{(x_{t}, y_{t}, t) \}_{t \in \mathcal{T}}$; $p(\mathcal{X}_{i}) \neq p(\mathcal{X}_{j})$ and $\mathcal{Y}_{i} \cap \mathcal{Y}_{j} = \emptyset $  for $i \neq j$ & $\{p(\mathcal{X}_{t})\}_{t \in \mathcal{T}}$; $t$ is unavailable  \\
      \hline
     TFCL & $\{\{(x_{t,b}, y_{t,b}) \}_{b \in \mathcal{B}_t}\}_{t \in \mathcal{T}}$; $p(\mathcal{X}_{i}) \neq p(\mathcal{X}_{j})$ and $\mathcal{Y}_{i} \cap \mathcal{Y}_{j} = \emptyset $  for $i \neq j$ & $\{p(\mathcal{X}_{t})\}_{t \in \mathcal{T}}$; $t$ is available or unavailable  \\
      \hline
     OCL & $\{\{(x_{t,b}, y_{t,b}) \}_{b \in \mathcal{B}_t}\}_{t \in \mathcal{T}}$, $[b]=1$; $p(\mathcal{X}_{i}) \neq p(\mathcal{X}_{j})$ and $\mathcal{Y}_{i} \cap \mathcal{Y}_{j} = \emptyset $  for $i \neq j$ & $\{p(\mathcal{X}_{t})\}_{t \in \mathcal{T}}$; $t$ is available or unavailable  \\
     \hline
     GCL & $\{(x_{t}, y_{t}, t)\}_{t \in \mathcal{T}}$; $p(\mathcal{X}_{i}) \neq p(\mathcal{X}_{j})$, $\mathcal{Y}_{i} \neq \mathcal{Y}_{j}$ and $\mathcal{Y}_{i} \cap \mathcal{Y}_{j} \neq \emptyset $  for $i \neq j$ & $\{p(\mathcal{X}_{t})\}_{t \in \mathcal{T}}$; $t$ is unavailable  \\
      \hline
     CPT & $\{(x_{t}^{pt}, y_{t}^{pt}, t) \}_{t \in \mathcal{T}^{pt}}$, followed by a downstream task $T$ & $\{p(\mathcal{X}_{t}^{ds})\}_{t=T}$; $t$ is not required\\

       \hline
	\end{tabular}

	\label{table:scenario}
	\vspace{-.2cm}
\end{table*}
\fi

\section{Setup}\label{Sec.2_Setup}
In this section, we first present a basic formulation of continual learning. Then we introduce typical scenarios and evaluation metrics.

\subsection{Basic Formulation}\label{Sec_2_1_Formulation}
Continual learning is characterized as learning from dynamic data distributions. In practice, training samples of different distributions arrive in sequence. A continual learning model parameterized by $\theta$ needs to learn corresponding task(s) with no or limited access to old training samples and perform well on their test sets.
Formally, an incoming batch of training samples belonging to a task $t$ can be represented as $\mathcal{D}_{t,b} = \{ \mathcal{X}_{t,b}, \mathcal{Y}_{t,b}\}$, where $\mathcal{X}_{t,b}$ is the input data, $\mathcal{Y}_{t,b}$ is the data label, $t \in \mathcal{T}=\{1,\cdots,T\}$ is the task identity
and $b \in \mathcal{B}_t$ is the batch index
($\mathcal{T}$ and $\mathcal{B}_t$ denote their space, respectively). Here we define a task $t$ by its training samples $\mathcal{D}_{t}$ following the distribution
$\mathbb{D}_{t} := p(\mathcal{X}_{t}, \mathcal{Y}_{t}) $ ($\mathcal{D}_{t}$
denotes the entire training set by omitting the batch index, likewise for $\mathcal{X}_{t}$ and $\mathcal{Y}_{t}$), and assume that there is no difference in distribution between training and test sets. Under realistic constraints, the data label $\mathcal{Y}_{t}$ and the task identity $t$ might not be always available. In continual learning, the training samples of each task can arrive in batches of various sizes (i.e., $\{\{\mathcal{D}_{t,b} \}_{b \in \mathcal{B}_t}\}_{t \in \mathcal{T}}$) or simultaneously (i.e., $\{\mathcal{D}_{t}\}_{t \in \mathcal{T}}$).

\subsection{Typical Scenario}\label{Sec.2.2_Scenario}
According to the division of incremental batches and the availability of task identities, we detail the typical continual learning scenarios as follows (please refer to Table~\ref{table:scenario} for a formal comparison):
\begin{enumerate}[\leftmargin=-2em]
\item[$\bullet$] \emph{Instance-Incremental Learning} (IIL):
All training samples belong to the same task and arrive in batches.

\item[$\bullet$] \emph{Domain-Incremental Learning} (DIL): Tasks have the same data label space but different input distributions. Task identities are not required.

\item[$\bullet$] \emph{Task-Incremental Learning} (TIL): Tasks have disjoint data label spaces. Task identities are provided in both training and testing.

\item[$\bullet$] \emph{Class-Incremental Learning} (CIL): Tasks have disjoint data label spaces. Task identities are only provided in training.

\item[$\bullet$] \emph{Task-Free Continual Learning} (TFCL): Tasks have disjoint data label spaces. Task identities are not provided in either training or testing.

\item[$\bullet$] \emph{Online Continual Learning} (OCL): Tasks have disjoint data label spaces. Training samples for different tasks arrive as an one-pass data stream.

\item[$\bullet$] \emph{Blurred Boundary Continual Learning} (BBCL): Task boundaries are blurred, characterized by distinct but overlapping data label spaces.

\item[$\bullet$] \emph{Continual Pre-training} (CPT): Pre-training data arrives in sequence. The goal is to improve the performance of learning a downstream task.

\end{enumerate}

If not specified, each task is generally assumed to have a sufficient number of labeled training samples, i.e., \emph{Supervised Continual Learning}.
According to the provided $\mathcal{X}_{t}$ and $\mathcal{Y}_{t}$ in each $\mathcal{D}_{t}$, continual learning can be further categorized into zero-shot \cite{ghosh2021dynamic,singh2021rectification}, few-shot \cite{tao2020few}, semi-supervised \cite{wang2021ordisco}, open-world (i.e., to identify unknown classes and then incorporate their labels) \cite{joseph2021towards,wang2022learngene} and un-/self-supervised \cite{rao2019continual,hu2021well} scenarios.
Besides, other practical challenges have been considered and incorporated, such as multiple labels \cite{kim2020imbalanced}, noisy labels \cite{kim2021continual,bang2022online}, hierarchical granularity \cite{abdelsalam2021iirc} and sub-populations \cite{liang2022balancing}, mixture of task similarity \cite{ke2020continual}, long-tailed distribution \cite{liu2022long}, domain alignment \cite{lin2022prototype}, domain shifting \cite{simon2022generalizing}, anytime inference \cite{koh2021online}, novel class discovery \cite{roy2022class,joseph2022novel}, multi-modality \cite{srinivasan2022climb}, etc.
Some recent work has focused on various combinations of these scenarios \cite{lomonaco2017core50,mi2020generalized,xie2022general,bang2022online,koh2021online,caccia2020osaka}, as a way to simulate the complexity of the real world.

\subsection{Evaluation Metric}\label{Sec.2.3_Metric}

In general, the performance of continual learning can be evaluated from three aspects: overall performance of the tasks learned so far, memory stability of old tasks, and learning plasticity of new tasks. Here we summarize the common evaluation metrics with classification as an example.

\textbf{Overall performance} is typically evaluated by \emph{average accuracy} (AA) \cite{chaudhry2018riemannian_rwalk,lopez2017gradient_gem} and \emph{average incremental accuracy} (AIA) \cite{hou2019learning_lucir,douillard2020podnet}. Let ${a}_{k,j} \in [0,1]$ denote the classification accuracy evaluated on the test set of the $j$-th task after incremental learning of the $k$-th task ($j \leq k$). The output space to compute ${a}_{k,j}$ consists of the classes in either $\mathcal{Y}_{j}$ or $\cup_{i=1}^{k} \mathcal{Y}_{i}$, corresponding to using multi-head evaluation (e.g., TIL) or single-head evaluation (e.g., CIL) \cite{chaudhry2018riemannian_rwalk}. The two metrics at the $k$-th task are then defined as
\begin{equation}
    {{\rm{AA}}_{k}} = \frac{1}{k} \sum_{j=1}^{k} {a}_{k,j},
\end{equation}
\begin{equation} {{\rm{AIA}}_{k}} = \frac{1}{k} \sum_{i=1}^{k} {\rm{AA}}_{i}, \end{equation}
where AA represents the overall performance at the current moment and AIA further reflects the historical variation.

\textbf{Memory stability} can be evaluated by \emph{forgetting measure} (FM) \cite{chaudhry2018riemannian_rwalk} and \emph{backward transfer} (BWT) \cite{lopez2017gradient_gem}. As for the former, the forgetting of a task is calculated by the difference between its maximum performance obtained in the past and its current performance:
\begin{equation}
f_{j,k} = \max_{i \in \{1,...,k-1\}} ({a}_{i,j} - {a}_{k,j}), \forall j<k.
\end{equation}
FM at the $k$-th task is the average forgetting of all old tasks:
\begin{equation}
    {{\rm{FM}}_{k}} = \frac{1}{k-1} \sum_{j=1}^{k-1} f_{j,k}.
\end{equation}
As for the latter, BWT evaluates the average influence of learning the $k$-th task on all old tasks:
\begin{equation}
    {\rm{BWT}}_k = \frac{1}{k-1} \sum_{j=1}^{k-1} ({a}_{k,j} - {a}_{j,j}),
\end{equation}
where forgetting is generally reflected as a negative BWT.

\textbf{Learning plasticity} can be evaluated by \emph{intransience measure} (IM) \cite{chaudhry2018riemannian_rwalk} and \emph{forward transfer} (FWT) \cite{lopez2017gradient_gem}. IM is defined as the inability of a model to learn new tasks, which is calculated by the difference of a task between its joint training performance and continual learning performance:
\begin{equation}
    {\rm{IM}}_k = a_{k}^{*} - a_{k,k},
\end{equation}
where $a_{k}^{*}$ is the classification accuracy of a randomly-initialized reference model jointly trained with $\cup_{j=1}^{k} \mathcal{D}_{j}$ for the $k$-th task. In comparison, FWT evaluates the average influence of all old tasks on the current $k$-th task:
\begin{equation}
    {\rm{FWT}}_k = \frac{1}{k-1} \sum_{j=2}^{k} (a_{j,j} - \tilde{a}_{j}),
\end{equation}
where $\tilde{a}_{j}$ is the classification accuracy of a randomly-initialized reference model trained with $\mathcal{D}_{j}$ for the $j$-th task.
Note that, $a_{k,j}$ can be adapted to other forms depending on the task type, such as average precision (AP) for object detection \cite{shmelkov2017incremental_ilod}, Intersection-over-Union (IoU) for semantic segmentation \cite{cermelli2020modeling}, Fréchet Inception Distance (FID) for image generation \cite{wu2018memory_mrgan}, normalized reward for reinforcement learning \cite{ahn2019uncertainty}, etc, and the above evaluation metrics should be adjusted accordingly.

Besides, there are many other useful metrics, such as linear probes for representation forgetting \cite{davari2022probing}, maximum eigenvalue of the Hessian matrix for flatness of loss landscape \cite{mirzadeh2020understanding}, area under the curve of accuracy for anytime inference \cite{koh2021online}, as well as the overheads of storage and computation for resource efficiency \cite{chaudhry2018efficient_agem}, etc. We refer readers to their original papers.

\section{Theoretical Foundation and Analysis}\label{Sec.3_Theory}

In this section, we summarize some theoretical efforts in continual learning, with respect to stability-plasticity trade-off and generalizability analysis, and relate them to the main motivations for representative strategies.

\subsection{Stability-Plasticity Trade-off}

With the basic formulation in Sec.~\ref{Sec_2_1_Formulation}, let's consider a general setup for continual learning, where a neural network with parameters $\theta \in \mathbb{R}^{|\theta|}$
needs to learn $k$ incremental tasks.
The training set and test set of each task are assumed to follow the same distribution $\mathbb{D}_{t}$, $t=1,...,k$, where the training set $\mathcal{D}_{t}=\{\mathcal{X}_{t}, \mathcal{Y}_{t}\} = \{(x_{t,n}, y_{t,n})\}_{n=1}^{N_t}$ includes $N_t$ data-label pairs.
The objective is to learn a probabilistic model $p(\mathcal{D}_{1:k}|\theta) = \prod_{t=1}^{k} p(\mathcal{D}_{t} | \theta)$
that can perform well on all tasks denoted as $\mathcal{D}_{1:k} := \{\mathcal{D}_{1},...,\mathcal{D}_{k}\}$.
The task-related performance for discriminative models can be expressed as $\log p(\mathcal{D}_{t}|\theta) = \sum_{n=1}^{N_t} \log p_{\theta}(y_{t,n}|x_{t,n})$. The central challenge of continual learning generally arises from the sequential nature of learning: when learning the $k$-th task from $\mathcal{D}_{t}$, the old training sets $\{\mathcal{D}_{1},...,\mathcal{D}_{k-1}\}$ are inaccessible. Therefore, it is critical but extremely challenging to capture the distributions of both old and new tasks in a balanced manner, i.e., ensuring a proper \textbf{stability-plasticity trade-off}, where excessive learning plasticity or memory stability can largely compromise each other (Fig.~\ref{fig:Theory}, a, b).

A straightforward idea is to approximate and recover the old data distributions by storing a few old training samples or training a generative model, known as the \emph{replay-based approach}.
According to the learning theory for supervised learning \cite{hastie2009elements}, the performance of an old task is improved with replaying more old training samples that approximate its distribution, but with potential privacy issues and a linear increase in \textbf{resource overhead}. The use of generative models is also limited by the huge resource overhead, as well as their own catastrophic forgetting and expressiveness, further detailed in Sec.~\ref{Sec.4.2_Replay}.

An alternative choice is to propagate the old data distributions in updating parameters through formulating continual learning in a Bayesian framework. Based on a prior $p(\theta)$ of the network parameters, the posterior after observing the $k$-th task can be computed with Bayes' rule:
\begin{equation}
    p(\theta | \mathcal{D}_{1:k}) \propto p(\theta) \prod_{t=1}^{k} p(\mathcal{D}_{t} | \theta)
    \propto p(\theta | \mathcal{D}_{1:k-1}) p(\mathcal{D}_{k} | \theta),
\label{eqn_cl_posterior}
\end{equation}
where the posterior $p(\theta | \mathcal{D}_{1:k-1})$ of the ($k$-$1$)-th task becomes the prior of the $k$-th task, and thus enables the new posterior $p(\theta | \mathcal{D}_{1:k})$ to be computed with only the current training set $\mathcal{D}_{k}$. However, as the posterior is generally intractable (except very special cases), a common option is to approximate it with $q_{k-1}(\theta) \approx p(\theta | \mathcal{D}_{1:k-1})$, likewise for $q_{k}(\theta) \approx p(\theta | \mathcal{D}_{1:k})$. In the following, we will introduce two widely-used approximation strategies:

The first is online \emph{Laplace approximation}, which approximates $p(\theta | \mathcal{D}_{1:k-1})$ as a multivariate Gaussian with local gradient information \cite{kirkpatrick2017overcoming,huszar2017quadratic,ritter2018online,kao2021natural,wang2021afec}.
Specifically, we can parameterize $q_{k-1}(\theta)$ with $\phi_{k-1}$ and construct an approximate Gaussian posterior $q_{k-1}(\theta) := q(\theta ; \phi_{k-1}) = \mathcal{N}(\theta; \mu_{k-1}, \Lambda_{k-1}^{-1})$ through performing a second-order Taylor expansion around the mode $\mu_{k-1} \in \mathcal{R}^{|\theta|}$ of $p(\theta | \mathcal{D}_{1:k-1})$, where $\Lambda_{k-1}$ denotes the precision matrix and $\phi_{k-1} = \{ \mu_{k-1}, \Lambda_{k-1}\}$, likewise for $q(\theta ; \phi_{k})$, $\mu_{k}$ and $\Lambda_{k}$.
According to Eqn.~\ref{eqn_cl_posterior}, the posterior mode for learning the current $k$-th task can be computed as
\begin{equation}
\begin{split}
    \mu_k &= {\arg\max}_{\theta} \log p(\theta | \mathcal{D}_{1:k}) \\
    & \approx {\arg\max}_{\theta} \log p(\mathcal{D}_{k} | \theta) + \log q(\theta ; \phi_{k-1}) \\
    & = {\arg\max}_{\theta} \log p(\mathcal{D}_{k} | \theta) - \frac{1}{2} (\theta - \mu_{k-1})^\top \Lambda_{k-1} (\theta - \mu_{k-1}),
\end{split}
\label{laplace_mode}
\end{equation}
which can be obtained from $\mu_{k-1}$ and $\Lambda_{k-1}$.
Similarly, $\Lambda_k$ can be updated recursively:
\begin{equation}
\begin{split}
    \Lambda_k &= - \nabla_{\theta}^{2} \log p(\theta | \mathcal{D}_{1:k})\big|_{\theta = \mu_k} \\
    &\approx - \nabla_{\theta}^{2} \log p(\mathcal{D}_{k}|\theta)\big|_{\theta = \mu_k} + \Lambda_{k-1},
\label{laplace_precision}
\end{split}
\end{equation}
where the first term on the right side is the Hessian of the negative log likelihood of $\mathcal{D}_{k}$ at $\mu_k$, denoted as $H(\mathcal{D}_{k}, \mu_k)$. In practice, $H(\mathcal{D}_{k}, \mu_k)$ is often computationally inefficient due to the great dimensionality of $\mathcal{R}^{|\theta|}$, and there is no guarantee that the approximated $\Lambda_k$ is positive semi-definite for the Gaussian assumption. To overcome these issues, the Hessian can be approximated by the Fisher information matrix (FIM):
\begin{equation}
F_k = \mathbb{E}[\nabla_{\theta} \log p(\mathcal{D}_{k}|\theta)\nabla_{\theta} \log p(\mathcal{D}_{k}|\theta)^{\top}]\big|_{\theta = \mu_k} \approx H(\mathcal{D}_{k}, \mu_k).
\label{eqn_fim}
\end{equation}
The computation of FIM can be further simplified with a diagonal approximation \cite{kirkpatrick2017overcoming,huszar2017quadratic} or a Kronecker-factored approximation \cite{ritter2018online,martens2015optimizing}. Formally, Eqn.~\ref{laplace_mode} is implemented by saving a copy of the old model $\mu_{k-1}$ to regularize parameter changes, which is known as the \emph{regularization-based approach} (Sec. 4.1). Here, we use EWC \cite{kirkpatrick2017overcoming} as an example and present its loss function:
\begin{equation}
\mathcal{L}_{{\rm{EWC}}}(\theta) = \ell_k(\theta) + \frac{\lambda}{2}   (\theta - \mu_{k-1})^\top \hat{F}_{1:k-1}  (\theta - \mu_{k-1}),
\label{eqn_EWC}
\end{equation}
where $\ell_k$ denotes the task-specific loss, the FIM $\hat{F}_{1:k-1} = \sum_{t=1}^{k-1} {\rm{diag}}({F}_{t})$ with a diagonal approximation ${\rm{diag}}(\cdot)$ of each ${F}_{t}$, and $\lambda$ is a hyperparameter to control the strength of regularization.

The second is online \emph{variational inference} (VI) \cite{nguyen2018variational,swaroop2019improving,adel2019continual,kurle2019continual,li2020variational,loo2020generalized,kapoor2021variational,rudner2022continual,derakhshani2021kernel}. There are many different ways to do this, and a representative one is to minimize the following KL-divergence over a family $\mathcal{Q}$ that satisfies $p(\theta|\mathcal{D}_{1:k}) \in \mathcal{Q}$ at
the current $k$-th task:
\begin{equation}
q_k(\theta) = {\arg \min}_{q \in \mathcal{Q}} {\rm{KL}}(q(\theta) \parallel \frac{1}{Z_{k}} q_{k-1}(\theta)p(\mathcal{D}_{k}|\theta)),
\end{equation}
where $Z_{k}$ is the normalizing constant of $q_{k-1}(\theta)p(\mathcal{D}_{k}|\theta)$. In practice, the above minimization can be achieved by employing an additional Monte Carlo approximation, with specifying $q_k(\theta) := q(\theta ; \phi_{k}) = \mathcal{N}(\theta; \mu_k, \Lambda_k^{-1})$ as a multivariate Gaussian. We use VCL \cite{nguyen2018variational} as an example, which minimizes the following objective (i.e., maximize its negative):
\begin{equation}
\mathcal{L}_{{\rm{VCL}}}(q_k(\theta)) = \mathbb{E}_{q_k(\theta)}(\ell_k(\theta)) + {\rm{KL}}(q_k(\theta) \parallel q_{k-1}(\theta)),
\label{eqn_VCL}
\end{equation}
where the KL-divergence can be computed in a closed-form and serves as an implicit regularization term.
In particular, although the loss functions of Eqn.~\ref{eqn_EWC} and Eqn.~\ref{eqn_VCL} take similar forms, the former is a local approximation at a set of deterministic parameters $\theta$, while the latter is computed by sampling from the variational distribution $q_{k}(\theta)$.
This is attributed to the fundamental difference between the two approximation strategies \cite{nguyen2018variational,tserannatural}.

In addition to the parameter space, the idea of sequential Bayesian inference is also applicable to the function space \cite{titsias2019functional,pan2020continual,rudner2022continual}, which generally enables more flexibility in updating parameters. Also, there are many other extensions of VI, such as improving posterior updates with variational auto-regressive Gaussian processes (VAR-GPs) \cite{kapoor2021variational}, constructing task-specific parameters \cite{adel2019continual,loo2020generalized,kumar2021bayesian,lee2019neural_cndpm}, and adapting to a non-stationary data stream \cite{kurle2019continual}.

\begin{figure}[t]
	\centering
	\includegraphics[width=0.98\columnwidth]{Theory.jpg}

	\caption{Analysis of critical factors for continual learning. \textbf{a}, \textbf{b}, Continual learning requires a proper balance between old and new tasks, where excessive learning plasticity or memory stability can affect the overall performance (adapted from \cite{wang2022coscl}).
    \textbf{c}, \textbf{d}, When the converged loss landscape is flatter or the learned task distributions are more similar, a properly balanced solution can better generalize to the task sequence (adapted from \cite{wang2022coscl}).
    \textbf{e}, The structure of parameter space determines the complexity and possibility of finding a desirable solution (adapted from \cite{knoblauch2020optimal}). The yellow area indicates the feasible parameter space shared by individual tasks, which tends to be narrow and irregular as more incremental tasks are introduced. }
	\label{fig:Theory}

\end{figure}

In essence, the constraint on continual learning for either replay or regularization is ultimately reflected in gradient directions.
As a result, some recent work directly manipulates the gradient-based optimization process, categorized as the \emph{optimization-based approach} (Sec. 4.3).
Specifically, when a few old training samples $\mathcal{M}_t$ for task $t$ are maintained in a memory buffer, gradient directions of the new training samples are encouraged to stay close to that of $\mathcal{M}_t$ \cite{lopez2017gradient_gem,chaudhry2018efficient_agem,tang2021layerwise}.
This is formulated as $\left \langle \nabla_{\theta} \mathcal{L}_k(\theta; \mathcal{D}_k), \nabla_{\theta} \mathcal{L}_k(\theta; \mathcal{M}_t) \right \rangle   \geq 0$ for $t \in \{1,...,k-1\}$,
so as to essentially enforce non-increase in the loss of old tasks, i.e.,
$\mathcal{L}_k(\theta; \mathcal{M}_t) \leq \mathcal{L}_k(\theta_{k-1}; \mathcal{M}_t) $, where $\theta_{k-1}$ is the network parameters at the end of learning the ($k$-1)-th task.

Alternatively, gradient projection can be performed without storing old training samples \cite{zeng2019continual,wang2021training,farajtabar2020orthogonal,saha2020gradient,kao2021natural,kong2022balancing,liu2021continual,lin2022towards,peng2022continual,chaudhry2020continual,guo2022adaptive}.
Here we take NCL \cite{kao2021natural} as an example, which can manipulate gradient directions with $\mu_{k-1}$ and $\Lambda_{k-1}$ in online Laplace approximation.
As in Eqn.~\ref{eqn_reg_grad}, NCL performs continual learning by minimizing the task-specific loss $\ell_k(\theta)$ within a region of radius $r$ centered around $\theta$ with a distance metric $d(\theta, \theta+\delta)=\sqrt{\delta^\top\Lambda_{k-1}\delta/2}$ that
takes into account the curvature of the prior via its precision matrix $\Lambda_{k-1}$:
\begin{equation}
\begin{split}
\delta^* &= \arg \min_{\delta}\ell_k(\theta+\delta)\\
&\approx \arg \min_{\delta} \ell_k(\theta)+\nabla_{\theta} \ell_k(\theta)^\top \delta \\
& \mathrm{s.t.}, d(\theta, \theta+\delta)=\sqrt{\delta^\top\Lambda_{k-1}\delta/2} \leq r.
\label{eqn_reg_grad}
\end{split}
\end{equation}
The solution to such an optimization problem in Eqn.~\ref{eqn_reg_grad} is given by $\delta^* \propto \Lambda_{k-1}^{-1}\nabla_{\theta} \ell_k(\theta)-(\theta-\mu_{k-1})$, which gives rise to the following update rule for a learning rate $\lambda$:
\begin{equation}
\begin{split}
\theta \leftarrow \theta+\lambda [ \Lambda_{k-1}^{-1}\nabla_{\theta} \ell_k(\theta)-(\theta-\mu_{k-1})],
\label{eqn_update_rule}
\end{split}
\end{equation}
in which the first term encourages parameter changes predominantly in directions that do not interfere with the old tasks via a preconditioner $\Lambda_{k-1}^{-1}$, while the second term enforces $\theta$ to stay close to the old task solution $\mu_{k-1}$.

Of note, the above analyses are mainly based on finding a shared solution for all incremental tasks, which is subject to severe inter-task interference \cite{wang2021afec,ramesh2021model,wang2022coscl}.
Alternatively, incremental tasks can be learned in a (partially) separated way, which is the dominant idea of the \emph{architecture-based approach}. This can be formulated as constructing a continual learning model with parameters $\theta = \cup_{t=1}^{k} \theta^{(t)}$, where $\theta^{(t)} = \{e^{(t)}, \psi\}$, $e^{(t)}$ is the task-specific/adaptive parameters, and $\psi$ is the task-sharing parameters.
The task-sharing parameters $\psi$ are omitted in some cases, where the task-specific parameters $e^{(i)}$ and $e^{(j)}$ ($i<j$) may overlap to enable parameter reuse and knowledge transfer. The overlapping part $e^{(i)} \cap e^{(j)}$ is frozen when learning the $j$-th task to avoid catastrophic forgetting \cite{rusu2016progressive,serra2018overcoming}. Then, as in Eqn.~\ref{eqn_task_inference}, each task can be performed as $p(\mathcal{D}_{t}|\theta^{(t)})$ instead of $p(\mathcal{D}_{t}|\theta)$ if given the task identity
$\mathbb{I}_{\mathcal{D}_{t}}$, in which the conflicts between tasks are explicitly controlled or even avoided if $\psi$ is omitted:
\begin{equation}
\begin{split}
p(\mathcal{D}_{t} | \theta) &= \sum_{i=1,\cdots,k} p(\mathcal{D}_{t} |\mathbb{I}_{\mathcal{D}_{t}} =  i, \theta) p(\mathbb{I}_{\mathcal{D}_{t}} =  i |\theta) \\
&= p(\mathcal{D}_{t} |\mathbb{I}_{\mathcal{D}_{t}} =  t, \theta) p(\mathbb{I}_{\mathcal{D}_{t}} =  t |\mathcal{D}_{t}, \theta) \\
 &= p(\mathcal{D}_{t} |\theta^{(t)}) p(\mathbb{I}_{\mathcal{D}_t }=  t |\mathcal{D}_{t}, \theta) \\
 &=p(\mathcal{D}_{t} |e^{(t)}, \psi) p(\mathbb{I}_{\mathcal{D}_t }=  t |\mathcal{D}_{t}, \theta).
\label{eqn_task_inference}
\end{split}
\end{equation}
However, there are two main challenges. The first is the scalability of model size due to incremental allocation of $\theta^{(t)}$, which depends on the sparsity of $e^{(t)}$, reusability of $e^{(i)} \cap e^{(j)}$ ($i<j$), and transferability of $\psi$ (detailed in Sec. 4.5).
The second is the accuracy of task-identity prediction, denoted as $p(\mathbb{I}_{\mathcal{D}_{t}} =  t |\mathcal{D}_{t}, \theta)$.
Except for the TIL setting that always provides the task identity $\mathbb{I}_{\mathcal{D}_{t}}$ \cite{rusu2016progressive,serra2018overcoming,fernando2017pathnet,ebrahimi2019uncertainty}, other scenarios generally require the model to determine which $\theta^{(t)}$ to use based on the input data, as in Eqn.~\ref{eqn_task_inference}. This is closely related to the out-of-distribution (OOD) detection, where the predictive uncertainty should be low for in-distribution data and high for OOD data \cite{henning2021posterior,kim2022theoretical,d2021uncertainty}.
More importantly, since the function of task-identity prediction as in Eqn.~\ref{eqn_task_inference_2} (equivalent to classifying tasks) needs to be continually updated, it also suffers from catastrophic forgetting.
To address this issue, the $i$-th task's distribution $p(\mathcal{D}_{t} |  i,\theta)$ can be recovered by incorporating replay \cite{henning2021posterior,kj2020meta,gong2022continual,lee2019neural_cndpm}:
\begin{equation} \label{eqn_task_inference_2}
p(\mathbb{I}_{\mathcal{D}_t }=  i |\mathcal{D}_{t},\theta) \propto
p(\mathcal{D}_{t} |  i,\theta)p(i),
\end{equation}
where the marginal task distribution $p(i) \propto N_i$ in general.

\begin{figure*}[ht]
	\centering	\includegraphics[width=1.98\columnwidth]{Methodv2.jpg}

	\caption{A state-of-the-art and elaborated taxonomy of representative continual learning methods. We have summarized 5 main categories (blue blocks), each of which is further divided into several sub-directions (red blocks).}
	\label{fig:Method}

\end{figure*}

\subsection{Generalizability Analysis}
Current theoretical analyses of continual learning have primarily been performed on training sets of incremental tasks, assuming that their test sets follow similar distributions and the candidate solutions have similar generalizability. However, since the objective for learning multiple tasks is typically highly non-convex, there are many local optimal solutions that perform similarly on each training set but have significantly different generalizability on test sets \cite{wang2022coscl,mirzadeh2020understanding}.
For continual learning, a desirable solution requires not only \emph{intra-task generalizability} from training sets to test sets, but also \emph{inter-task generalizability} to accommodate their distributional differences.
Here we provide a conceptual demonstration with a task-specific loss $\ell_t(\theta; \mathcal{D}_t)$ and its empirical optimal solution $\theta_t^{*} = {\arg \min}_{\theta} \ell_t(\theta; \mathcal{D}_t)$. When a task $i$ needs to accommodate another task $j$, the maximum sacrifice of its performance can be estimated by performing a second-order Taylor expansion of
$\ell_i(\theta; \mathcal{D}_i)$ around $\theta_i^*$:
\begin{equation}
\begin{split}
    \ell_i(\theta_j^*; \mathcal{D}_i)
    &\approx \ell_i(\theta_i^*; \mathcal{D}_i) + (\theta_j^* - \theta_i^*)^\top \nabla_{\theta} \ell_i(\theta; \mathcal{D}_i) \big|_{\theta = \theta_i^*} \\
    & + \frac{1}{2}(\theta_j^* - \theta_i^*)^\top \nabla_{\theta}^2 \ell_i(\theta; \mathcal{D}_i) \big|_{\theta = \theta_i^*} (\theta_j^* - \theta_i^*) \\
    &\approx \ell_i(\theta_i^*; \mathcal{D}_i) + \frac{1}{2} \Delta \theta^{\top} \nabla_{\theta}^2 \ell_i(\theta; \mathcal{D}_i) \big|_{\theta = \theta_i^*} \Delta \theta,
\end{split}
\label{eqn_compatibility_error}
\end{equation}
where $\Delta \theta := \theta_j^* - \theta_i^*$ and $\nabla_{\theta} \ell_i(\theta; \mathcal{D}_i) \big|_{\theta = \theta_i^*} \approx \textbf{0}$. Then, the performance degradation of task $i$ is upper-bounded by
\begin{equation}
    \ell_i(\theta_j^*; \mathcal{D}_i) - \ell_i(\theta_i^*; \mathcal{D}_i) \leq \frac{1}{2} \lambda_i^{max} \| \Delta \theta \|^2,
\label{eqn_compatibility_bound}
\end{equation}
where $\lambda_i^{max}$ is the maximum eigenvalue of the Hessian matrix $\nabla_{\theta}^2 \ell_i(\theta; \mathcal{D}_i) \big|_{\theta = \theta_i^*}$. Note that the order of task $i$ and $j$ can be arbitrary, that is, Eqn.~\ref{eqn_compatibility_bound} demonstrates both forward and backward effects.
Therefore, the robustness of an empirical optimal solution $\theta_i^*$ to parameter changes is closely related to $\lambda_i^{max}$, which has been commonly used to describe the \textbf{flatness of loss landscape} \cite{mirzadeh2020understanding,hochreiter1997flat,keskar2017large}.

Intuitively, convergence to a local minima with a flatter loss landscape will be less sensitive to modest parameter changes and thus benefit both old and new tasks (see Fig.~\ref{fig:Theory}, c). To find such a \emph{flat minimum},
there are three widely-used strategies in continual learning.
The first is derived from its definition, i.e., the flatness metric. Specifically, the minimization of $\ell_t(\theta; \mathcal{D}_t)$ can be replaced by a robust task-specific loss $\ell_t^{b}(\theta; \mathcal{D}_t) := \max_{\| \delta \| \leq b} \ell_t(\theta+\delta; \mathcal{D}_t)$, thus the obtained solution guarantees low error not only at a specific point but also in its neighborhood with a ``radius'' of $b$. However, due to the great dimensionality of $\theta$, the calculation of $\ell_t^{b}(\theta; \mathcal{D}_t)$ cannot cover all possible $\delta$ but only a few directions \cite{shi2021overcoming}, similar to the complexity issue of computing the Hessian matrix in Eqn.~\ref{eqn_compatibility_error}.
The alternatives include using an approximation of the Hessian \cite{mirzadeh2020understanding,deng2021flattening} or calculating $\delta$ only along the trajectory of forward and backward parameter changes \cite{huang2021altersgd,mirzadeh2020linear,cha2021swad,izmailov2018averaging,mehta2021empirical}.
The second is to operate the loss landscape by constructing an ensemble model under the restriction of mode connectivity, i.e., integrating multiple minima in parameter or function space along the low-error path, since connecting them ensures flatness on that path \cite{mirzadeh2020linear,wang2022coscl,izmailov2018averaging,garipov2018loss,wortsman2022model,cha2020cpr}.
These two strategies are closely related to the \emph{optimization-based approach}.
The third comes down to obtaining well-distributed representations, which tend to be more robust to distributional differences in function space, such as by using pre-training \cite{ramasesh2021effect,mehta2021empirical,hu2021well}, wider network architectures \cite{ramasesh2020anatomy,mirzadeh2022wide,ramasesh2021effect} and self-supervised learning \cite{hu2021well,pham2021dualnet,cha2021co2l,madaan2021representational}.
Observing the substantial attention to large-scale pre-training and self-supervised learning, we group this direction into the \emph{representation-based approach} in Sec. 4.4.

There are many other factors critical for continual learning performance. As shown in Eqn.~\ref{eqn_compatibility_bound}, the upper bound of performance degradation also depends on the difference of the empirical optimal solution $\theta_t^{*} = {\arg \min}_{\theta} \ell_t(\theta; \mathcal{D}_t)$ for each task, i.e., the \textbf{discrepancy of task distribution} (see Fig.~\ref{fig:Theory}, d), which is further validated by a theoretical analysis of the forgetting-generalization trade-off \cite{ramakrishnan2020relationship} and the PAC-Bayes bound of generalization errors \cite{pentina2014pac,wang2022coscl}. Therefore, how to exploit task similarity is directly related to the performance of continual learning.
The generalization error for each task can improve with synergistic tasks but deteriorate with competing tasks \cite{ramesh2021model}.
As a naive baseline, learning all incremental tasks equally
in a shared solution would lead to a severe compromise of each task in performance \cite{ramesh2021model,wang2022coscl,doan2022efficient}. To facilitate knowledge transfer and prevent inter-task interference, a promising idea is to combine multiple continual learning models with a proper diversity of task expertise \cite{ramesh2021model,wang2022coscl,doan2022efficient}, as an emerging direction of the \emph{architecture-based approach}.

In addition, the complexity of finding a desirable solution for continual learning is determined to a large extent by the \textbf{structure of parameter space}. Learning all incremental tasks with a shared solution is equivalent to learning each new task in a limited parameter space that prevents performance degradation of all old tasks. Such a classical continual learning problem has been shown to be generally NP-hard \cite{knoblauch2020optimal}, because the feasible parameter space tends to be narrow and irregular as more tasks are introduced, thus difficult to identify (see Fig.~\ref{fig:Theory}, e). This challenging issue can be mitigated by replaying representative old training samples \cite{knoblauch2020optimal}, restricting the feasible parameter space to a hyperrectangle \cite{wolczyk2022continual}, or altering the premise of using a single parameter space (i.e., using multiple continual learning models) \cite{ramesh2021model,wang2022coscl,doan2022efficient}.

As a brief summary, continual learning not only requires a proper balance between stability and plasticity, but also adequates generalizability to accommodate distributional differences within and between tasks, where the complexity of finding such a desirable solution depends on the structure of parameter space.
One recent work \cite{wang2022coscl} presents a unified generalization bound for continual learning that incorporates the above factors, serving as an integrated perspective.
Besides, the theoretical foundation and analysis provide clear motivation for various continual learning strategies, which will be further detailed in the next section.

\section{Method}\label{Sec.4_Method}
In this section, we present an elaborated taxonomy of representative continual learning methods (see Fig.~\ref{fig:Method} and also Fig.~\ref{fig:CL_Framework}, c), and extensively analyze their main motivations, typical implementations and empirical properties.

\subsection{Regularization-based Approach}
This direction is characterized by adding explicit regularization terms to balance the old and new tasks, which usually requires storing a frozen copy of the old model for reference (see Fig.~\ref{fig:Regularization}). Depending on the target of regularization, such methods can be divided into two groups.

The first is \textbf{weight regularization}, which selectively regularizes the variation of network parameters. A typical implementation is to add a quadratic penalty in loss function that penalizes the variation of each parameter depending on its contribution or ``importance'' to performing the old tasks (see Eqn.~\ref{eqn_EWC}), in a form originally derived from online Laplace approximation of the posterior under the Bayesian framework. The importance can be calculated by the Fisher information matrix (FIM), such as EWC \cite{kirkpatrick2017overcoming} and some more advanced versions \cite{ritter2018online,schwarz2018progress}.
Meanwhile, numerous efforts have been devoted to designing a better importance measurement. SI \cite{zenke2017continual_si} online approximates the importance of each parameter by its contribution to the total loss variation and its update length over the entire training trajectory. MAS \cite{aljundi2018memory_mas} accumulates importance measures based on the sensitivity of predictive results to parameter changes, which is both online and unsupervised. RWalk \cite{chaudhry2018riemannian_rwalk} combines the regularization terms of SI \cite{zenke2017continual_si} and EWC \cite{kirkpatrick2017overcoming} to integrate their advantages. Interestingly, these importance measurements have been shown to be all tantamount to an approximation of the FIM \cite{benzing2022unifying}, although stemming from different motivations.

There are also several works focusing on refining the implementation of the quadratic penalty.
Since the diagonal approximation of the FIM in EWC \cite{kirkpatrick2017overcoming} might lose information about the old tasks, R-EWC \cite{liu2018rotate} performs a factorized rotation of the parameter space to diagonalize the FIM. XK-FAC \cite{lee2020continual} further considers the inter-example relations in approximating the FIM to better accommodate batch normalization. Observing the asymmetric effect of parameter changes on old tasks, ALASSO \cite{park2019continual} designs an asymmetric quadratic penalty with one of its sides overestimated.

\begin{figure}[th]
	\centering
	\includegraphics[width=0.82\columnwidth]{Regularizationv2.jpg}

	\caption{Regularization-based approach. This direction is characterized by adding explicit regularization terms to mimic the parameters (weight regularization) or behaviors (function regularization) of the old model.}
	\label{fig:Regularization}

\end{figure}

Compared to learning each task within the constraints of the old model, which typically exacerbates the intransience, an \emph{expansion-renormalization} process of obtaining separately the new task solution and renormalizing it with the old model can provide a better stability-plasticity trade-off \cite{lee2017overcoming_imm,lee2020residual,schwarz2018progress,wang2021afec,lin2022towards}. IMM \cite{lee2017overcoming_imm} is an early attempt that incrementally matches the moment of the posterior distributions of old and new tasks, i.e., a weighted average of their solutions.
ResCL \cite{lee2020residual} extends this idea with a learnable combination coefficient.
P\&C \cite{schwarz2018progress} learns each task individually with an additional network, and then distills it back to the old model with a generalized version of EWC \cite{kirkpatrick2017overcoming}. AFEC \cite{wang2021afec} introduces a forgetting rate to eliminate the potential negative transfer from the original posterior $p(\theta | \mathcal{D}_{1:k-1})$ in Eqn.~\ref{eqn_cl_posterior}, and derives quadratic terms to penalize differences of the network parameters $\theta$ with both the old and new task solutions. To reliably average the old and new task solutions, a linear connector \cite{lin2022towards} is constructed by constraining them on a linear low-error path.

Other forms of regularization that target the network itself also belong to this sub-direction. As discussed before, online VI of the posterior distribution can serve as an implicit regularization of parameter changes, such as VCL \cite{nguyen2018variational,swaroop2019improving}, NVCL \cite{tserannatural}, CLAW \cite{adel2019continual}, GVCL \cite{loo2020generalized}, KCL \cite{derakhshani2021kernel} and VAR-GPs \cite{kapoor2021variational}. Instead of consolidating parameters, NPC \cite{paik2020overcoming} estimates the importance of each neuron and selectively reduces its learning rate. UCL \cite{ahn2019uncertainty} and AGS-CL \cite{jung2020continual} freeze
the parameters connecting the important neurons, equivalent to a hard version of weight regularization.

The second is \textbf{function regularization}, which targets the intermediate or final output of the prediction function.
This strategy typically uses the previously-learned model as the teacher and the currently-trained model as the student, while implementing knowledge distillation (KD) \cite{gou2021knowledge} to mitigate catastrophic forgetting.
Ideally, the target of KD should be all old training samples, which are unavailable in continual learning. The alternatives can be new training samples \cite{li2017learning,dhar2019learning,iscen2020memory,rannen2017encoder}, a small fraction of old training samples \cite{rebuffi2017icarl,castro2018end,hou2019learning_lucir,douillard2020podnet}, external unlabeled data \cite{lee2019overcoming}, generated data \cite{wu2018memory_mrgan,zhai2019lifelong_gan}, etc., suffering from different degrees of distribution shift.

As a pioneer work, LwF \cite{li2017learning} and LwF.MC \cite{rebuffi2017icarl} learn \emph{new training samples} while using their predictions from the output head of the old tasks to compute the distillation loss.
LwM \cite{dhar2019learning} exploits the attention maps of new training samples for KD.
EBLL \cite{rannen2017encoder} learns task-specific autoencoders and prevents changes in feature reconstruction.
GD \cite{lee2019overcoming} further distills knowledge on the large stream of \emph{unlabeled data} available in the wild.
When old training samples are faithfully recovered, the potential of function regularization can be largely released.
Thus, function regularization often collaborates with replaying a few \emph{old training samples}, such as iCaRL \cite{rebuffi2017icarl}, EEIL \cite{castro2018end}, LUCIR \cite{hou2019learning_lucir}, PODNet \cite{douillard2020podnet}, SS-IL \cite{ahn2021ss}, DER \cite{buzzega2020dark}, etc., discussed latter in Sec.~\ref{Sec.4.2_Replay}. Meanwhile, sequential Bayesian inference over function space can be seen as a form of function regularization, which generally requires storing some old training samples (called ``coreset'' in literature), such as FRCL \cite{titsias2019functional}, FROMP \cite{pan2020continual} and S-FSVI \cite{rudner2022continual}.
For conditional generation, the \emph{generated data} of previously-learned conditions and their output values are regularized to be consistent between the teacher and student models, such as MeRGANs \cite{wu2018memory_mrgan}, DRI \cite{wang2022continual_dri} and LifelongGAN \cite{zhai2019lifelong_gan}.

\subsection{Replay-based Approach}\label{Sec.4.2_Replay}
We group the methods for approximating and recovering old data distributions into this direction (see Fig.~\ref{fig:Replay}). Depending on the content of replay, these methods can be further divided into three sub-directions, each with its own challenges.

The first is \textbf{experience replay}, which typically stores a few old training samples within a small memory buffer. Due to the extremely limited storage space, the key challenges consist of \emph{how to construct} and \emph{how to exploit} the memory buffer. As for construction, the stored training samples should be carefully selected, compressed, augmented, and continually adjusted, so as to maximize the adaptive information for recovering the past. Earlier work adopts fixed principles for \emph{sample selection}. For example, Reservoir Sampling \cite{vitter1985random,chaudhry2019tiny,riemer2018learning} randomly stores a fixed amount of training samples from each input batch. Ring Buffer \cite{lopez2017gradient_gem} further ensures an equal number of old training samples per class.
Mean-of-Feature \cite{rebuffi2017icarl} selects an equal number of old training samples that are closest to the feature mean of each class. There are many other fixed principles, such as k-means \cite{chaudhry2019tiny}, plane distance \cite{riemer2018learning} and entropy \cite{riemer2018learning}, but all perform mediocrely \cite{riemer2018learning,chaudhry2019tiny}. More advanced strategies are typically gradient-based or optimizable, by maximizing such as
the sample diversity in terms of parameter gradients (GSS \cite{aljundi2019gradient}), performance of corresponding tasks with cardinality constraints (CCBO \cite{borsos2020coresets}), mini-batch gradient similarity and cross-batch gradient diversity (OCS \cite{yoon2021online}), ability of optimizing latent decision boundaries (ASER \cite{shim2021online}), diversity of robustness against perturbations (RM \cite{bang2021rainbow}), similarity to the gradients of old training samples with respect to the current parameters (GCR \cite{tiwari2022gcr}), etc.

To improve \emph{storage efficiency}, AQM \cite{caccia2020online} performs online continual compression based on a VQ-VAE framework \cite{van2017neural} and saves compressed data for replay. MRDC \cite{wang2021memory} formulates experience replay with data compression through determinantal point processes (DPPs) \cite{kulesza2012determinantal}, and derives a computationally efficient way for online determination of the optimal compression rate. RM \cite{bang2021rainbow} adopts conventional and label mixing-based strategies of data augmentation to enhance the diversity of old training samples. RAR \cite{kumariretrospective} synthesizes adversarial samples near the forgetting boundary and performed MixUp \cite{zhang2018mixup} for data augmentation.
The auxiliary information with low storage cost, such as class statistics (IL2M \cite{belouadah2019il2m}, SNCL \cite{gong2022continual}) and attention maps (RRR \cite{ebrahimi2020remembering}, EPR \cite{saha2020gradient}), can be further incorporated to maintain old knowledge. Besides, the old training samples can be continually adjusted to accommodate incremental changes, such as by making them more representative (Mnemonics \cite{liu2020mnemonics}) or more challenging (GMED \cite{jin2021gradient}) for separation.

As for exploitation, experience replay requires an adequate use of the memory buffer to recover the past information. There are many different implementations, closely related to other continual learning strategies.
First, the effect of old training samples in \emph{optimization} can be constrained to avoid catastrophic forgetting and facilitate knowledge transfer. For example, GEM \cite{lopez2017gradient_gem} constructs individual constraints based on the old training samples for each task to ensure non-increase in their losses.
A-GEM \cite{chaudhry2018efficient_agem} replaces the individual constraints with a global loss of all tasks to improve training efficiency.
LOGD \cite{tang2021layerwise} decomposes the gradient of each task into task-sharing and task-specific components to leverage inter-task information. To achieve a good trade-off in interference-transfer \cite{riemer2018learning} (i.e., forgetting-generalization \cite{ramakrishnan2020relationship}), MER \cite{riemer2018learning} employs meta-learning for gradient alignment in experience replay. BCL \cite{ramakrishnan2020relationship} explicitly pursues a saddle point of the cost of the old and new training samples. MetaSP \cite{sun2022exploring} leverages the Pareto optimum of example influence on stability-plasticity to control the model and storage updates.
To selectively utilize the memory buffer, MIR \cite{aljundi2019mir} prioritizes the old training samples that subject to more forgetting, while HAL \cite{chaudhry2021using} uses them as ``anchors'' and stabilizes their predictions.

\begin{figure}[th]
	\centering
	\includegraphics[width=1\columnwidth]{Replay.jpg}

	\caption{Replay-based approach. This direction is characterized by approximating and recovering the old data distributions. Typical sub-directions include experience replay, which saves a few old training samples in a memory buffer; generative replay, which trains a generative model to provide generated samples; and feature replay, which recovers the distribution of old features through saving prototypes, saving statistical information or training a generative model.}
	\label{fig:Replay}

\end{figure}

On the other hand, experience replay can be naturally combined with \emph{knowledge distillation} (KD), which additionally incorporates the past information from the old model. iCaRL \cite{rebuffi2017icarl} and EEIL \cite{castro2018end} are two early works that perform KD on both old and new training samples. Some subsequent improvements focus on different issues in experience replay.
To mitigate data imbalance of the limited old training samples, LUCIR \cite{hou2019learning_lucir} encourages similar feature orientation of the old and new models, while performing cosine normalization of the last layer and mining hard negatives of the current task. BiC \cite{wu2019large_bic} and WA \cite{zhao2020maintaining} attribute this issue to the bias of the last fully connected layer, and resolve it by either learning a bias correction layer with a balanced validation set \cite{wu2019large_bic}, or normalizing the output weights \cite{zhao2020maintaining}. SS-IL \cite{ahn2021ss} adopts separated softmax in the last layer and task-wise KD to mitigate the bias.
DRI \cite{wang2022continual_dri} trains a generative model to supplement the old training samples with additional generated data.
To alleviate dramatic representation shifts, PODNet \cite{douillard2020podnet} employs a spatial distillation loss to preserve representations throughout the model. Co2L \cite{cha2021co2l} introduces a self-supervised distillation loss to obtain robust representations against catastrophic forgetting.
GeoDL \cite{simon2021learning} performs KD along a path that connects the low-dimensional projections of the old and new feature spaces for a smooth transition between them.
ELI \cite{joseph2022energy} learns an energy manifold with the old and new models to realign the representation shifts for optimizing incremental tasks.
To adequately exploit the past information, AU \cite{kurmi2021not} incorporates uncertainty and self-attention into the distillation loss, while CSC \cite{ashok2022class} additionally leverages the structure of the feature space.
DDE \cite{hu2021distilling} distills colliding effects from the features of the new training samples, which is causally equivalent to replaying more old training samples.
To further enhance learning plasticity, D+R \cite{hou2018lifelong} performs KD from an additional model dedicated to each new task.
FOSTER \cite{wang2022foster} dynamically expands new modules to fit the residuals of the old model and then distills them into a single model.
Besides, many weight regularization approaches can be combined with experience replay to achieve better performance and generality \cite{chaudhry2018riemannian_rwalk,wang2021afec}.

It is worth noting that the merits and potential limitations of experience replay remain largely open. In addition to the intuitive benefits of staying in the low-loss region of the old tasks \cite{verwimp2021rehearsal}, theoretical analysis has demonstrated its contribution to resolving the NP-hard problem of optimal continual learning \cite{knoblauch2020optimal}. However, it risks overfitting to only a few old training samples retained in the memory buffer, which potentially affects generalizability \cite{verwimp2021rehearsal}.
To alleviate overfitting, LiDER \cite{bonicelli2022effectiveness} takes inspirations from adversarial robustness and enforces the Lipschitz continuity of the model to its inputs.
MOCA \cite{yu2022continual} enlarges the variation of representations to prevent the old ones from shrinking in their space.
Interestingly, several simple baselines of experience replay can achieve considerable performance. DER \cite{buzzega2020dark} and its extended version \cite{boschini2022class} store old training samples together with their logits, and perform logit-matching throughout the optimization trajectory. GDumb \cite{prabhu2020gdumb} greedily collects incoming training samples in a memory buffer and then uses them to train a model from scratch for testing. These efforts can serve as an evaluation criterion for more advanced strategies in this sub-direction.

The second is \textbf{generative replay} or pseudo-rehearsal, which generally requires training an additional generative model to replay generated data. This sub-direction is closely related to continual learning of generative models themselves, as they also need to be updated sequentially.
DGR \cite{shin2017continual_dgr} provides an initial framework in which learning each generation task is accompanied with replaying generated data sampled from the old generative model, so as to inherit the previously-learned knowledge. MeRGAN \cite{wu2018memory_mrgan} further employs replay alignment to enforce consistency of the generative data sampled with the same random noise between the old and new generative models, similar to the role of function regularization.
Besides, other continual learning strategies can be incorporated into generative replay.
Weight regularization \cite{seff2017continual,nguyen2018variational,wang2021triple,wang2021ordisco} and experience replay \cite{he2018exemplar,wang2021ordisco} have been shown to be effective in mitigating catastrophic forgetting of generative models.
DGMa/DGMw \cite{ostapenko2019learning_dgmw} adopts binary masks to allocate task-specific parameters for overcoming inter-task interference, and an extendable network to ensure scalability.
If pre-training is available, it can provide a relatively stable and strong reference model for continual learning. For example, FearNet \cite{kemker2018fearnet} and ILCAN \cite{xiang2019incremental} additionally preserves statistical information of the old features acquired from a pre-trained feature extractor, while GAN-Memory \cite{cong2020gan} continually adjusts a pre-trained generative model with task-specific parameters.

The generative models for pseudo-rehearsal can be of various types, such as generative adversarial networks (GANs) and (variational) autoencoder (VAE).
A majority of state-of-the-art approaches have focused on GANs to enjoy its advantages in fine-grained generation, but usually suffer from label inconsistency in continual learning \cite{ostapenko2019learning_dgmw,ayub2020eec}. In contrast, autoencoder-based strategies, such as FearNet \cite{kemker2018fearnet}, SRM \cite{riemer2019scalable}, CLEER \cite{rostami2019complementary}, EEC \cite{ayub2020eec}, GMR \cite{pfulb2021continual} and Flashcards \cite{gopalakrishnan2022knowledge}, can explicitly control the labels of the generated data, although relatively blurred. L-VAEGAN \cite{ye2020learning} instead employs a hybrid model for both high-quality generation and accurate inference.
However, since continual learning of generative models is extremely difficult and requires significant resource overhead, generative replay is typically limited to relatively simple datasets \cite{van2020brain,wang2021triple}. An alternative is to convert the target of generative replay from data level to feature level, which can largely reduce the complexity of conditional generation and more adequately exploit semantic information. For example, GFR \cite{liu2020generative} trains a conditional GANs to replay generated features after the feature extractor. BI-R \cite{van2020brain} incorporates context-modulated feedback connections in a standard VAE to replay internal representations.

In fact, maintaining feature-level rather than data-level distributions enjoys numerous benefits in terms of efficiency and privacy. We categorize this sub-direction as \textbf{feature replay}. However, a central challenge is the \emph{representation shift} caused by sequentially updating the feature extractor, which reflects the feature-level catastrophic forgetting. To address this issue, GFR \cite{liu2020generative}, FA \cite{iscen2020memory} and DSR \cite{zhu2022self} perform feature distillation between the old and new models. IL2M \cite{belouadah2019il2m} and SNCL \cite{gong2022continual} recover statistics of feature representations (e.g., mean and covariance) on the basis of experience replay.
RER \cite{toldo2022bring} explicitly estimates the representation shift to update the preserved old features.
REMIND \cite{hayes2020remind} and ACAE-REMIND \cite{wang2021acae} instead fix the early layers of the feature extractor and reconstruct the intermediate representations to update the latter layers.
FeTrIL \cite{petit2022fetril} employs a fixed feature extractor learned from the initial task and replays generated features afterwards.

For continual learning from scratch, the required changes in representation are often dramatic, while stabilizing the feature extractor may interfere with accommodating new representations.
In contrast, the use of strong pre-training can provide robust representations that are generalizable to downstream tasks and remain relatively stable in continual learning.
An empirical study \cite{ostapenko2022foundational} has systematically investigated feature replay for continual learning with large-scale pre-training.
Recent prompting-based approaches such as L2P \cite{wang2022learning_l2p}, DualPrompt \cite{wang2022dualprompt} and S-Prompts \cite{wang2022sprompts} adopt a small set of learnable parameters (i.e., prompts) to instruct the representations of a pre-trained transformer and preserve such parameters in a memory space, which can be seen as an extension of this strategy.

\subsection{Optimization-based Approach}\label{Sec.4.3_Optimization}
Continual learning can be achieved not only by adding additional terms to the loss function (e.g., regularization and replay), but also by explicitly designing and manipulating the optimization programs.

A typical idea is to perform \textbf{gradient projection}.
Some replay-based approaches such as GEM \cite{lopez2017gradient_gem}, A-GEM \cite{chaudhry2018efficient_agem}, LOGD \cite{tang2021layerwise} and MER \cite{riemer2018learning} constrain parameter updates to align with the direction of experience replay, corresponding to preserving the previous input space and gradient space through some old training samples.
In contrast to saving old training samples, OWM \cite{zeng2019continual} and AOP \cite{guo2022adaptive} modify parameter updates to the orthogonal direction of the previous input space.
OGD \cite{farajtabar2020orthogonal} stores the old gradient directions and rectifies the current gradient orthogonal to them.
Orthog-Subspace \cite{chaudhry2020continual} performs continual learning with orthogonal low-rank vector subspaces and keeps the gradients of different tasks orthogonal to each other.
GPM \cite{saha2020gradient} maintains the gradient subspace important to the old tasks (i.e., the bases of core gradient space) for orthogonal projection in updating parameters.
CGP \cite{chen2022class} calculates such gradient subspace from individual classes to additionally mitigate inter-class interference.
FS-DGPM \cite{deng2021flattening} dynamically releases unimportant bases of GPM \cite{saha2020gradient} to improve learning plasticity and encourages the convergence to a flat loss landscape.
CUBER \cite{lin2022beyond} selectively projects gradients to update the knowledge of old tasks that are positively related to the new task.
TRGP \cite{lin2021trgp} defines the ``trust region'' through the norm of gradient projection onto the subspace of previous inputs, so as to selectively reuse the frozen weights of old tasks.
Adam-NSCL \cite{wang2021training} instead projects candidate parameter updates into the current null space approximated by the uncentered feature covariance of the old tasks, while AdNS \cite{kong2022balancing} considers the shared part of the previous and the current null spaces.
NCL \cite{kao2021natural} unifies Bayesian weight regularization and gradient projection, encouraging parameter updates in the null space of the old tasks while converging to a maximum of the Bayesian approximation posterior.
Under the upper bound of the quadratic penalty in Bayesian weight regularization, RGO \cite{liu2021continual} modifies gradient directions with a recursive optimization procedure to obtain the optimal solution.
Therefore, as regularization and replay are ultimately achieved by rectifying the current gradient, gradient projection corresponds to a similar modification but explicitly in parameter updates.

\begin{figure}[th]
	\centering
	\includegraphics[width=1\columnwidth]{Optimization.jpg}

	\caption{Optimization-based approach. This direction is characterized by explicit design and manipulation of the optimization programs, such as gradient projection with reference to the gradient space or input space of the old tasks (adapted from \cite{farajtabar2020orthogonal}), meta-learning of sequentially arrived tasks within the inner loop, and exploitation of mode connectivity and flat minima in loss landscape (adapted from \cite{mirzadeh2020linear,lin2022towards}). $\theta_A^*$, $\theta_B^*$ and $\theta_{A,B}^*$ are desirable solutions for task $A$, task $B$ and both of them, respectively. }
	\label{fig:Optimization}

\end{figure}

Another attractive idea is \textbf{meta-learning} or learning-to-learn for continual learning, which attempts to automatically learn a data-driven inductive bias for various scenarios, rather than designing it manually \cite{hadsell2020embracing}.
OML \cite{javed2019meta} provides a meta-training strategy that performs online updates on the sequentially arrived inputs and minimizes their interference, which can naturally obtain sparse representations suitable for continual learning.
ANML \cite{beaulieu2020learning} extends this idea by meta-learning of a context-dependent gating function to selectively activate neurons with respect to incremental tasks.
AIM \cite{lee2021few} learns a mixture of experts to make predictions with the representations of OML \cite{javed2019meta} or ANML \cite{beaulieu2020learning}, further sparsifying the representations at the level of architecture.
Meanwhile, meta-learning can work with experience replay to better utilize both the old and new training samples. For example, MER \cite{riemer2018learning} aligns their gradient directions, while iTAML \cite{rajasegaran2020itaml} applies a meta-updating rule to keep them in balance with each other. With the help of experience replay, La-MAML \cite{gupta2020look} optimizes the OML \cite{javed2019meta} objective in an online fashion with an adaptively modulated learning rate.
OSAKA \cite{caccia2020osaka} proposes a hybrid objective of knowledge accumulation and fast adaptation, which can be resolved by obtaining a good initialization with meta-training and then incorporating knowledge of incremental tasks into the initialization.
Meta-learning can also be used to optimize specialized architectures.
MERLIN \cite{kj2020meta} consolidates a meta-distribution of model parameters given the representations of each task, which allows to sample task-specific models and ensemble them for inference. Similarly, PR \cite{henning2021posterior} adopts a Bayesian strategy to learn task-specific posteriors with a shared meta-model.
MARK \cite{hurtado2021optimizing} maintains a set of shared weights that are incrementally updated with meta-learning and selectively masked to solve specific tasks.
ARI \cite{wang2022anti} combines adversarial attacks with experience replay to obtain task-specific models, which are then fused together through meta-training.

Besides, some other works refine the optimization process from a \textbf{loss landscape} perspective.
For example, rather than dedicating an algorithm, Stable-SGD \cite{mirzadeh2020understanding} enables SGD to find a flat local minima by adapting the factors in training regime, such as dropout, learning rate decay and batch size.
MC-SGD \cite{mirzadeh2020linear} empirically demonstrates that the local minima obtained by multi-task learning (i.e., joint training of all incremental tasks) and continual learning can be connected by a linear path of low error, and applies experience replay to find a better solution along it.
Linear Connector \cite{lin2022towards} adopts Adam-NSCL \cite{wang2021training} and feature distillation to obtain respective solutions of the old and new tasks connected by a linear path of low error, followed by linear averaging.
Further, un-/self-supervised learning (than traditional supervised learning) \cite{gallardo2021self,hu2021well,madaan2021representational} and large-scale pre-training (than random initialization) \cite{hu2021well,wu2022class,davari2022probing,ramasesh2021effect,mehta2021empirical} have been shown to suffer from less catastrophic forgetting.
Empirically, both can be attributed to obtaining a more robust (e.g., orthogonal, sparse and uniformly-scattered) representation \cite{hu2021well,madaan2021representational,shi2022mimicking,ramasesh2021effect}, and converging to a wider loss basin \cite{hu2021well,madaan2021representational,mehta2021empirical,neyshabur2020being,hao2019visualizing}, suggesting a potential link among the sensitivity of representations, parameters and task-specific errors.
Many efforts seek to leverage these advantages in continual learning, as we discuss next.

\subsection{Representation-based Approach}\label{Sec.4.4_Representation}
We group the approaches that create and exploit the strengths of representations for continual learning into this category.
In addition to an earlier work on obtaining sparse representations through meta-training \cite{javed2019meta}, recent works have attempted to incorporate the advantages of self-supervised learning (SSL) \cite{gallardo2021self,madaan2021representational,pham2021dualnet} and large-scale pre-training \cite{mehta2021empirical,shi2022mimicking,wu2022class} to improve the representations in initialization and in continual learning.
Note that these two strategies are closely related, since the pre-training data is usually of a huge amount and without explicit labels, while the performance of SSL itself is mainly evaluated by fine-tuning on (a sequence of) downstream tasks.
Below, we will discuss representative sub-directions.

The first is to implement \textbf{self-supervised learning} (basically with contrastive loss) for continual learning.
Observing that self-supervised representations are more robust to catastrophic forgetting, LUMP \cite{madaan2021representational} acquires further improvements by interpolating between instances of the old and new tasks.
MinRed \cite{purushwalkam2022challenges} further promotes the diversity of experience replay by de-correlating the stored old training samples.
CaSSLe \cite{fini2022self} converts the self-supervised loss to a distillation strategy by mapping the current state of a representation to its previous state.
Co2L \cite{cha2021co2l} adopts a supervised contrastive loss to learn each task and a self-supervised loss to distill knowledge between the old and new models.
DualNet \cite{pham2021dualnet} trains a fast learner with supervised loss and a slow learner with self-supervised loss, with the latter helping the former to acquire generalizable representations.

\begin{figure}[th]
	\centering
	\includegraphics[width=0.85\columnwidth]{Representation.jpg}

	\caption{Representation-based approach. This direction is characterized by creating and leveraging the strengths of representations for continual learning, such as by using self-supervised learning and pre-training. In particular, both upstream pre-training and downstream fine-tuning require continual learning, while the pre-trained representations are optionally fixed for learning specific downstream tasks.}
	\label{fig:Representation}

\end{figure}

The second is to use \textbf{pre-training for downstream continual learning}.
Several empirical studies suggest that downstream continual learning benefits from pre-training, which brings not only strong knowledge transfer but also robustness to catastrophic forgetting \cite{gallardo2021self,ramasesh2021effect,mehta2021empirical,ostapenko2022foundational}. In particular, the benefits for downstream continual learning tend to be more apparent when pre-training with larger data size \cite{ramasesh2021effect,ostapenko2022foundational}, larger model size \cite{ramasesh2021effect} and contrastive loss \cite{gallardo2021self,davari2022probing}.
However, a critical challenge is that the pre-trained knowledge needs to be adaptively leveraged for the current task while maintaining generalizability to future tasks. There are various strategies for this problem, depending on whether the pre-trained representations are fixed or not.

As for adapting the \emph{fixed representations}, Side-Tuning \cite{zhang2020side} and DLCFT \cite{shon2022dlcft} train a lightweight network in parallel with the pre-trained model and fuse their outputs linearly.
TwF \cite{boschini2022transfer} also trains a sibling network, but distills knowledge from the pre-trained model in a layer-wise manner.
GAN-Memory \cite{cong2020gan} takes advantage of FiLM \cite{perez2018film} and AdaFM \cite{zhao2020leveraging} to learn task-specific parameters for each layer of a pre-trained generative model, while ADA \cite{ermis2022memory} employs Adapters \cite{houlsby2019parameter} with knowledge distillation to adjust a pre-trained transformer.
L2P \cite{wang2022learning_l2p}, DualPrompt \cite{wang2022dualprompt} and S-Prompts \cite{wang2022sprompts} learn incremental tasks via a few prompting parameters attached to a pre-trained transformer.
As for optimizing the \emph{updatable representations}, F2M \cite{shi2021overcoming} searches for flat local minima in the pre-training stage, and then learns incremental tasks within the flat region.
CwD \cite{shi2022mimicking} regularizes the initial-phase representations to be uniformly scattered, which can empirically mimic the representations of joint training.
SAM \cite{mehta2021empirical,foret2020sharpness} encourages finding a wide basin in downstream continual learning by optimizing the flatness metric.

The third is \textbf{continual pre-training} (CPT) or continual meta-training.
As the huge amount of data required for pre-training is typically collected in an incremental manner, performing upstream continual learning to improve downstream performance is particularly important.
For example, an empirical study finds that self-supervised pre-training is more effective than supervised protocols for continual learning of vision-language models \cite{cossu2022continual}, consistent with the results for only visual tasks \cite{hu2021well}.
Since texts are generally more efficient than images, IncCLIP \cite{yan2022generative} replays generated hard negative texts conditioned on images and performs multi-modal knowledge distillation for updating CLIP \cite{radford2021learning}.
For CPT of language models, ECONET \cite{han2021econet} designs a self-supervised framework with generative replay.
Meanwhile, continual meta-training needs to address a similar issue that the base knowledge is incrementally enriched and adapted.
IDA \cite{liu2020incremental} imposes discriminants of the old and new models to be aligned relative to the old centers, and otherwise leaves the embedding free to accommodate new tasks.
ORDER \cite{wang2022meta} employs unlabeled OOD data with experience replay and feature replay to cope with highly dynamic task distributions.

\subsection{Architecture-based Approach}
The above strategies basically focus on learning all incremental tasks with a shared set of parameters (i.e., a single model as well as one parameter space), which is a major cause of the inter-task interference. In contrast, constructing task-specific parameters can explicitly resolve this problem.
Previous work generally separates this direction into \emph{parameter isolation} and \emph{dynamic architecture}, depending on whether the network architecture is fixed or not.
Here, we instead focus on the way of implementing task-specific parameters, extending the above concepts to parameter allocation, model decomposition and modular network (Fig.~\ref{fig:Architecture}).

\textbf{Parameter allocation} features an isolated parameter subspace dedicated to each task throughout the network, where the architecture can be fixed or dynamic in size.
Within a \emph{fixed} network architecture, Piggyback \cite{mallya2018piggyback}, HAT \cite{serra2018overcoming}, SupSup \cite{wortsman2020supermasks}, MEAT \cite{xue2022meta}, WSN \cite{kang2022forget} and H$^2$ \cite{jin2022helpful} explicitly optimize a binary mask to select dedicated neurons or parameters for each task, with the masked regions of the old tasks (almost) frozen to prevent catastrophic forgetting.
PackNet \cite{mallya2018packnet}, UCL \cite{ahn2019uncertainty}, CLNP \cite{golkar2019continual}, AGS-CL \cite{jung2020continual} and NISPA \cite{gurbuz2022nispa} explicitly identify the important neurons or parameters for the current task and then release the unimportant parts to the following tasks, which can be achieved by iterative pruning \cite{mallya2018packnet}, activation value \cite{golkar2019continual,jung2020continual,gurbuz2022nispa}, uncertainty estimation \cite{ahn2019uncertainty}, etc.
Since the network capacity is limited, ``free'' parameters tend to saturate as more incremental tasks are introduced. Therefore, these methods typically require sparsity constraints on parameter usage and selective reuse of the frozen old parameters, which might affect the learning of each task.
To alleviate this dilemma, the network architecture can be \emph{dynamically expanded} if its capacity is not sufficient to learn a new task well, such as DEN \cite{yoon2018lifelong}, CPG \cite{hung2019compacting} and DGMa/DGMw \cite{ostapenko2019learning_dgmw}.
The dynamic architecture can be explicitly optimized to improve parameter efficiency and knowledge transfer, such as by reinforcement learning (RCL \cite{xu2018reinforced}, BNS \cite{qin2021bns}), architecture search (LtG \cite{li2019learn}, BNS \cite{qin2021bns}), variational Bayes (BSA \cite{kumar2021bayesian}), etc. As the network expansion should be much slower than the task increase to ensure scalability, constraints on sparsity and reusability remain important.

\begin{figure}[th]
	\centering
	\includegraphics[width=0.92\columnwidth]{Architecture.jpg}

	\caption{Architecture-based approach. This direction is characterized by constructing task-specific/adaptive parameters with a properly-designed architecture, such as assigning dedicated parameters to each task (parameter allocation), constructing task-adaptive sub-modules or sub-networks (modular network), and decomposing the model into task-sharing and task-specific components (model decomposition). Here we exhibit two types of model decomposition, corresponding to parameters (low-rank factorization, adapted from \cite{hyder2022incremental}) and representations (masking of intermediate features).}
     \label{fig:Architecture}
    \vspace{-.2cm}
\end{figure}

\textbf{Model decomposition} explicitly separates a model into task-sharing and task-specific components, where the task-specific components are typically expandable.
For a regular network, the task-specific components could be parallel branches (ACL \cite{ebrahimi2020adversarial}, ReduNet \cite{wu2021incremental}, EPIE-Net \cite{du2022efficient}), adaptive layers (GVCL \cite{loo2020generalized}, DyTox \cite{douillard2022dytox}), masks or mask generators for intermediate features (CCLL \cite{singh2020calibrating}, CCG \cite{abati2020conditional}, MARK \cite{hurtado2021optimizing}).
Note that the feature masks for model decomposition do not operate in parameter space and are not binary for each task, thus fundamentally different from the binary masks for parameter allocation.
Besides, the network parameters themselves can be decomposed into task-sharing and task-specific elements, such as by additive decomposition (APD \cite{yoon2019scalable}), singular value decomposition (RCM \cite{kanakis2020reparameterizing}), filter atom decomposition (FAS \cite{miao2021continual}) and low-rank factorization (IBP-WF \cite{mehta2021continual}, IRU \cite{hyder2022incremental}).
As the number of task-specific components usually grows linearly with incremental tasks, their resource efficiency determines the scalability of this sub-direction.

\textbf{Modular network} leverages parallel sub-networks or sub-modules to learn incremental tasks in a differentiated manner, without pre-defined task-sharing or task-specific components. As an early work, Progressive Networks \cite{rusu2016progressive} introduces an identical sub-network for each task and allows knowledge transfer from other sub-networks via adaptor connections.
Expert Gate \cite{aljundi2017expert} and a subsequent work \cite{collier2020routing} employ a mixture of experts \cite{jacobs1991adaptive} to learn incremental tasks, expanding one expert as each task is introduced.
PathNet \cite{fernando2017pathnet} and RPSNet \cite{rajasegaran2019random} pre-allocate multiple parallel networks to construct a few candidate paths, i.e., layer-wise compositions of network modules, and select the best path for each task.
MNTDP \cite{veniat2020efficient} and LMC \cite{ostapenko2021continual} seek to explicitly find the optimal layout from previous sub-modules and potentially new sub-modules.
Similar to parameter allocation, these efforts are intentional to construct task-specific models, while the combination of sub-networks or sub-modules allows explicit reuse of knowledge.
In addition, the sub-networks can be encouraged to learn incremental tasks in parallel.
Model Zoo \cite{ramesh2021model} expands a sub-network to learn each new task with experience replay of the old tasks, and ensembles all sub-networks for prediction.
CoSCL \cite{wang2022coscl} ensembles multiple continual learning models and modulates the predictive similarity between them, proving to be effective in resolving the discrepancy of task distributions and improving the flatness of loss landscape.

In a broader sense, sampling parameters from task-conditioned parameter distributions (MERLIN \cite{kj2020meta}, PR \cite{henning2021posterior}, PGMA \cite{hu2019overcoming}, HNET \cite{von2019continual}), as well as stabilizing important parameters with weight regularization, can be seen as a form of deriving task-specific/adaptive parameters.
In contrast to other directions, most architecture-based approaches amount to de-correlating incremental tasks in network parameters, which can almost avoid catastrophic forgetting but affect scalability and inter-task generalizability. In particular, task identities are often required to determine which set of parameters to use, thus greatly limiting their applications.
To overcome this limitation, task identities can be inferred from the responses (e.g., predictive uncertainty) of each task-specific model \cite{aljundi2017expert,henning2021posterior,kim2022theoretical}.
Also, the function of task-identity prediction can be explicitly learned from incremental tasks, using other continual learning strategies to mitigate catastrophic forgetting \cite{collier2020routing,jin2022helpful,ebrahimi2020adversarial,abati2020conditional,mehta2021continual,henning2021posterior,kj2020meta}.

Besides, the design and choice of a \textbf{basic architecture} can largely impact the continual learning performance. For example, wider networks tend to be more robust to catastrophic forgetting due to more orthogonality and sparsity in gradient directions, as well as a lazier training regime \cite{mirzadeh2022wide,mirzadeh2022architecture}.
Batch Normalization (BN) layers \cite{ioffe2015batch} tend to introduce biased moments towards the current task, resulting in sub-optimal performance of the previous tasks \cite{pham2021continual}.
Dropout \cite{srivastava2014dropout} in a stable network behaves like a gating mechanism to create task-specific pathways and thus can mitigate catastrophic forgetting \cite{mirzadeh2020dropout}.

\section{Scenario Complexity in Application: The Case of Visual Classification}\label{Sec.5_Scenario}
Due to the complexity of the real world, practical applications present a variety of specialized challenges. Here we categorize these challenges into \textbf{scenario complexity} and \textbf{task specificity}, with an extensive analysis of how continual learning strategies are adapted to them.
In this section, we demonstrate the challenges of scenario complexity, using visual classification as an example.

\subsection{Class-Incremental Learning}
For continual learning, \emph{Task Incremental Learning} (TIL) usually serves as a basic setting, i.e., task identities are provided in both training and testing. In contrast, \emph{Class Incremental Learning} (CIL) is characterized by task-agnostic inference, i.e., testing without providing task identities, which is more natural but also more challenging for classification tasks.
For example, let's consider two binary classification tasks: (1) ``zebra'' and ``elephant''; and (2) ``hare'' and ``robin''. After learning these two tasks, TIL needs to know which task it is and then classify the two classes accordingly, while CIL directly classifies the four classes at the same time. Therefore, CIL has received increasingly more attention and become almost the most representative scenario for continual learning.

\begin{figure}[ht]
	\centering
	\includegraphics[width=0.98\columnwidth]{CIL.jpg}

	\caption{Representative strategies for class-incremental learning. Catastrophic forgetting can be mitigated with respect to data space (experience replay), feature space (knowledge distillation) and label space (knowledge distillation). This figure is adapted from \cite{hu2021distilling}.}
	\label{fig:CIL}

\end{figure}

The CIL problem can be disentangled into \emph{within-task prediction} and \emph{task-identity prediction}, where the latter is a particular challenge posed by task-agnostic inference and has been shown to be closely related to the OOD detection \cite{kim2022theoretical}.
To overcome catastrophic forgetting in CIL, numerous efforts attempt to impose the behaviors of the previous model on the current model, in terms of data, feature, and label spaces (Fig.~\ref{fig:CIL}).
Since the incremental classes are disjoint, the new training samples are usually OOD from the old ones. In this regard, replay of the old training samples is able to impose an end-to-end effect, while feature and label distillations of the new training samples are limited to the output side with a biased distribution \cite{hu2021distilling}.
Therefore, many state-of-the-art methods are built on the framework of \emph{experience replay} and then incorporate \emph{knowledge distillation}, as we have discussed in Sec.~\ref{Sec.4.2_Replay}. In Table~\ref{table:CIL}, we summarize these CIL methods based on their main focus.

\begin{table}[th]
    \renewcommand\arraystretch{1.8}

	\caption{Summary of the state-of-the-art CIL methods with experience replay.
	These methods further improve the memory buffer, feature distillation or label distillation to achieve better performance, corresponding to data, feature and label spaces, respectively.}

       \centering
      \smallskip

      \begin{tabular}{l|m{5.9cm}}
	 \hline
      Main Focus & Representative Method \\
    \hline
     Data Space &iCaRL \cite{rebuffi2017icarl}, GSS \cite{aljundi2019gradient}, Mnemonics \cite{liu2020mnemonics}, TPCIL \cite{tao2020topology}, GDumb \cite{prabhu2020gdumb}, DER++ \cite{buzzega2020dark}, RMM \cite{liu2021rmm}, HAL \cite{chaudhry2021using}, MRDC \cite{wang2021memory}, CSI \cite{kim2022theoretical}, X-DER \cite{boschini2022class}  \\
      \hline
     Feature Space&LUCIR \cite{hou2019learning_lucir}, PODNet \cite{douillard2020podnet}, TPCIL \cite{tao2020topology}, PCL \cite{hu2021continual}, AANets \cite{liu2021adaptive}, DER \cite{yan2021dynamically}, DDE \cite{hu2021distilling}, GeoDL \cite{simon2021learning}, PASS \cite{zhu2021prototype}, Co2L \cite{cha2021co2l}, AFC \cite{kang2022class}, SP-CIL \cite{wu2022class}, ELI \cite{joseph2022energy}, CwD \cite{shi2022mimicking}, CSCCT \cite{ashok2022class}, FOSTER \cite{wang2022foster}, FASP \cite{miao2021continual}, CLS-ER \cite{arani2021learning}  \\
      \hline
     Label Space&LwF \cite{li2017learning}, iCaRL \cite{rebuffi2017icarl}, GEM \cite{lopez2017gradient_gem}, A-GEM \cite{chaudhry2018efficient_agem}, EEIL \cite{castro2018end}, BiC \cite{wu2019large_bic}, WA \cite{zhao2020maintaining}, DER++ \cite{buzzega2020dark}, ScaIL \cite{belouadah2020scail}, S\&B \cite{kim2021split}, SS-IL \cite{ahn2021ss}, Coil \cite{zhou2021co} \\

    \hline
	\end{tabular}

	\label{table:CIL}

\end{table}

To avoid the additional resource overhead and potential privacy issues of retaining old training samples, many efforts attempt to perform CIL without experience replay, i.e., \emph{Data-Free} CIL. An intriguing idea is to replay synthetic data produced by inverting a frozen copy of the old classification model, such as DeepInversion \cite{yin2020dreaming}, ABD \cite{smith2021always}, RRL \cite{gao2022r} and CF-IL \cite{pourkeshavarzi2021looking}, which usually further incorporate knowledge distillation to compensate the lost information in model inversion.
Other methods exploit the class-wise statistics of feature representations to obtain a balanced classifier, such as by imposing the representations to be transferable and invariant (SPB \cite{wu2021striking}, IL2A \cite{zhu2021class}, SSRE \cite{zhu2022self}, FeTrIL \cite{petit2022fetril}), or compensating explicitly the representation shifts (SDC \cite{yu2020semantic}, RER \cite{toldo2022bring}).

For continual learning (especially CIL) of visual classification tasks, the current state-of-the-art methods mainly focus on image classification with attention to relatively complex and large-scale datasets, such as ILSVRC2012 \cite{krizhevsky2012imagenet} and its derivatives.
There are also many benchmarks for video classification \cite{lomonaco2017core50,roady2020stream,park2021class,villa2022vclimb}, varying in size and purpose.

\subsection{Scarcity of Labeled Data}
Most of the current continual learning settings assume that incremental tasks have sufficiently large amounts of labeled data, which is often expensive and difficult to obtain in practical applications. For this reason, there is a growing body of work focusing on the scarcity of labeled data in continual learning.

A representative scenario is \emph{Few-Shot Continual Learning} (FSCL) or further specified as \emph{Few-Shot} CIL (FSCIL) \cite{tao2020few}, where the model first learns some base classes for initialization with a large number of training samples, and then learns a sequence of novel classes with only a small number of training samples.
The extremely limited training samples exacerbate the overfitting of previously-learned representations to subsequent tasks, which can be alleviated by recent work such as preserving the topology of representations (TOPIC \cite{tao2020few}), constructing an exemplar relation graph for knowledge distillation (ERL \cite{dong2021few}), selectively updating only the unimportant parameters (FSLL \cite{mazumder2021few}) or stabilizing the important parameters (LCwoF \cite{kukleva2021generalized}), cooperating fast-slow weights (MgSvF \cite{zhao2021mgsvf}), updating parameters within the flat region of loss landscape (F2M \cite{shi2021overcoming}), meta-learning of a good initialization (MetaFSCIL \cite{chi2022metafscil}), and generative replay of old data distributions (ERDFR \cite{liu2022few}).

There are many other efforts keeping the representations (i.e., the backbone) fixed in subsequent continual learning, so as to decouple the learning of \emph{representation} and \emph{classifier}. Under this framework, representative strategies can be conceptually separated into (1) obtaining compatible and extensible representations from the base classes, such as by enforcing the representations compatible with simulated incremental tasks (SPPR \cite{zhu2021self}, LIMIT \cite{zhou2022few}), reserving the feature space with virtual prototypes for future classes (Fact \cite{zhou2022forward}), using angular penalty loss with data augmentation (ALICE \cite{peng2022few}), using self-supervised learning (S3C \cite{kalla2022s3c}), and providing extra constraints from margin-based representations (CLOM \cite{zou2022margin}); and (2) obtaining an adaptive classifier from a sequence of novel classes, such as by evolving the classifier weights with a graph attention network (CEC \cite{zhang2021few}), developing a tree-based hierarchical classifier with Gaussian processes (GP-Tree \cite{achituve2021gp}), performing hyperdimensional computing (C-FSCIL \cite{hersche2022constrained}), and sampling stochastic classifiers from a weight distribution (S3C \cite{kalla2022s3c}).
Besides, auxiliary information such as semantic word vectors (SKD \cite{cheraghian2021semantic}, Mixture of Subspaces \cite{cheraghian2021synthesized}, Subspace Reg \cite{akyurek2021subspace}, FSIL-GAN \cite{agarwal2022semantics}) and sketch exemplars (DIY \cite{bhunia2022doodle}) can be incorporated to enrich the limited training samples.

In addition to a few labeled data, there is usually a large amount of unlabeled data available and collected over time. Based on this observation, ORDisCo \cite{wang2021ordisco} first presents the setting of \emph{Semi-Supervised Continual Learning} (SSCL), where the incremental data are partially labeled, and then extends it to IIL and CIL scenarios. However, representative continual learning strategies such as experience replay and weight regularization are less effective in exploiting unlabeled data in SSCL \cite{wang2021ordisco}. To address this challenging issue, ORDisCo \cite{wang2021ordisco} learns a semi-supervised classification model together with a conditional GANs for generative replay, and regularizes discriminator consistency to mitigate catastrophic forgetting.
Targeting a similar setting, subsequent work includes training an adversarial autoencoder to reconstruct images \cite{lechat2021semi}, and imposing predictive consistency among augmented and interpolated data \cite{boschini2022continual}.
Besides, other work considers a slightly different setting, where an additional unlabeled dataset is used for knowledge distillation (GD \cite{lee2019overcoming}) and data augmentation (L2I \cite{tang2022learning}), in order to facilitate supervised continual learning.

As for all incremental data being unlabeled, i.e., \emph{Unsupervised Continual Learning} (UCL), CURL \cite{rao2019continual} is an initial attempt that performs task-identity prediction with a mixture-of-Gaussian latent space and mitigates catastrophic forgetting with generative replay. Subsequent work applies UCL primarily in upstream representation learning or pre-training, exploiting and improving the advantages of self-supervised representations in suffering less catastrophic forgetting \cite{hu2021well,madaan2021representational}, as discussed in Sec.~\ref{Sec.4.3_Optimization}, \ref{Sec.4.4_Representation}.

\subsection{General Continual Learning}
\emph{General Continual Learning} (GCL) \cite{aljundi2019task,de2021continual_review,buzzega2020dark} is a broad concept that describes potential challenges in terms of learning paradigms, where the model observes incremental data in an online fashion without explicit task boundaries. Correspondingly, GCL consists of two interconnected settings: \emph{Task-Free Continual Learning} (TFCL) \cite{aljundi2019task}, where the task identities are not accessible in either training or testing; and \emph{Online Continual Learning} (OCL) \cite{aljundi2019gradient}, where the training samples are observed in an one-pass data stream. Since TFCL usually accesses only a small batch of training samples at a time for gradual changes in task distributions, while OCL usually requires only the data label rather than the task identity for each training sample, many methods for TFCL and OCL are compatible.
Here we summarize these methods and present their target scenarios in Table~\ref{table:gcl}.

Some of them attempt to learn specialized parameters in a growing architecture. For TFCL, CN-DPM \cite{lee2019neural_cndpm} adopts Dirichlet process mixture models to construct a growing number of neural experts, while a concurrent work \cite{jerfel2019reconciling} derives such mixture models from a probabilistic meta-learner. VariGrow \cite{ardywibowo2022varigrow} employs an energy-based novelty score to decide whether to extend a new expert or update an old one. ODDL \cite{ye2022task} estimates the discrepancy between the current memory buffer and the previously-learned knowledge as an expansion signal. For OCL, InstAParam \cite{chen2020mitigating} selects and consolidates appropriate network paths for individual training samples.

\begin{table}[th]
    \renewcommand\arraystretch{1.8}
     \centering

	\caption{Summary of TFCL, OCL and GCL (both TFCL and OCL) methods. Of note, some methods are not designed for specific scenarios, but are used as strong baselines in subsequent work.}

      \smallskip

      \begin{tabular}{l|m{6.7cm}}
	 \hline
     Scenario & Representative Method \\
    \hline
     TFCL & CN-DPM \cite{lee2019neural_cndpm}, \cite{jerfel2019reconciling}, VariGrow \cite{ardywibowo2022varigrow}, ODDL \cite{ye2022task} \\
      \hline
     OCL & InstAParam \cite{chen2020mitigating}, CBRS \cite{chrysakis2020online}, ASER \cite{shim2021online}, InfoRS \cite{sun2021information}, DVC \cite{gu2022not}, CTN \cite{pham2020contextual}, NCCL \cite{yin2021mitigating}, ILOS \cite{he2020incremental}, SCR \cite{mai2021supervised}, CVT \cite{wang2022online}, OCM \cite{guo2022online}, ER-ACE \cite{caccia2021new}, RAR \cite{zhang2022simple}, PoLRS \cite{cai2021online}, AOP \cite{guo2022adaptive}, CV \cite{he2022online} \\
      \hline
     GCL & GSS \cite{aljundi2019gradient}, MIR \cite{aljundi2019mir}, CLIB \cite{koh2021online}, GMED \cite{jin2021gradient}, DRO \cite{wang2022improving}, GDumb \cite{prabhu2020gdumb}, DER \cite{buzzega2020dark}, GEM \cite{lopez2017gradient_gem}, A-GEM \cite{chaudhry2018efficient_agem}, DSDM \cite{pourcel2022online}, CoPE \cite{de2021continual}, BLD \cite{fini2020online} \\
    \hline
	\end{tabular}

	\label{table:gcl}

\end{table}

On the other hand, many TSCL methods and most OCL methods are built on experience replay, focusing on construction, management and exploitation of a memory buffer. Since training samples of the same distribution arrive in small batches, the information of task boundaries is less effective, and reservoir sampling usually serves as an effective baseline strategy for sample selection. More advanced strategies prioritize the replay of those training samples that are informative (InfoRS \cite{sun2021information}), diversified in parameter gradients (GSS \cite{aljundi2019gradient}) or previously-learned knowledge (ODDL \cite{ye2022task}), balanced in class labels (CBRS \cite{chrysakis2020online}, GDumb \cite{prabhu2020gdumb}, CoPE \cite{de2021continual}), and beneficial to latent decision boundaries (ASER \cite{shim2021online}). Meanwhile, the memory buffer can be dynamically managed, such as by removing less important training samples (CLIB \cite{koh2021online}), editing the old training samples (GMED \cite{jin2021gradient}) or their distributions (DRO \cite{wang2022improving}) to be more likely forgotten, and retrieving the old training samples that are susceptible to interference (MIR \cite{aljundi2019mir}, DVC \cite{gu2022not}).
To adaptively exploit the memory buffer, representative strategies include using additional parameters to construct task-specific features (CTN \cite{pham2020contextual}) or calibrate the network (NCCL \cite{yin2021mitigating}), performing knowledge distillation (ILOS \cite{he2020incremental}, DER \cite{buzzega2020dark}), evolving prototypes in feature space (CoPE \cite{de2021continual}),
improving representations with contrastive learning (SCR \cite{mai2021supervised}, DVC \cite{gu2022not}, CVT \cite{wang2022online}), maximizing mutual information between input data and their label predictions (OCM \cite{guo2022online}), using asymmetric cross-entropy (ER-ACE \cite{caccia2021new}) or constrained gradient directions (GEM \cite{lopez2017gradient_gem}, A-GEM \cite{chaudhry2018efficient_agem}) of the old and new training samples, repeated rehearsal with data augmentation (RAR \cite{zhang2022simple}), properly adjusting the learning rate (PoLRS \cite{cai2021online}, CLIB \cite{koh2021online}),
training from scratch with the memory buffer (GDumb \cite{prabhu2020gdumb}), etc.

In addition, there are some OCL methods without using experience replay, such as by distilling knowledge from the current batch of training samples (BLD \cite{fini2020online}), and using a pre-trained backbone for orthogonal gradient projection (AOP \cite{guo2022adaptive}) or feature replay (CV \cite{he2022online}).

\section{Task Specificity in Application}\label{Sec.6_Task}
In this section, we will discuss the task-specific challenges in various practical applications of continual learning.

\subsection{Object Detection}
A typical extension of continual learning for object detection is called \emph{Incremental Object Detection} (IOD), where the training samples annotated with different classes are introduced in sequence, and the model needs to correctly locate and identify the objects belonging to the previously learned classes.
Unlike visual classification with only one object appearing in each training sample, object detection usually has multiple objects belonging to the old and new classes appearing together.
Such \emph{co-occurrence} poses an additional challenge for IOD, where old classes are marked as the background (i.e., unannotated) when learning new classes, thus exacerbating catastrophic forgetting.
On the other hand, this makes knowledge distillation a naturally powerful strategy for IOD, since old class objects can be obtained from new training samples to constrain the differences in responses between the old and new models.

As an early work, ILOD \cite{shmelkov2017incremental_ilod} distills the responses for old classes to prevent catastrophic forgetting on Fast R-CNN \cite{girshick2015fast}, followed by RKT \cite{ramakrishnan2020relationship} that further distills the relation of co-occurrence in selected proposals.
The idea of knowledge distillation is then introduced to other detection frameworks, such as CenterNet \cite{zhou2019objects} (SID \cite{peng2021sid}), RetinaNet \cite{lin2017focal} (RILOD \cite{li2019rilod}), GFLV1 \cite{li2020generalized} (ERD \cite{feng2022overcoming}) and Faster R-CNN \cite{ren2015faster} (CIFRCN \cite{hao2019end}, Faster ILOD \cite{peng2020faster}, DMC \cite{zhang2020class}, BNC \cite{dong2021bridging}, IOD-ML \cite{kj2021incremental}).
Some approaches exploit the unlabeled in-the-wild data to distill the old and new models into a shared model, in order to bridge potential non co-occurrence (BNC \cite{dong2021bridging}) and achieve a better stability-plasticity trade-off (DMC \cite{zhang2020class}).
To further mitigate the negative effects of knowledge distillation on learning plasticity, IOD-ML \cite{kj2021incremental} adopts meta-learning to reshape parameter gradients into a balanced direction between the old and new classes.

IOD is not only applicable for 2D images, but also for 3D images \cite{zhao2022static} and videos \cite{wang2021wanderlust}. Besides, there are many other related settings, such as incremental few-shot detection \cite{perez2020incremental}, where a pre-trained detection model registers new classes with only a few annotated data; and open world object detection \cite{joseph2021towards}, where the model needs to identify potential objects of unknown classes and register them after receiving the corresponding labels.

\subsection{Semantic Segmentation}
\emph{Continual Semantic Segmentation} (CSS) aims at pixel-wise prediction of classes and learning new classes in addition to the old ones.
Similar to IOD, objects of the old and new classes can appear together.
Some early efforts utilize full annotations of both the old and new classes in continual learning \cite{michieli2019incremental,michieli2021knowledge}.
However, due to the significant expense and time consumption of re-annotating the old classes, more attention has been focused on using annotations of only the new classes, which leads to the old classes being treated as the background (known as the \emph{background shift}) and thus exacerbates catastrophic forgetting.

A common strategy is to distill knowledge adaptively from the old model, which can faithfully distinguish unannotated old classes from the background.
For example, MiB \cite{cermelli2020modeling} calibrates regular cross-entropy (CE) and knowledge distillation (KD) losses of the background pixels with predictions from the old model.
ALIFE \cite{oh2022alife} further improves the calibrated CE and KD with logit regularization, and fine-tunes the classifier with feature replay.
RCIL \cite{zhang2022representation} reparameterizes the network into two parallel branches, where the old branch is frozen for KD between intermediate layers.
SDR \cite{michieli2021continual} and UCD \cite{yang2022uncertainty} introduce contrastive learning into distillation of latent representations, where pixels of the same class are clustered and pixels of different classes are separated.
PLOP \cite{douillard2021plop}, RECALL \cite{maracani2021recall}, SSUL \cite{cha2021ssul}, EM \cite{yan2021framework}, Self-Training \cite{yu2022self}, UCD \cite{yang2022uncertainty} and WILSON \cite{cermelli2022incremental} explicitly use the old model to generate pseudo-labels of the old classes.
Auxiliary data resources such as a web crawler (RECALL-Web \cite{maracani2021recall}), a pre-trained generative model (RECALL-GAN \cite{maracani2021recall}), large amounts of unlabeled data (Self-Training \cite{yu2022self}), and a few old training samples (EM \cite{yan2021framework}, SSUL \cite{cha2021ssul}, ILLR \cite{fortincontinual}) have been exploited to facilitate KD and prevent catastrophic forgetting.
Besides, saliency maps are commonly used to locate unannotated objects in CSS, in response to weak supervision of only image-level annotations (WILSON \cite{cermelli2022incremental}, ILLR \cite{fortincontinual}), as well as defining unknown classes within the background to benefit learning plasticity (SSUL \cite{cha2021ssul}).

In addition to the regular CSS, other relevant settings include unsupervised domain adaptation for CSS \cite{stan2021unsupervised}, where a pre-trained model adapts to the unannotated data of the target domain without using the annotated data of the source domain; incremental few-shot semantic segmentation \cite{shi2022incremental}, which performs CSS with only a few annotated data; incremental instance segmentation \cite{gu2021class}, which incrementally learns new classes but requires individual segmentation for each instance; as well as incremental few-shot instance segmentation \cite{ganea2021incremental}.

\subsection{Conditional Generation}
\emph{Continual Learning for Conditional Generation} (CLCG) is closely related to generative replay.
In general, continual learning of generative models can benefit continual learning of discriminative models through recovering the old data distributions, which motivates much of the work in this direction.
On the other hand, generative replay can also mitigate catastrophic forgetting of the currently trained generative model, which usually requires a frozen copy of the old one.
Since the (conditional) sampling distributions are often provided in both training and testing, CLCG is similar to the setting of TIL, without the challenge of task-identity prediction.

As a generic framework, DGR \cite{shin2017continual_dgr} performs generative replay for continual learning of both discriminative and generative models.
Many efforts focus on continual learning of conditional GANs, such as by weight regularization (ORDisCo \cite{wang2021ordisco}, \cite{seff2017continual}), generative replay (MeRGANs \cite{wu2018memory_mrgan}, L-VAEGAN \cite{ye2020learning}), knowledge distillation (LifelongGAN \cite{zhai2019lifelong_gan}, Hyper-LifelongGAN \cite{zhai2021hyper}), parameter allocation (PiggybackGAN \cite{zhai2020piggyback}, DGMa/DGMw \cite{ostapenko2019learning_dgmw}, TMNs \cite{wang2021triple}), model decomposition (Hyper-LifelongGAN \cite{zhai2021hyper}, FILIT \cite{chen2022few}, GAN-Memory \cite{cong2020gan}), etc.
Other work performs CLCG with VAE, which often relies on a multi-head architecture with encoder expansion, as well as weight regularization (VCL \cite{nguyen2018variational}) or generative replay (CURL \cite{rao2019continual}, VASE \cite{achille2018life}) to overcome catastrophic forgetting.
BooVAE \cite{egorov2021boovae} instead adopts a static architecture through incorporating each task into an additive aggregated posterior and using it as the prior for the next task.

\subsection{Reinforcement Learning}
\emph{Continual Reinforcement Learning} (CRL) is required to address dynamic data distributions between and within tasks (corresponding to the settings of TIL and TFCL, respectively), as states, actions, and environments interact and change over time.
Various types of continual learning strategies have been shown to work not only for visual classification but also for reinforcement learning (RL), such as EWC \cite{kirkpatrick2017overcoming}, MAS \cite{aljundi2018memory_mas}, VCL \cite{nguyen2018variational}, P\&C \cite{schwarz2018progress}, AFEC \cite{wang2021afec}, UCL \cite{ahn2019uncertainty}, AGS-CL \cite{jung2020continual}, CPR \cite{cha2020cpr}, Progressive Networks \cite{rusu2016progressive}, PackNet \cite{mallya2018packnet}, ER, A-GEM \cite{chaudhry2018efficient_agem}, etc.
With respect to adaptive strategies, bio-inspired weight regularization (Benna-Fusi \cite{kaplanis2018continual}) and function regularization (PC \cite{kaplanis2019policy}) are proposed to mitigate within-task catastrophic forgetting. To cope with multiple tasks, OWL \cite{kessler2022same} implements EWC \cite{kirkpatrick2017overcoming} in a shared feature extractor of separate output heads, and adopts multi-armed bandit for task-identity prediction.
An empirical study \cite{isele2018selective} evaluates different principles of selecting old experiences for replay, with matching training distributions usually performing best.
CLEAR \cite{rolnick2019experience} mixes off-policy learning from old experiences and on-policy learning from novel experiences, with behavioral cloning to further promote stability.
MTR \cite{kaplanis2020continual} employs a cascade of interacting sub-buffers to accumulate experiences at different timescales.
LPG-FTW \cite{mendez2020lifelong} instead factorizes a policy gradient model into task-sharing and task-specific parameters.
ClonEx-SAC \cite{wolczyk2022disentangling} investigates the effects of actor, critic, exploration and experience replay on knowledge transfer and provides a set of general recommendations.
COMPS \cite{berseth2021comps} considers continual meta-training of RL, using behavioral cloning to quickly learn a new task from the knowledge of previously-learned incremental tasks.

A distinctive feature of CRL is its high diversity of applications and benchmarks, including continuous control \cite{kaplanis2019policy,lu2019adaptive,wolczyk2021continual,mendez2020lifelong}, maze navigation \cite{rusu2016progressive, lomonaco2020continual}, video games (Atari \cite{kirkpatrick2017overcoming,rusu2016progressive,wang2021afec,ahn2019uncertainty,jung2020continual,cha2020cpr}, StarCraft \cite{schwarz2018towards}, Minecraft \cite{tessler2017deep}, Hanabi \cite{nekoei2021continuous}), etc., varying widely in task type and computation overhead \cite{wolczyk2021continual}. A systematic comparison of CRL methods in various contexts is a promising future work.

\subsection{Natural Language Processing}
Continual learning in \emph{Natural Language Processing} (NLP) has received increasing attentions, which differs from continual learning in visual domains for two aspects: (1) the greater efforts in pre-training, including both upstream \cite{sun2020ernie,han2021econet,jang2021towards} and downstream \cite{wu2021pretrained,ermis2022memory} continual learning; and (2) the specialized architecture of transformer.
As for typical settings, continual learning in NLP covers a wide range, including DIL, TIL, CIL, OCL and CPT \cite{ke2022continual_review}, and accordingly extends the application of representative continual learning strategies, such as weight regularization (RMR-DSE\cite{li2022overcoming}, SRC \cite{liu2019continual}), knowledge distillation (ExtendNER\cite{monaikul2021continual}, CFID \cite{li2022continual}, CID \cite{liu2021lifelong_cid}, PAGeR \cite{varshney2022prompt}, LFPT5 \cite{qin2021lfpt5}, DnR \cite{sun2020distill}, CL-NMT \cite{cao2021continual}, COKD \cite{shao2022overcoming}), experience replay (CFID \cite{li2022continual}, CID \cite{liu2021lifelong_cid}, ELLE \cite{qin2022elle}, IDBR \cite{huang2021continual}, MBPA++ \cite{de2019episodic}, Meta-MBPA++ \cite{wang2020efficient}, EMAR \cite{xu2020continual}, DnR \cite{sun2020distill}, ARPER \cite{mi2020continual}, Total Recall \cite{li2021total}), generative replay (PAGeR \cite{varshney2022prompt}, LAMOL \cite{sun2019lamol}, ACM \cite{zhang2022continual}, NER \cite{wang2022few}), parameter allocation (TPEM \cite{geng2021continual}), modular network (ProgModel \cite{shen2019progressive}), meta-learning (Meta-MBPA++ \cite{wang2020efficient}, MeLL \cite{wang2021mell}, CML \cite{wu2021curriculum}), etc.

On the other hand, the specialized architecture of the (pre-trained) transformer has derived multiple NLP-adaptive techniques. For example, a pre-trained transformer can effectively accommodate downstream tasks by learning only a few task-specific parameters, including \emph{adaptor-tuning} \cite{houlsby2019parameter} with inserted fully-connected layers (CTR \cite{ke2021achieving}, B-CL \cite{ke2021adapting}, CLASSIC \cite{ke2020continual}, CPT \cite{ke2022continual}, CLIF \cite{jin2021learn}, AdapterCL \cite{madotto2021continual}, ACM \cite{zhang2022continual}, ADA \cite{ermis2022memory}), and \emph{prompt-tuning} \cite{lester2021power} with trainable prompt tokens (C-PT \cite{zhu2022continual}, LFPT5 \cite{qin2021lfpt5}, EMP \cite{liu2022incremental}).
Task specificity can also be provided by an extra short text that describes the core concept of each task, i.e., \emph{instruction} \cite{efrat2020turking}, such as PAGeR \cite{varshney2022prompt}, ConTinTin \cite{yin2022contintin} and ENTAILMENT \cite{xia2021incremental}.
In particular, these techniques are increasingly being used in visual domains due to the growing interest in pre-training and vision transformer \cite{wang2022learning_l2p,wang2022dualprompt,wang2022continual_lifelongvit,wang2022sprompts,cong2020gan,ermis2022memory,wang2022online}.

The applications of continual learning in NLP are also highly diverse, such as dialogue system \cite{wang2019incremental,madotto2021continual,geishauser2022dynamic,geng2021continual,mi2020continual,liu2021lifelong,liu2021lifelong}, text classification \cite{pasunuru2021continual,huang2021continual,xia2021incremental,de2019episodic}, sentence generation \cite{zhang2022continual,li2022overcoming,mi2020continual}, relation learning \cite{xu2020continual,qin2022continual,wu2021curriculum,ren2020two}, neural machine translation \cite{gu2020investigating,cao2021continual,garcia2021towards,shao2022overcoming}, named entity recognition \cite{wang2022few,monaikul2021continual,qin2021lfpt5}, etc.
Some other work considers multimodality of vision and language for continual pre-training \cite{cossu2022continual,yan2022generative} or downstream tasks \cite{greco2019psycholinguistics,del2020ratt,srinivasan2022climb}.
We refer readers to their original papers for technical details.

\subsection{Beyond Task Performance}
Continual learning can benefit many considerations beyond task performance, such as efficiency, privacy and robustness.
A major purpose of continual learning is to avoid retraining all old training samples and thus improve \textbf{resource efficiency} of model updates, which is not only applicable to learning multiple incremental tasks, but also important for regular single-task training.
Due to the nature of gradient-based optimization, a network tends to ``forget'' the observed training samples and thus requires repetitive training to capture a distribution, especially for some hard examples \cite{hadsell2020embracing,arpit2017closer,chang2017active}.
Recent work has shown that the one-pass performance of visual classification can be largely improved by experience replay of hard examples \cite{hu2021one} or orthogonal gradient projection \cite{min2022one}.
Similarly, resolving within-task catastrophic forgetting can facilitate reinforcement learning \cite{kaplanis2018continual,kaplanis2019policy} and stabilize the training of GANs \cite{thanh2020catastrophic,lee2021infomax}.

Meanwhile, continual learning is relevant to two important directions of \textbf{privacy protection}. The first is \emph{Federated Learning} \cite{mcmahan2017communication,li2020federated}, where the server and clients are not allowed to communicate with data. A typical scenario is that the server aggregates the locally trained parameters from multiple clients into a single model and then sends it back. As the incremental data collected by clients is dynamic and variable, federated learning needs to overcome catastrophic forgetting and facilitate knowledge transfer across clients, i.e., federated continual learning.
To achieve these aims, FedWeIT \cite{yoon2021federated} decomposes network parameters into global federated parameters and sparse task-specific parameters, with the latter selectively aggregated.
FedSpeech \cite{jiang2021fedspeech} adopts a set of gradual pruning masks for parameter allocation and a set of selective masks to reuse old knowledge.
GLFC \cite{dong2022federated} overcomes catastrophic forgetting through knowledge distillation of a few old training samples, and alleviates class imbalance through gradient compensation.
CFeD \cite{macontinual} instead performs knowledge distillation with auxillary unlabeled data.

The second is \emph{Machine Unlearning}, which aims to eliminate the influence of specific training samples when their access is lost, without affecting other knowledge. Representative methods in this direction are closely related to continual learning, such as learning separate models with subsets of training samples \cite{bourtoule2021machine}, utilizing historical parameters and gradients \cite{wu2020deltagrad}, removing old knowledge from parameters with Fisher information matrix \cite{golatkar2020eternal}, adding adaptive parameters on a pre-trained model \cite{golatkar2021mixed}, etc.
In the context of their intersection, retaining all old knowledge for continual learning may suffer from data leakage and privacy invasion, especially regarding data-level replay.
Mnemonic Code \cite{shibata2021learning} embeds a class-specific code when learning each class, enabling to selectively forget them through discarding the corresponding codes.
LIRF \cite{ye2022learning} designs a distillation framework to remove specific old knowledge and store it in a pruned lightweight network for selective recovery.

As a strategy for adapting to variable inputs, continual learning can assist a robust model to eliminate or resist external disturbances.
For example, PIGWM \cite{zhou2021image} learns adaptive mapping functions for image de-raining and overcomes catastrophic forgetting by weight regularization.
NACL \cite{rostami2021detection} enables face antispoofing systems to detect potential novel attacks and remember the old ones with experience replay.
To cope with noisy labeled incremental data, SPR \cite{kim2021continual} obtains self-supervised representations to purify a memory buffer and then performs supervised fine-tuning.
In fact, \textbf{robustness} and continual learning are intrinsically linked, as they correspond to generalizability in the spatial and temporal dimensions, respectively.
Many ideas for improving robustness to adversarial examples have been used to improve continual learning, such as flat minima \cite{mirzadeh2020understanding,buzzega2020dark}, model ensemble \cite{wang2022coscl}, Lipschitz continuity \cite{bonicelli2022effectiveness} and adversarial training \cite{yu2022continual}.
Subsequent work could further interconnect excellent ideas from both fields, e.g., designing particular algorithms and training paradigms to actively ``forget'' \cite{wang2021afec} external disturbances.

\section{Discussion}\label{Sec.7_Discussion}
In this section, we present an in-depth discussion of continual learning, including the current trends, cross-directional prospects and interdisciplinary connections with neuroscience.

\subsection{Observation of Current Trend}
As continual learning is directly affected by catastrophic forgetting, previous efforts seek to address this problem by promoting memory stability of the old knowledge.
However, recent work has increasingly focused on facilitating learning plasticity and inter-task generalizability.
This essentially advances the understanding of continual learning:
a desirable solution requires a proper balance between the old and new tasks, with adequate generalizability to accommodate their distributional differences.

To promote learning plasticity on the basis of memory stability, emergent strategies include renormalization of the old and new task solutions \cite{wang2021afec,schwarz2018progress,lin2022towards,hou2018lifelong}, balanced exploitation of the old and new training samples \cite{hou2019learning_lucir,wu2019large_bic,zhao2020maintaining,ahn2021ss,wang2022continual_dri,hu2021distilling}, reserving space for subsequent tasks \cite{ahn2019uncertainty,jung2020continual,zhou2022forward}, etc.
On the other hand, solution generalizability can be explicitly improved by converging to a flat loss landscape, where representative strategies include optimizing the flatness metric \cite{shi2021overcoming,mehta2021empirical,mirzadeh2020understanding,deng2021flattening,mirzadeh2020linear}, constructing an ensemble model at either spatial scale \cite{cha2020cpr,wang2022coscl} or temporal scale \cite{buzzega2020dark,boschini2022class}, and obtaining well-distributed representations \cite{ramasesh2021effect,mehta2021empirical,hu2021well,pham2021dualnet,cha2021co2l,madaan2021representational}.
In particular, since self-supervised and pre-trained representations are naturally more robust to catastrophic forgetting \cite{hu2021well,ramasesh2021effect,madaan2021representational,mehta2021empirical}, creating, improving and exploiting such representational advantages has become a promising direction.

We also observe that the applications of continual learning are becoming more diverse and widespread.
In addition to various scenarios of visual classification, current extensions of continual learning have covered many other visual domains, as well as other areas such as RL, NLP and ethic considerations.
We present only some representative applications in the above two sections, with other more specialized and cross-cutting scenarios to be explored, such as robotics \cite{lesort2020continual,churamani2020continual,ayub2022few}, graph learning \cite{wang2022lifelong,kimdygrain,zhang2022cglb}, bioimaging \cite{zhang2022epicker}, etc.
Notably, existing work on applications has focused on providing basic benchmarks and baseline approaches.
Future work could develop more specialized approaches to obtain stronger performance, or evaluate the generality of current approaches in different applications.

\subsection{Cross-Directional Prospect}
Continual learning demonstrates vigorous vitality, as most of the state-of-the-art AI models require flexible and efficient updates, and their advances have contributed to the development of continual learning.
Here, we discuss some attractive intersections of continual learning with other topics of the broad AI community:

\emph{Diffusion Model} \cite{song2020score,dhariwal2021diffusion,bao2021analytic,lu2022dpm-solver} is a rising state-of-the-art generative model, which constructs a Markov chain of discrete steps to progressively add random noise for the input and learns to gradually remove the noise to restore the original data distribution. This provides a new target for continual learning of generative models, and its outstanding performance in conditional generation can facilitate the efficacy of generative replay.

\emph{Neural Compression} \cite{mentzer2020high,agustsson2017soft,wang2022compress} instead expliots generative models to optimize the storage overhead of input data without much distortion, which can benefit both efficacy and efficiency of experience replay \cite{wang2021memory}.
A promising direction is to learn the mapping function of data compression adaptively from the distribution of incremental data, in order to obtain a better compression rate.

\emph{Foundation Model}, such as BERT \cite{devlin2018bert}, GPT \cite{brown2020language}, CLIP \cite{radford2021learning}, etc., relies on large-scale pre-training to acquire impressive results in downstream tasks \cite{bommasani2021opportunities,han2021pre}.
The pre-training data is usually huge in volume and collected incrementally, creating urgent demands for efficient updates. Meanwhile, increasing the scale of pre-training would facilitate knowledge transfer and mitigate catastrophic forgetting for downstream continual learning \cite{ramasesh2021effect,mehta2021empirical}.

\emph{Vision Transformer} \cite{liu2021swin,khan2022transformers} extends the success of transformer-based architectures in NLP to numerous visual domains. This requires specialized designs to overcome catastrophic forgetting \cite{wang2022continual_lifelongvit,wang2022online} while providing new insights for maintaining task specificity in continual learning \cite{wang2022learning_l2p,wang2022dualprompt,wang2022sprompts}. Similarly, specialized techniques for continual learning in NLP can serve as a good reference.

\emph{Embodied AI}~\cite{franklin1997autonomous, chrisley2003embodied, duan2022survey}, an emerging paradigm shift from the era of ``internet AI'', aims to enable AI algorithms and agents to learn through interactions with their environments, no longer from datasets of images, videos or texts curated primarily from the internet.
Of important, the study of general continual learning helps the embodied agents to learn from an egocentric perception similar to humans, and provides a unique opportunity for researchers to pursue the essence of lifelong learning by observing the same person in a long time span~\cite{wang2021wanderlust,gadre2022continuous}.

\subsection{Connection with Neuroscience}
Inspirations from neuroscience have played an important role in the development of continual learning. As biological learning is naturally on a continual basis \cite{kudithipudi2022biological,hadsell2020embracing}, its underlying mechanisms provide an excellent reference for AI models.
Here, we briefly overview the neurological basis of continual learning strategies, from the levels of synaptic plasticity to regional collaboration.

Biological neural networks are capable of flexibly modulating \textbf{synaptic plasticity} in response to dynamic inputs, including
(1) stabilization of previously-learned synaptic changes to overcome subsequent interference \cite{zhang2018active,yang2009stably,hayashi2015labelling}, which motivates the strategies of weight regularization to selectively penalize parameter changes \cite{kirkpatrick2017overcoming,aljundi2018memory_mas,zenke2017continual_si}; (2) number expansion and pruning of functional connections to provide flexibility for new memory formation \cite{richards2017persistence,dong2016inability,bailey2015structural}, which derives the expansion-renormalization paradigm of creating extra space for learning the current task and renormalizing it with the previous one \cite{wang2021afec,schwarz2018progress}; (3) activity-dependent and persistent regulation of synaptic plasticity, i.e., meta-plasticity or ``plasticity of synaptic plasticity'' \cite{abraham1996metaplasticity,abraham2008metaplasticity,clem2008ongoing}, corresponding to the use of meta-learning \cite{finn2017model,javed2019meta,riemer2018learning}; and
(4) inhibitory synapses for excited neurons to reduce the activity of other neurons \cite{cayco2019re,arevian2008activity,inada2017origins,espinoza2018parvalbumin}, which acts similarly to the binary mask for parameter allocation \cite{serra2018overcoming,wang2021triple}.

As for \textbf{regional collaboration}, the \emph{complementary learning system} (CLS) theory \cite{mcclelland1995there,kumaran2016learning} has been widely used to inspire continual learning, which attributes the advantages of biological learning and memory to the complementary functions of hippocampus and neocortex. The hippocampus is responsible for rapid acquisition of separated representations of specific experiences, while the neocortex enables progressive acquisition of structured knowledge for generalization. In particular, the hippocampus can replay neural representations of the previous experiences, so its function is often represented by a memory buffer
\cite{rostami2019complementary,rolnick2019experience,wang2021memory,wang2022continual_dri,boschini2022class} or a generative model \cite{wang2021triple,van2020brain,shin2017continual_dgr,wu2018memory_mrgan,rostami2020generative} to recover the old data distributions.
Instead of using the original data format, however, the hippocampal replay is in a temporally compressed form (e.g., an experience of around $6$ seconds can be compressed into $0.2$ seconds) \cite{kudithipudi2022biological,davidson2009hippocampal,carr2011hippocampal}, which potentially improves resource efficiency. Such biological advantage is further incorporated by replaying compressed data \cite{wang2021memory} or feature representations \cite{ayub2020eec,hayes2020remind}.
In contrast, the neocortex-dependent memories are typically long-term and generalizable, where the representations can be progressively acquired in an unsupervised fashion \cite{doya1999computations,doya2000complementary} and exhibit orthogonality \cite{flesch2022orthogonal,xie2022geometry}, consistent with the advantages of self-supervised and pre-trained representations in continual learning \cite{pham2021dualnet,hu2021well,ramasesh2021effect,madaan2021representational,mehta2021empirical}.
Besides, some regions of the biological brain have a modular architecture similar to the mixture of experts \cite{jacobs1991adaptive}, such as the prefrontal cortex (i.e., a neocortex region) regulating sensory information from various cortical modules \cite{tsuda2020modeling,frankland2005organization,ott2019dopamine} and the mushroom body (i.e., a memory center in fruit fly) coordinating multiple parallel compartments to process the sequentially-arrived experiences \cite{cohn2015coordinated,waddell2016neural,modi2020drosophila,aso2016dopaminergic,aso2014mushroom}, corresponding to the use of modular networks in continual learning \cite{wang2022coscl,aljundi2017expert,ramesh2021model}.

\section{Conclusion}\label{Sec.8_Conclusion}
In this work, we present an up-to-date and comprehensive survey of continual learning, bridging the latest advances in theory, method and application. We summarize both general objectives and particular challenges in this field, with an extensive analysis of how representative strategies address them.
Encouragingly, we observe a growing and widespread interest in continual learning from the broad AI community, bringing novel understandings, diversified applications and cross-directional opportunities.
Based on such a holistic perspective, we expect the development of continual learning to eventually empower AI systems with human-like adaptability, responding flexibly to real-world dynamics and evolving themselves in a lifelong manner.

\ifCLASSOPTIONcompsoc

  \section*{Acknowledgments}
\else

  \section*{Acknowledgment}
\fi
This work was supported by the National Key Research and Development Program of China (2020AAA0106000, 2020AAA0104304, 2020AAA0106302, 2021YFB2701000), NSFC Projects (Nos. 62061136001, 62106123, 62076147, U19B2034, U1811461, U19A2081, 61972224), BNRist (BNR2022RC01006), Tsinghua Institute for Guo Qiang, Tsinghua-OPPO Joint Research Center for Future Terminal Technology, and the High Performance Computing Center, Tsinghua University.

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

\clearpage
\bibliographystyle{ieee_fullname}

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)