(1) Experience Replay.
[1]} suggests that the online learning agent equips a fixed-sized memory to store the learnt examples, then the model repeatedly replays the samples from the memory to alleviate catastrophic forgetting. In particular, they adopt the reservoir sampling [2]} strategy to draw the samples from the memory for model training, such strategy ensures the sampling result being equivalent to having uniform sampling from the stream data without knowing the sequence length.
Follow-up works [3]}, [4]}, [5]} assume that every training data has different importance, and remembering a few critical samples is enough for keeping the data distribution. Nevertheless, [6]} show that the simple greedy selection (i.e. storing the latest examples) is actually able to achieve comparable performance without any cost to evaluate the importance of samples.
