In practice, the networks are trained with mini-batches, therefore the frames outside the current mini-batch are not available. To deal with that, we use a memory bank [1]} to store representations from previous mini-batches. The size of the memory bank is \(N_{mem}\) . After each training step, the current batch of representations are stored in memory bank, replacing the oldest ones. In other words, the memory bank works as a queue of representation. This allows the representation to be regularized by as much pairs as possible, and it is proved to be beneficial to learning good representation [2]}, [1]}.
