After each learning iteration, we can compute the Fisher Information on the new dataset, and add it to the current Fisher [1]}. As a result, this approach stays at a constant memory cost (equalling the number of trainable parameters in the network).
