AGEM blocks on Split CIFAR-100
From https://github.com/ContinualAI/reproducible-continual-learning/issues/16

AGEM blocks in the `sample_from_memory` call when retrieving the next element from the `GroupBalancedInfiniteDataLoader`, which represents the AGEM buffer. 
The process does not always block at the same experience and the problem is not there for other benchmarks like Permuted MNIST.

You can reproduce the problem with this script

```python
import avalanche as avl
from avalanche.evaluation import metrics
import torch

device = torch.device('cuda:0')

benchmark = avl.benchmarks.SplitCIFAR100(17, return_task_id=True, fixed_class_order=list(range(85)))
model = avl.models.SimpleCNN(num_classes=100)
criterion = torch.nn.CrossEntropyLoss()

interactive_logger = avl.logging.InteractiveLogger()

evaluation_plugin = avl.training.plugins.EvaluationPlugin(
    metrics.accuracy_metrics(epoch=True, experience=True, stream=True),
    loggers=[interactive_logger], benchmark=benchmark)

cl_strategy = avl.training.AGEM(
    model, torch.optim.SGD(model.parameters(), lr=1e-3, momentum=0.9), criterion,
    patterns_per_exp=65, sample_size=200,
    train_mb_size=10, train_epochs=1, eval_mb_size=128,
    device=device, evaluator=evaluation_plugin)

for experience in benchmark.train_stream:
    cl_strategy.train(experience)
    res = cl_strategy.eval(benchmark.test_stream)
```
I am currently trying to better investigate the problem but I also wanted to add the issue here for reference
diff --git a/avalanche/training/plugins/agem.py b/avalanche/training/plugins/agem.py
index da352a4d..c6a0b84d 100644
--- a/avalanche/training/plugins/agem.py
+++ b/avalanche/training/plugins/agem.py
@@ -1,3 +1,5 @@
+import warnings
+
 import torch
 from torch.utils.data import random_split
 
@@ -110,10 +112,13 @@ class AGEMPlugin(SupervisedPlugin):
         return next(self.buffer_dliter)
 
     @torch.no_grad()
-    def update_memory(self, dataset, num_workers=4, **kwargs):
+    def update_memory(self, dataset, num_workers=0, **kwargs):
         """
         Update replay memory with patterns from current experience.
         """
+        if num_workers > 0:
+            warnings.warn("Num workers > 0 is known to cause heavy"
+                          "slowdowns in AGEM.")
         removed_els = len(dataset) - self.patterns_per_experience
         if removed_els > 0:
             dataset, _ = random_split(
['avalanche/training/plugins/agem.py']