To demonstrate the importance of the ideas used in the NACL algorithm, we preform ablative experiments. We considered the single-attack per task scenario and used the PADISI-Face dataset in these experiments.
We first demonstrate the importance of detecting OTDS samples. Consider that OTDS samples are not detected but the model is updated using a binary prediction baseline. This means that in a CL setting, we always store all the testing samples that are identified as PAs during execution, assuming all to be new attack types, and use them to update the model at each time-step. We refer to this approach as No GMM (NG). In a second experiment,
we reported performance of the FR setting when real labels (FRR) are used, i.e., performance in the absence of label pollution. This means that upon identifying the novel attack data points, rather than using the labels predicted by the model, we use the real labels to update the model. Performance results for these setting are summarized in Table REF .
Extremely poor performance of NG, measured in the APCER, demonstrates that detecting OTDS is necessary for PAD in a continual learning setting. We also observe that, when real labels are used, as expected from the previous discussion, FRR converges to an upper-bound for NACL, close to the visualized JT performance in Figure REF (a). This observation suggests a future direction for improving our algorithm is tackling the challenge of label pollution [1]}. We can also conclude that to mitigate catastrophic forgetting further, a larger buffer size should be used.
<TABLE>