To overcome this problem, inspired by multi-task learning using multiple branches to optimize different tasks, we employ an independent branch for each domain. To avoid introduce any additional parameters, we divide the last embedding layer of the model into four non-overlapping heads as in BIER [1]}. In this manner, we could assign one independent embedding subspace to each domain. Specifically, each augmented image can only forward through their corresponding head, and each head's parameters would only be updated by the domain it is responsible to. Empirically, our IDEAL leads to additional gains in performance with this independence design at model structural level.
