In our framework, both the testee and tester perform learning. The testee learns how to best conduct a target task \(J_1\)  and the tester learns how to create difficult and meaningful tests. To encourage a created test \(T\)  to be meaningful, the tester trains a model using \(T\)  to perform a target task \(J_2\) . If the model performs well on \(J_2\) , it indicates that \(T\)  is meaningful.
The testee has two sets of learnable parameters: neural architecture and network weights. The tester has three learnable modules: data encoder, test creator, and target-task executor. The learning is organized into three stages. In the first stage, the testee trains its network weights on the training set of task \(J_1\)  with the architecture fixed. In the second stage, the tester trains its data encoder and target-task executor on a created test to perform the target task \(J_2\) , with the test creator fixed. In the third stage, the testee updates its model architecture by minimizing the predictive loss \(L\)  on the test created by the tester; the tester updates its test creator by maximizing \(L\)  and minimizing the loss on the validation set of \(J_2\) . The testee and tester interact on the loss function \(L\)  in an adversarial manner, where the testee minimizes this loss while the tester maximizes this loss.
The three stages are performed jointly end-to-end in a multi-level optimization framework, where a latter stage influences an earlier stage and vice versa. We apply our method for neural architecture search [1]}, [2]}, [3]} in image classification tasks on CIFAR-100, CIFAR-10, and ImageNet [4]}. Our method achieves significant improvement over state-of-the-art baselines.
