In this work, inspired by the desiderata of robust multi-task and representation learning, we first propose a new framework for task-robust bilevel optimization. Our framework builds off of the existing bilevel optimization framework and adds an extra maximization step for robustness. Our new framework provides full flexibility and generality, allowing each task to have a different loss function and inner optimization procedure. Next, we proposed MORBiT, a single-loop gradient descent-ascent based algorithm for finding an optimal solution to our proposed framework. Finally, we show theoretically, that MORBiT converges at a rate of \(O(K^{-2/5})\) . As special cases, MORBiT reduces to TR-MAML [1]} and TTSA [2]}, and our theory recovers the convergence results in those two works. We hope that our framework inspires future algorithms for robust bilevel optimization and that others find novel applications of our ideas.
