In this paper, we consider two scenarios: (a) training data for task \(t\)  is not available after \(t\)  is trained and (b) training data for all tasks is available throughout, but training is incremental per task.
In the first scenario, L2BG performs task-incremental optimization only, whereas in the second one, it performs global optimization across tasks also.
Several large-scale multi-task problems across text, vision and RL domains [1]}, [2]}, [3]}, [4]} fall into the latter scenario.
