The gating mechanism was explored in the literature in various settings and applications. The first system incorporating this mechanism [1]} was the modular version of a multi-layer supervised network, where many separate networks (experts) were operating on the same input but they were handling different learning sub-tasks.
The gating mechanism has often been used as an enabling technique for scaling algorithms to large data sets and has been flexible enough to adapt to different learning systems, including those with experts realized as SVMs [2]} or neural networks [3]}, [4]}, [5]}, [6]} and those where experts were hierarchically-structured [7]}. The works on conditional computation [3]}, [4]}, [5]} explore the notion that irrelevant or redundant experts need not be activated at all when forming a prediction. Sparsely-gated mixture-of-experts [6]} realizes an extreme version of this concept, where a large number of experts is used, but only a very small subset is activated at each inference time step. None of the gating schemes discussed above was applied to autonomous driving, but they were explored in the context of more traditional AI problems such as language modeling, machine translation, or image classification. Our work falls into the category of conditional computations schemes, but as opposed to existing methods experts do not share the same input. Finally, gating mechanism was also used for sensor fusion [12]}, [13]} in autonomous driving, where all experts are active. These approaches therefore are not concerned with constraints on the computational budget since they allow wasteful feature extractions from irrelevant or redundant sensors. Our work fundamentally differs from these works as in our case such experts are identified at each inference time step and are deactivated, which leads to computational savings.
