The advancements in deep learning [1]} enabled artificial neural networks to invade a diverse set of real-life applications that need to rely on systems capable of learning, among which autonomous driving is an important and still largely unsolved task. Using multiple sensors improves perception abilities, leads to a better understanding of the environment, and consequently increases the safety and reliability of deep-learning-based autonomous platforms (some examples in the context of self-driving cars include [2]}, [3]}, [4]}). Nevertheless, it is highly non-trivial to teach the autonomous vehicle to use the information registered by different sensors potentially capturing various modalities. When there exists an overlap of information as is the case for camera and LiDAR registrations that cover the same or partially overlapping field of view, one needs to ensure the network is evenly trained to use any of the available relevant inputs. Otherwise, the failure of the sensor that the network was preferring at training leads to significant performance degradation. This could be avoided if instead of ignoring the information provided by other relevant sensors, the network would adapt to them as well. Another obstacle when using many sensors is the explosion of computational complexity. Addition of an extra sensor requires the expansion of the network to accommodate an extra feature extractor and increases computational cost.
This largely increases the inference time step and eventually paralyzes vehicle's real-time operation. The objective of this work is to address the above challenges.
