In the yellowthird step of the training procedure, we train the final multi-modal experts network. We use the pre-trained experts from previous steps and set the learning rate to be \(0.0001\) . In particular, we use the pre-trained LiDAR expert obtained from the first step and pre-trained camera experts obtained in the second step in the case where LiDAR was disabled. Finally, we use all five settings of enabled/disabled sensors, to fine-tune the entire network in an end-to-end fashion. In particular, we lock the gating networks (will not update the parameters) and train the whole network. The test results of the obtained final network are shown in Fig. REF  and Table REF , where the test performance is measured as the mean squared error(MSE) between the predicted and the reference steering command. The obtained network correctly chooses the useful sensor for driving and accurately predicts the steering command.
Finally, we compare the yellowtest performance of our model with several different approaches. The first three methods listed in Table REF  describe models that use inputs from single modality. They are constructed by one or more feature extractors followed by a fully-connected layer to make the final prediction. yellowThe baseline model is obtained by concatenating the features from all of the experts without any gating mechanism. It is trained end-to-end. NetGated and Sensor Dropout were introduced in [1]} and [2]} respectively. We followed the setups described in their original papers to train the networks. Also note that our approach is orthogonal to the existing network compression methods [3]}, [4]}, [5]}, [6]} since compression schemes are applicable to our approach and can lead to further computational reductions. To make fair comparisons, each network uses the same feature extractor architectures and the gating networks are designed to require significantly less computations than feature extractors, they are all described in the Appendix. We compare MSE between the predicted and actual steering command and the amount of computations, i.e. the number of floating-point operations (FLOPs) needed to perform a single inference. For the multisensory models, their performances are measured in five different scenarios when one of the sensors is disabled. The results are captured in Table REF . The proposed network yellowachieves comparable performance to the baseline model and outperforms the majority of the competitive methods in terms of test loss except for NetGated. However, yellowboth the baseline and NetGated require significantly more computations and they do not have the ability to interpret the importance of each sensor at any given moment. Meanwhile, our method uses almost the same amount of computations as the single modality networks. Therefore multi-modal experts network is superior over other sensor fusion techniques as it can make accurate predictions and handle sensor failures better while at the same time is computationally much more efficient.
We tested our proposed network on the autonomous platform and used the predicted steering angles to steer the vehicle in an indoor environment. The snapshots of the test video are shown in Fig. REF .
<FIGURE>