As for the switching methods, the results are quite interesting. Training an ensemble and using disagreement as a stand in for "the predictor is uncertain here" [1]} fails to identify many of the cases that it should, because the members agree even when making the wrong prediction. Switching based on observing the real human online to take actions that the predictor is assigning low probability to is very reliable, but it does introduce a significant switching delay because the robot has to observe enough such "low probability" human actions. Surprisingly, a simple classifier that we train to label predictions as good or bad based on the predictor's performance on training data is also very reliable. The classifier is not meant to quantify uncertainty or detect out of distribution issues, but it implicitly does that by learning to pick up on features of predicted trajectories that are suggestive of something having gone wrong, like going off the road or not stopping at a stop sign.
