Beyond RMSE: Do machine-learned models of road user interaction produce human-like behavior?
Autonomous vehicles use a variety of sensors and machine-learned models to
predict the behavior of surrounding road users. Most of the machine-learned
models in the literature focus on quantitative error metrics like the root mean
square error (RMSE) to learn and report their models' capabilities. This focus
on quantitative error metrics tends to ignore the more important behavioral
aspect of the models, raising the question of whether these models really
predict human-like behavior. Thus, we propose to analyze the output of
machine-learned models much like we would analyze human data in conventional
behavioral research. We introduce quantitative metrics to demonstrate presence
of three different behavioral phenomena in a naturalistic highway driving
dataset: 1) The kinematics-dependence of who passes a merging point first 2)
Lane change by an on-highway vehicle to accommodate an on-ramp vehicle 3) Lane
changes by vehicles on the highway to avoid lead vehicle conflicts. Then, we
analyze the behavior of three machine-learned models using the same metrics.
Even though the models' RMSE value differed, all the models captured the
kinematic-dependent merging behavior but struggled at varying degrees to
capture the more nuanced courtesy lane change and highway lane change behavior.
Additionally, the collision aversion analysis during lane changes showed that
the models struggled to capture the physical aspect of human driving: leaving
adequate gap between the vehicles. Thus, our analysis highlighted the
inadequacy of simple quantitative metrics and the need to take a broader
behavioral perspective when analyzing machine-learned models of human driving
predictions.