1. A system for controlling an autonomous vehicle, comprising: one or more cameras for obtaining image data of neighboring vehicles in a proximity of the autonomous vehicle; a processor and a non-transitory data storage on which is stored computer code which, when executed on the processor, causes the autonomous driving system to identify, based on comparing the obtained image data with stored sample image data for a plurality of vehicles of particular makes and models, physical characteristics of the neighboring vehicles, wherein the physical characteristics include: classes, makes, models, and trims of the neighboring vehicles; colors of the neighboring vehicles; and exposed cargo being carried by the neighboring vehicles; means for mapping the identified physical characteristics into predicted potential on-road events in the proximity of the autonomous vehicle; means for determining driving decisions based on the predicted potential on-road events; and vehicle controls that implement the determined driving decisions, wherein the means for mapping the identified physical characteristics into predictions includes a database that: associates classes, makes, models, and trims of vehicles with respective expected driving patterns for vehicles of those makes, models, and trims; associates colors of vehicles with respective expected driving patterns for vehicles of those colors; and associates certain vehicle makes and models with aggressive driving patterns; wherein the respective expected driving patterns are further defined by road type and road condition; wherein the means for determining driving decisions includes software that adopts a defensive driving profile upon a determination that a vehicle having an aggressive driving pattern is in the proximity of the autonomous vehicle; wherein the defensive driving profile is a driving profile that accounts for neighboring vehicles that will distract other drivers and a prediction of how the other drivers will drive in response to the distraction.