1. A system for controlling an autonomous vehicle, comprising: one or more cameras for obtaining image data of neighboring vehicles in a proximity of the autonomous vehicle; a processor and a non-transitory data storage on which is stored computer code which, when executed on the processor, causes the autonomous driving system to identify, based on the obtained image data, physical characteristics of the neighboring vehicles, wherein the physical characteristics include: makes and models of the neighboring vehicles; colors of the neighboring vehicles; and exposed cargo being carried by the neighboring vehicles; means for mapping the identified physical characteristics into predicted potential on-road events in the proximity of the autonomous vehicle; means for determining driving decisions based on the predicted potential on-road events; and vehicle controls for implementing the determined driving decisions, wherein the means for mapping the identified physical characteristics into predictions includes a database that: associates makes and models of vehicles with respective expected driving patterns for vehicles of those makes and models; and associates colors of vehicles with respective expected driving patterns for vehicles of those colors, and wherein the respective expected driving patterns are further defined by road type and road condition.