1. A method comprising: obtaining, by a computer system, image data from at least one sensor of a vehicle configured to operate in an autonomous mode in an environment, wherein the at least one sensor is coupled to the vehicle, and wherein the image data includes data indicative of a known structure on which the vehicle is traveling; determining, based on a first portion of the image data, information indicative of an appearance of the known structure in the environment; determining, based on a second portion of the image data, information indicative of an appearance of an unknown object in the environment, wherein the second portion of the image data is different than the first portion of the image data; comparing the information indicative of the appearance of the known structure in the environment with the information indicative of the appearance of the unknown object in the environment so as to determine whether the unknown object is part of the known structure; and providing, by the computer system, instructions to control the vehicle in the autonomous mode based on the comparison.