1. A vision system for a vehicle, said vision system comprising: a camera disposed at a vehicle equipped with said vision system, said camera comprising an image sensor and a lens, wherein, when said camera is disposed at the equipped vehicle, said lens is exposed to the environment exterior the equipped vehicle; wherein said camera has an exterior field of view exterior the equipped vehicle and is operable to capture image data representative of a scene occurring in the field of view of said camera; an image processor operable to process multiple frames of image data captured by said camera while the vehicle is moving; wherein said image processor processes captured image data while the vehicle is moving to detect a blob in a first frame of captured image data; wherein, responsive to detecting, via processing the first frame of captured image data, the blob in the first frame of captured image data, said vision system determines at a first confidence level that the blob detected in the first frame of captured image data is indicative of a contaminant; wherein, responsive to said image processor determining, with the first confidence level being at least 80%, that the blob detected in the first frame of captured image data is indicative of a contaminant, said image processor adjusts processing when processing subsequent frames of captured image data; wherein adjustment of processing when processing the subsequent frames of captured image data comprises detecting, in a second frame of captured image data, image data associated with the blob detected in the first frame of captured image data; and wherein, responsive to said image processor determining, via processing of the second frame of captured image data, at a second confidence level of at least 80%, that the blob detected in the second frame of captured image data is indicative of a contaminant, said image processor determines that the blob detected in the second frame of captured image data is representative of a contaminant at said lens of said camera.