1. A vision system of a vehicle, said vision system comprising: a camera disposed at a vehicle and having a field of view exterior of the vehicle, said camera capturing frames of image data; a control having an image processor operable to process frames of image data captured by said camera; wherein said image processor is operable, via image processing of frames of captured image data, to detect an object present in the field of view of said camera; wherein, when the vehicle is moving, said control, responsive at least in part to vehicle motion information and image processing of frames of captured image data, detects an object present in the field of view of said camera and determines motion of the detected object relative to the moving vehicle; and wherein said control, responsive at least in part to vehicle motion information and image processing of frames of captured image data, determines the motion of the detected object relative to the moving vehicle by (i) determining corresponding object points in at least two frames of captured image data, (ii) estimating object motion trajectory of the detected object responsive to the determination of corresponding object points and (iii) determining the structure of the detected object along the estimated object motion trajectory.