1. A moving amount calculation system characterized by comprising: a camera which is mounted on a moving body and senses an image at a first viewpoint and at a second viewpoint different from the first viewpoint as the moving body moves; a moving data measurement unit which measures a moving amount of the moving body from the first viewpoint to the second viewpoint; and an arithmetic device which updates, in accordance with the image sensed by said camera, the moving amount measured by said moving data measurement unit, said arithmetic device comprising: a predicted image generation unit which generates a second viewpoint predicted image as a predicted image at the second viewpoint based on a first viewpoint image as an image sensed at the first viewpoint by said camera and the moving amount measured by said moving data measurement unit; an evaluation value calculation unit which calculates an evaluation value representing coincidence between the second viewpoint predicted image generated by said predicted image generation unit and a second viewpoint image as an image sensed at the second viewpoint by said camera; and a moving amount updating unit which updates the moving amount based on the evaluation value calculated by said evaluation value calculation unit, the moving amount including a horizontal component moving amount which is, out of the moving amount measured by said moving data measurement unit, a component on a plane of movement of the moving body in a direction perpendicular to a moving direction of the moving body that moves straight ahead.