1. An image processing apparatus, comprising: one or more processors configured to decide a movement amount to move a target object, that is a movement target among objects included in a plurality of captured images imaged by an imaging device that includes plurality of optical units that have imaging ranges that overlap with each other in part, on the basis of: a distance from the imaging device to the target object, and an angle from a reference line at a reference position that include a central position of the imaging device to the target object in a manner that the plurality of captured images look like images imaged from the reference position, wherein the imaging device uses all directions as an imaging range, the plurality of captured images corresponding to respective optical units of the plurality of optical units to generate a viewpoint image that is an image that corresponds to a set direction on the basis of the plurality of captured images in which the target object has been moved.