1. An image processing apparatus for playing back a moving image including a plurality of time-divided frame images so as to reflect a human spatial visual characteristic and temporal visual characteristic, the apparatus comprising: an object extraction unit configured to extract object images from a frame image of interest in the plurality of frame images and to store a flag representing whether or not an operator is paying attention to the extracted object image, wherein the flag corresponds to the extracted object image and is based on an input of line of sight of the operator; an object adaptation time acquisition unit configured to acquire an object adaptation time of each of the extracted object images for which a flag is stored representing that the operator is paying attention, based on a period during which the extracted object is drawn in a predetermined processing period up to the frame of interest; an object adaptation feature amount calculation unit configured to calculate an object adaptation feature amount of each object image based on the acquired object adaptation time, wherein the object adaptation feature amount represents an adaptation weight reflecting the human temporal visual characteristic with respect to the extracted object images; and a color conversion unit configured to perform, for the frame image of interest, color conversion based on the calculated object adaptation feature amount of each object image.