1. An image processing device comprising: an image sequence acquisition section that acquires an input image sequence that includes first to N-th (N is an integer equal to or larger than 2) images as constituent images; and a processing section that performs an image summarization process that deletes some of the first to Nth images included in the input image sequence acquired by the image sequence acquisition section to generate a summary image sequence, the processing section selecting an s-th (s is an integer that satisfies 0≦s≦N+1) image included in the input image sequence to be a provisional summary image, selecting a t-th (t is an integer that satisfies 0≦t≦s−1) image included in the input image sequence to be a provisional preceding summary image, selecting a u-th (u is an integer that satisfies t<u<s) image included in the input image sequence to be a determination target image, calculating a deletion evaluation value of the determination target image based on deformation information about the provisional summary image and the determination target image, and the deformation information about the provisional preceding summary image and the determination target image, calculating a summarization interval evaluation value G(t, s) that is an evaluation value when (t+1)-th to (s−1)-th images are deleted, based on the deletion evaluation values of the (t+1)-th to (s−1)-th images, and performing the image summarization process based on the summarization interval evaluation value.