1. An image display medium driver comprising: a voltage applying unit that applies a voltage between a pair of substrates of an image display medium that displays an image based on image information, the image display medium including a plurality of groups of colored particles colored in a color which is different for every group, enclosed between the pair of substrates, at least one of the substrates having transparent properties, each group of colored particles moved when the voltage equal to or higher than a threshold voltage in terms of absolute value, that is different for every group; and a controller that determines a substrate on which the colored particles are present for each group of colored particles based on the last image information used for displaying an image, and when the colored particles are present on both of the pair of substrates, controls the voltage applying unit so as to apply a voltage having a smaller absolute value than any threshold voltage of colored particles present on one substrate, and the smaller voltage applied in a direction from the one substrate toward the other substrate side.