1. A method of driving an electrophoretic display device including a display unit that has a plurality of pixels and an electrophoretic element disposed between substrates forming one pair, the method comprising: displaying a second gray scale at first pixels in a first area of the display unit and second pixels in a second area of the display unit, the second area surrounding the first area; selectively displaying a first gray scale at the first pixels in the first area of the display unit; and selectively displaying the second gray scale at pixels in an image removing area, the image removing area including the first pixels in the first area and a part of the second pixels that is disposed to be adjacent to the pixel forming the contour of the first area, wherein, during displaying the first gray scale at the first pixels in the first area of the display unit, an area which displays the first gray scale expands beyond the first area to the second area.