1. A method of driving an electrophoretic display device including a display unit that has a plurality of pixels and an electrophoretic element disposed between substrates forming one pair, the method comprising: setting an area that at least includes a pixel forming an image component that is formed to have a first gray scale and a pixel that is disposed to be adjacent to the pixel forming the contour of the image component and represents a second gray scale as an image removing area and selectively changing the pixels that constitute the image removing area to have the second gray scale.