1. An underline removal apparatus that removes an underline area from binary image data including the underline area touching a character string, the underline removal apparatus comprising: an underline search processing unit that executes a line template matching process by setting a point on the binary image data as a starting point to set a rectangular line template, tracing pixels included in the line template, and extracting a polyline indicating underline position coordinates; and an underline removal processing unit that uses the polyline to execute a process of obtaining background borderline coordinates between the underline area and a background area and character borderline coordinates between the underline area and the character string obtained by applying an interpolation process to a part in the underline area touching the character string and to execute a process of removing an area surrounded by the background borderline coordinates and the character borderline coordinates from the binary image data.