1. A method of processing an image for detecting an abnormal area from an intraluminal image, the method comprising: generating a reference surface indicating a reference value of each of a plurality of color elements of pixels constituting an intraluminal image at a pixel position of the each color element by performing a morphology process using pixel values of the color elements; and detecting an abnormal area from the intraluminal image based on a difference for each of the color elements between of the pixel value of each pixel and a reference value on the reference surface, the reference value on the reference surface corresponding to a position of the each pixel; and dividing the intraluminal image into a plurality of areas based on edge information of the intraluminal image; wherein generating the reference surface is performed by performing the morphology process in a three-dimensional space defined by the pixel position and the pixel value, and wherein the generating generates the reference surface for each of the color elements for each of the areas by performing the morphology process for the each of the areas; wherein detecting an abnormal area is performed by comparing the difference for one color element with the difference for another color element, and wherein the detecting detects an abnormal area for each of the areas based on a difference for each of the color elements between the pixel value of each pixel positioned inside the area and the reference surface generated for the area; and determining a size of a structure element used in the morphology process for each of the areas based on pixel value information of the inside of the area, or a size of the area and the pixel value information of the inside of the area.