1. A color sorting machine which is an apparatus for sorting out a conforming granular object from a non-conforming granular object by color, comprising: a primary sorting unit including transport means for primary sorting for letting granular objects serving as a material flow down in the form of a belt, an optical detection unit provided along a fall path for a granular object falling down from the transport means, and ejector means provided below the optical detection unit for removing primary non-conforming grains from among normal granular objects; a secondary sorting unit including transport means for secondary sorting for letting the primary non-conforming grains removed by the primary sorting unit flow down in a plurality of rows, an optical detection unit provided along a fall path for a granular object falling down from the transport means, and ejector means provided below the optical detection unit for removing a secondary non-conforming grain from the primary non-conforming grains; and ejector operation determination means, wherein the optical detection units in the primary sorting unit and secondary sorting unit include: binarization means for binarizing an image of a granular object obtained through imaging; defect detection means for extracting an image of a defective part of the granular object from the image obtained by the binarization means; outline detection means for extracting an image of an outline part of the granular object from the image obtained by the binarization means; erosion means for eroding the image of the outline part extracted by the outline detection means; dilation means for dilating the image of the defective part obtained by the defect detection means; and merging means for merging the eroded image of the outline part obtained by the erosion means and the dilated image of the defective part obtained by the dilation means, and the ejector operation determination means brings the ejector means for the primary sorting unit into air injection operation that simultaneously operates an air injection nozzle port corresponding to a position of a non-conforming grain and an adjacent air injection nozzle port and the ejector means for the secondary sorting unit into air injection operation that operates only an air injection nozzle port corresponding to a position of a secondary non-conforming grain, on the basis of an image obtained by the merging means.