1. A color sorting machine for sorting out a conforming granular object from a non-conforming granular object by color, comprising: a primary sorting unit including transport means for primary sorting for letting granular objects serving as a material flow down in the form of a belt, an optical detection unit provided along a fall path for a granular object falling down from the transport means, and ejector means provided below the optical detection unit for removing primary non-conforming grains from among normal granular objects; a secondary sorting unit including transport means for secondary sorting for letting the primary non-conforming grains removed by the primary sorting unit flow down in a plurality of rows, an optical detection unit provided along a fall path for a granular object falling down from the transport means, and ejector means provided below the optical detection unit for removing a secondary non-conforming grain from the primary non-conforming grains; and ejector operation determination means including an overlap processing unit and a valve assignment unit, wherein the optical detection units in the primary sorting unit and secondary sorting unit include: binarization means for binarizing an image of a granular object obtained through imaging; defect detection means for extracting an image of a defective part of the granular object from the image obtained by the binarization means; outline detection means for extracting an image of an outline part of the granular object from the image obtained by the binarization means; erosion means for eroding the image of the outline part extracted by the outline detection means; dilation means for dilating the image of the defective part obtained by the defect detection means; and merging means for merging the eroded image of the outline part obtained by the erosion means and the dilated image of the defective part obtained by the dilation means, and the ejector operation determination means causes the ejector means for the primary sorting unit to execute an air injection operation that simultaneously operates multiple adjacent air injection nozzle ports at positions corresponding to a position of a primary non-conforming grain and causes the ejector means for the secondary sorting unit to execute an air injection operation that operates only a single air injection nozzle port at a position corresponding to a position of a secondary non-conforming grain, on the basis of an image obtained by the merging means, wherein the erosion means performs four-neighbors erosion on the image of the outline part of the granular object at least once for the primary sorting unit and performs transverse erosion on the image of the outline part of the granular object a plurality of times for the secondary sorting unit, wherein the primary sorting unit and the secondary sorting unit perform different processes, with the primary sorting unit configured to conduct overlap processing and the secondary sorting unit configured to not perform overlap processing.