1. An apparatus, comprising: a memory; and a processor configured to execute instructions stored in the memory to: define a notch candidate on a boundary of a polygon of mask data, the notch candidate formed of a plurality of line segments, the plurality of line segments including an initial line segment, a final line segment and at least one line segment between the initial line segment and the final line segment, the initial line segment and the final line segment defining adjacent edges of the notch candidate and having an angle therebetween within a defined range; define a direction of each of the plurality of line segments as a direction of travel from the initial line segment to the final line segment; and apply a plurality of conditions to the plurality of line segments and the direction of travel, wherein the notch candidate is a notch when each of the plurality of conditions is satisfied.