1. A pattern matching apparatus for matching an input pattern with reference patterns by extracting feature points of the input pattern to obtain a feature vector, and comparing the feature vector of the input pattern with the previously stored feature vectors of the reference patterns, comprising: a database for storing information of each reference pattern, the reference pattern information including indices of nearest/farthest neighbor reference patterns, distances between the reference patterns and each of the neighbor reference patterns and direction of the neighbor reference patterns with respect to the reference patterns, wherein the number of the nearest/farthest neighbor reference patterns depends on the features of the reference pattern; a pattern classifier for selecting a reference pattern from the database and calculating distance and direction between the input pattern and the selected reference pattern, to thereby include the reference pattern whose distance from the input pattern is equal to, or less than, a first threshold value in a matching comparison target, and determining whether the selected reference pattern and its neighbor reference patterns are included in the matching comparison target considering the direction of the neighbor reference patterns with respect to the reference pattern, located within the radius of a circle having the point of the selected reference pattern as a center point, if the distance between the input pattern and the selected reference pattern is greater than the first threshold value and the difference between the distance and the first threshold value is equal to or less than a second threshold value, wherein the neighbor reference patterns, whose direction with respect to the selected reference pattern is opposite to the direction of the reference pattern with respect to the input pattern in the matching comparison target, are included in the matching comparison target, and the remaining neighbor reference patterns within the radius are excluded from the matching comparison target; and a pattern matching unit for matching the patterns determined as the matching comparison target by the pattern classifier with the input pattern.