1. A face clustering device comprising: a face detection unit that detects a face included in an image; a face-direction detection unit that detects a direction of the face detected by the face detection unit; a face identification unit that identifies, taking into account the direction of the face detected by the face-direction detection unit, another face with a similar feature and forming a collection of pieces of face information showing a feature of said another face; a unit group setting unit that narrows down, for each collection of pieces of face information formed by the face identification unit, a number of pieces of face information to a predetermined number for each face direction based on the direction of the face detected by the face-direction detection unit and sets each collection of pieces of face information for which the number has been narrowed down as a unit group; and a clustering unit that performs, with the unit group set by the unit group setting unit as a unit, clustering based on pieces of face information included in each unit group, wherein, when the unit group set by the unit group setting unit is expressed as a first unit group and a unit group set based on a k-th (k=1 to N) unit group is expressed as a (k+1)-th unit group, the clustering unit sequentially merges, for k=2 to N, while taking the first unit group, present in each of sections divided by a predetermined first time T 1 as a target, unit groups with a similar feature and sets a second unit group, and while taking the k-th unit group, present in each of sections divided by a predetermined k-th time T k (T k >T k-1 ) as a target, merges unit groups with a similar feature and sets a (k+1)-th unit group.