1. A facial expression recognition apparatus, comprising: an image input unit configured to sequentially input images; a face detection unit configured to detect faces in images obtained by the image input unit; a start determination unit configured to determine whether to start facial expression determination based on facial image information of each of the faces detected by the face detection unit; a first extraction unit configured to extract first feature information from the facial image information when the start determination unit determines that facial expression determination should be started based on the facial image information; a second extraction unit configured to extract second feature information from facial image information of a face detected by the face detection unit after the start determination unit determines that the facial expression determination should be started; and a facial expression determination unit configured to determine a facial expression of the detected face corresponding to the second feature information based on the first and second feature information, wherein at least the facial expression determination unit is implemented by a processor, and wherein the start determination unit determines that facial expression determination should be started when the variation of the features obtained from the facial image information detected in a series of images over a predetermined number of frames or a predetermined time stays within a prescribed range.