1. A facial expression recognition apparatus, comprising: an image input unit configured to sequentially input images; a face detection unit configured to detect faces in images obtained by the image input unit; a start determination unit configured to determine whether to start facial expression determination based on facial image information detected by the face detection unit; an acquisition unit configured to acquire reference feature information based on the facial image information detected by the face detection unit, when the start determination unit determines that facial expression determination should be started; and a facial expression determination unit configured to extract feature information from the facial image information detected by the face detection unit, when the start determination unit determines that the facial expression determination should be started, and to determine facial expressions of the detected faces based on the extracted feature information and the reference feature information.