I managed to create an emotion recognition system that uses dense optical flow on each entire frame. While the accuracy range is within 80-90% with cross-validation, I am aiming to improve the accuracy of the program. There are four emotions: Neutral, happy, surprised, and angry. So far my classifier works pretty well, though it tends to over guess 'neutral' when  the answer is 'happy' or 'surprised'. This tends to happen when the mouth is only slightly opened, but the subject is visible smiling or have mouth opened in shock while the classifier still thinks the mouth is closed.Confusion Matrix for Dense Optical Flow:   [[27 22  0  0]   [ 0 57  1  0]   [ 0 12 60  0]   [ 0  9  3 68]]  Accuracy: 80-90% rangeThere is something I want to try in order to solve this though.I have the ability to get the position of facial landmarks, though I don't know what I can do to turn this information into an effective additional feature I can use that would increase the accuracy. I was thinking of just simply getting the face landmark coordinates at the end of each video, but I feel like that it would not be the solution to differentiate between a closed mouth and slightly opened one (the difference in coordinate values will be small I think and am guessing that machine learning won't notice the difference).I considered the possibility of just simply taking a still image of the subject's mouth and just analyzing that, but rejected it as it vulnerable to factors like lighting and people's appearance, and inconsistent matrix sizes. Plus I want my additional feature to take advantage of facial movement tracking.I was wondering if there is a smart way to implement facial landmark tracking into a feature that would increase the accuracy of my classifier by dealing with my classifier's tendency to over predict the emotion 'neutral'. Any ways I can accomplish that?
Trying to come up with a feature to improve emotion classifier based on facial movement using facial landmarks
machine learning;classification;feature selection;confusion matrix;computer vision