TECHNICAL FIELD

The present subject matter is generally related to emotion recognition, more particularly, but not exclusively, to a method and a system for real time multimodal analysis based emotion recognition.

BACKGROUND

Emotion recognition involves detection of human emotion. With the advancement in technology, human emotions are studied to identify emotion recognition symptoms demonstrated in most neuropsychiatric disorders like bipolar affective disorder and body dysmorphic disorder. It is related as trait marker for schizophrenia and has been demonstrated in various neurological disorders. Similar deficits in negative emotions of anger and disgust are seen in Huntington's disease. Significant efforts have been made in past for emotion recognition and analysing different emotions. Earlier most of the detection work was based on single model using either visual, speech or text. Each of these models performed well in one scenario but failed in other scenarios. For example, emotion recognition model on images worked well if facial features were clearly visible but failed when a targeted person tilts his/her face a little or if the lighting conditions become poor. Considering limitations related to each of the visual, speech or text model, the detection work lead to combination of the inputs from these 3 models to form a fused model. Insofar, the fused model is not efficient in detecting Facial Emotion Recognition Deficits (FERD) and at the same time, it is time consuming.

The information disclosed in this background of the disclosure section is for enhancement of understanding of the general background of the invention and should not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person skilled in the art.

SUMMARY

In an embodiment, the present disclosure may relate to a method for real time multimodal analysis for emotion recognition. The method includes segmenting video data of a user into a plurality of video segments, wherein each of the plurality of video segments are identified by a video segment identifier. In the next step, a plurality of visual features, a plurality of voice features and a plurality of text features are extracted from the plurality of video segments. The plurality of visual features, the plurality of voice features, and the plurality of text features are identified by the corresponding video segment identifier. The text features are extracted from voice data in the video data of a user. Further, autocorrelation values among each of the plurality of visual features, the plurality of voice features, and the plurality of text features is determined. In the following step, each of the plurality of visual features, the plurality of voice features and the plurality of text features is aligned based on the video segment identifier and the autocorrelation values to obtain a plurality of aligned multimodal features. One of two classes of emotions for each of the plurality of aligned multimodal features is determined based on correlation values determined for each of the plurality of aligned multimodal features. The two classes of emotions comprise positive class of emotions, and negative class of emotions. Lastly, the determined emotion for each of the plurality of aligned multimodal features is compared with historic multimodal features from a database and emotion of the user is determined at real time based on comparison.

In an embodiment, the present disclosure may relate to a real time multimodal assisting system for multimodal analysis for emotion recognition. The multimodal assisting system may include a processor and a memory communicatively coupled to the processor, wherein the memory stores processor-executable instructions, which on execution, cause the processor to segment video data of a user into a plurality of video segments. Each of the plurality of video segments are identified by a video segment identifier. In the next step, the processor extracts a plurality of visual features, a plurality of voice features and a plurality of text features from the plurality of video segments. The plurality of visual features, the plurality of voice features, and the plurality of text features are identified by the corresponding video segment identifier. The text features are extracted from voice data in the video data of a user. The processor determines autocorrelation values among each of the plurality of visual features, the plurality of voice features, and the plurality of text features. In the following step, the processor aligns each of the plurality of visual features, the plurality of voice features and the plurality of text features based on the video segment identifier and the autocorrelation values to obtain a plurality of aligned multimodal features. In the next step, the processor determines one of two classes of emotions for each of the plurality of aligned multimodal features based on correlation values determined for each of the plurality of aligned multimodal features. The two classes of emotions comprise positive class of emotions, and negative class of emotions. Lastly, the processor compares the determined emotion for each of the plurality of aligned multimodal features with historic multimodal features from a database, and determines emotion of the user at real time based on comparison.

In an embodiment, the present disclosure may relate to a non-transitory computer readable medium including instructions stored thereon that when processed by at least one processor cause a multimodal assisting system to perform operations comprising segmenting video data of a user into a plurality of video segments, wherein each of the plurality of video segments are identified by a video segment identifier. In the next step, a plurality of visual features, a plurality of voice features and a plurality of text features is extracted from the plurality of video segments. The plurality of visual features, the plurality of voice features, and the plurality of text features are identified by the corresponding video segment identifier. The text features are extracted from voice data in the video data of a user. Autocorrelation values among each of the plurality of visual features, the plurality of voice features, and the plurality of text features is determined. In the following step, each of the plurality of visual features, the plurality of voice features and the plurality of text features is aligned based on the video segment identifier and the autocorrelation values to obtain a plurality of aligned multimodal features. One of two classes of emotions for each of the plurality of aligned multimodal features is determined based on correlation values determined for each of the plurality of aligned multimodal features. The two classes of emotions comprise positive class of emotions, and negative class of emotions. Lastly, the determined emotion for each of the plurality of aligned multimodal features is compared with historic multimodal features from a database and emotion of the user is determined at real time based on comparison.

The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description.

