FIELD OF THE DISCLOSURE

The present disclosure relates to systems, methods and apparatuses for detecting muscle activity, and in particular, to systems, methods and apparatuses for detecting facial expression according to muscle activity.

BACKGROUND OF THE DISCLOSURE

In some known systems, online activities can use user facial expressions to perform actions for an online activity. For example, in some known systems, the systems may estimate a user's facial expressions so as to determine actions to perform within an online activity. Various algorithms can be used to analyze video feeds provided by some known systems (specifically, to perform facial recognition on frames of video feeds so as to estimate user facial expressions). Such algorithms, however, are less effective when a user engages in virtual reality (VR) activities. Specifically, VR hardware (such as VR helmets, headsets, and/or other apparatuses) can obscure portions of a user's face, making it difficult to detect a user's facial expressions while using the VR hardware.

Thus, a need exists for apparatuses, methods and systems that can accurately and efficiently detect user facial expressions even when the user's face is partially obscured.

SUMMARY OF THE DISCLOSURE

Apparatuses, methods, and systems herein facilitate a rapid, efficient mechanism for facial expression detection according to electromyography (EMG) signals. In some implementations, apparatuses, methods and system herein can detect facial expressions according to EMG signals that can operate without significant latency on mobile devices (including but not limited to tablets, smartphones, and/or the like).

For example, in some implementations, systems, methods and apparatuses herein can detect facial expressions according to EMG signals that are obtained from one or more electrodes placed on a face of the user. In some implementations, the electrodes can be unipolar electrodes. The unipolar electrodes can be situated on a mask that contacts the face of the user, such that a number of locations on the upper face of the user are contacted by the unipolar electrodes.

In some implementations, the EMG signals can be preprocessed to remove noise. The noise removal can be common average reference (i.e. CAR) (i.e., in which interfering signals from one or more neighboring electrodes are removed). After preprocessing the EMG signals, apparatuses, methods and systems can be analyzed to determine roughness.

The EMG signals can also be normalized. Normalization can allow facial expressions to be categorized into one of a number of users. The categorization can subsequently be used to identify facial expressions of new users (e.g., by comparing EMG signals of new users to those categorized from previous users. In some implementations, deterministic or probabilistic classifiers can be used to classify EMG signals representing facial expressions.

In some implementations, a user state can be determined before classification of the signals is performed. For example, if the user is in a neutral state (i.e., a state in which the user has a neutral expression on his/her face), the structure of the EMG signals (even after normalization) is different from the signals from a non-neutral state (i.e., a state in which the user has a non-neutral expression on his or her face). Determining whether or not a user is in a neutral state can increase the accuracy of the user's EMG signal classification.

In some implementations, a number of classification methods may be performed as described herein, including but not limited to: a categorization classifier; discriminant analysis (including but not limited to LDA (linear discriminant analysis), QDA (quadratic discriminant analysis) and variations thereof such as sQDA (time series quadratic discriminant analysis)); Riemannian geometry; a linear classifier; a Naïve Bayes Classifier (including but not limited to Bayesian Network classifier); a k-nearest neighbor classifier; a RBF (radial basis function) classifier; and/or a neural network classifier, including but not limited to a Bagging classifier, a SVM (support vector machine) classifier, a NC (node classifier), a NCS (neural classifier system), SCRLDA (Shrunken Centroid Regularized Linear Discriminate and Analysis), a Random Forest, and/or a similar classifier, and/or a combination thereof. Optionally, after classification, the determination of the facial expression of the user is adapted according to one or more adaptation methods, using one or more adaptation methods (for example, by retraining the classifier on a specific expression of the user and/or applying a categorization (pattern matching) algorithm).

According to at least some embodiments, there is provided a facial expression determination system for determining a facial expression on a face of a user comprising: an apparatus comprising a plurality of EMG (electromyography) electrodes configured for contact with the face of the user; and a computational device configured with instructions operating thereon to cause the computational device to: preprocess a plurality of EMG signals received from said EMG electrodes to form preprocessed EMG signals; and classify a facial expression according to said preprocessed EMG using a classifier, wherein: said preprocessing comprises determining a roughness of said EMG signals according to a predefined window, and said classifier classifies the facial expression according to said roughness.

Optionally classifying comprises determining whether the facial expression corresponds to a neutral expression or a non-neutral expression based upon.

Optionally upon determining a non-neutral expression, classifying includes determining said non-neutral expression.

Optionally said predefined window is of 100 ms.

Optionally said classifier classifies said preprocessed EMG signals of the user using at least one of (1) a discriminant analysis classifier; (2) a Riemannian geometry classifier; (3) Naïve Bayes classifier, (4) a k-nearest neighbor classifier, (5) a RBF (radial basis function) classifier, (6) a Bagging classifier, (7) a SVM (support vector machine) classifier, (8) a node classifier (NC), (9) NCS (neural classifier system), (10) SCRLDA (Shrunken Centroid Regularized Linear Discriminate and Analysis), or (11) a Random Forest classifier.

Optionally said discriminant analysis classifier is one of (1) LDA (linear discriminant analysis), (2) QDA (quadratic discriminant analysis), or (3) sQDA.

Optionally said classifier is one of (1) Riemannian geometry, (2) QDA and (3) sQDA.

Optionally the system further comprises a classifier training system for training said classifier, said training system configured to receive a plurality of sets of preprocessed EMG signals from a plurality of training users, wherein: each set including a plurality of groups of preprocessed EMG signals from each training user, and each group of preprocessed EMG signals corresponding to a previously classified facial expression of said training user; said training system additionally configured to: determine a pattern of variance for each of said groups of preprocessed EMG signals across said plurality of training users corresponding to each classified facial expression, and compare said preprocessed EMG signals of the user to said patterns of variance to adjust said classification of the facial expression of the user. Optionally the instructions are additionally configured to cause the computational device to receive data associated with at least one predetermined facial expression of the user before classifying the facial expression as a neutral expression or a non-neutral expression. Optionally said at least one predetermined facial expression is a neutral expression. Optionally said at least one predetermined facial expression is a non-neutral expression. Optionally the instructions are additionally configured to cause the computational device to: retrain said classifier on said preprocessed EMG signals of the user to form a retrained classifier, and classify said expression according to said preprocessed EMG signals by said retrained classifier to determine the facial expression.

Optionally the system further comprises a training system for training said classifier and configured to receive a plurality of sets of preprocessed EMG signals from a plurality of training users, wherein: each set comprising a plurality of groups of preprocessed EMG signals from each training user, each group of preprocessed EMG signals corresponding to a previously classified facial expression of said training user; said training system additionally configured to: determine a pattern of variance of for each of said groups of preprocessed EMG signals across said plurality of training users corresponding to each classified facial expression; and compare said preprocessed EMG signals of the user to said patterns of variance to classify the facial expression of the user.

Optionally said electrodes comprise unipolar electrodes.

Optionally preprocessing said EMG signals comprises removing common mode interference of said unipolar electrodes.

Optionally said apparatus further comprises a local board in electrical communication with said EMG electrodes, the local board configured for converting said EMG signals from analog signals to digital signals, and a main board configured for receiving said digital signals.

Optionally said EMG electrodes comprise eight unipolar EMG electrodes and one reference electrode, the system further comprising: an electrode interface in electrical communication with said EMG electrodes and with said computational device, and configured for providing said EMG signals from said EMG electrodes to said computational device; and a mask configured to contact an upper portion of the face of the user and including an electrode plate; wherein said EMG electrodes being configured to attach to said electrode plate of said mask, such that said EMG electrodes contact said upper portion of the face of the user.

Optionally the system further comprises: a classifier training system for training said classifier, said training system configured to receive a plurality of sets of preprocessed EMG signals from a plurality of training users, wherein:

each set comprising a plurality of groups of preprocessed EMG signals from each training user, and each group of preprocessed EMG signals corresponding to a previously classified facial expression of said training user;

wherein said training system configured to:

compute a similarity score for said previously classified facial expressions of said training users,

fuse together each plurality of said previously classified facial expressions having said similarity score above a threshold indicating excessive similarity, so as to form a reduced number of said previously classified facial expressions; and

train said classifier on said reduced number of said previously classified facial expressions.

Optionally the instructions are further configured to cause the computational device to determine a level of said facial expression according to a standard deviation of said roughness.

Optionally said preprocessing comprises removing electrical power line interference (PLI).

Optionally said removing said PLI comprising filtering said EMG signals with two series of Butterworth notch filters of order 1, a first series of filter at 50 Hz and all its harmonics up to the Nyquist frequency, and a second series of filter with cutoff frequency at 60 Hz and all its harmonics up to the Nyquist frequency.

Optionally said determining said roughness further comprises calculating an EMG-dipole.

Optionally said determining said roughness further comprises a movement of said signals according to said EMG-dipole.

Optionally said classifier determines said facial expression at least partially according to a plurality of features, wherein said features comprise one or more of roughness, roughness of EMG-dipole, a direction of movement of said EMG signals of said EMG-dipole and a level of facial expression.

According to at least some embodiments, there is provided a facial expression determination system for determining a facial expression on a face of a user, comprising:

an apparatus comprising a plurality of EMG (electromyography) electrodes in contact with the face of the user; and

a computational device in communication with said electrodes and configured for receiving a plurality of EMG signals from said EMG electrodes, said computational device including:

a signal processing abstraction layer configured to preprocess said EMG signals to form preprocessed EMG signals; and

a classifier configured to receive said preprocessed EMG signals, the classifier configured to retrain said classifier on said preprocessed EMG signals of the user to form a retrained classifier; the classifier configured to classify said facial expression based on said preprocessed EMG signals and said retrained classifier.

According to at least some embodiments, there is provided a facial expression determination system for determining a facial expression on a face of a user, comprising:

an apparatus comprising a plurality of EMG (electromyography) electrodes in contact with the face of the user;

a computational device in communication with said electrodes and configured for receiving a plurality of EMG signals from said EMG electrodes, said computational device including:

a signal processing abstraction layer configured to preprocess said EMG signals to form preprocessed EMG signals; and

a classifier configured to receive said preprocessed EMG signals and for classifying the facial expression according to said preprocessed EMG signals; and

a training system configured to:

train said classifier, said training system configured to receive a plurality of sets of preprocessed EMG signals from a plurality of training users,

wherein:

each set comprising a plurality of groups of preprocessed EMG signals from each training user,

each group of preprocessed EMG signals corresponding to a previously classified facial expression of said training user;

determine a pattern of variance of for each of said groups of preprocessed EMG signals across said plurality of training users corresponding to each classified facial expression;

and

compare said preprocessed EMG signals of the user to said patterns of variance to classify the facial expression of the user.

According to at least some embodiments, there is provided a facial expression determination system for determining a facial expression on a face of a user, comprising:

an apparatus comprising a plurality of unipolar EMG (electromyography) electrodes in contact with the face of the user; and

a computational device in communication with said electrodes and configured with instructions operating thereon to cause the computational device to:receive a plurality of EMG signals from said EMG electrodes,

preprocess said EMG signals to form preprocessed EMG signals by removing common mode effects,

normalize said preprocessed EMG signals to form normalized EMG signals, and

classify said normalized EMG signals to determine the facial expression.

According to at least some embodiments, there is provided a system for determining a facial expression on a face of a user, comprising an apparatus comprising a plurality of EMG (electromyography) electrodes in contact with the face of the user; a computational device in communication with said electrodes and configured for receiving a plurality of EMG signals from said EMG electrodes, said computational device including: a signal processing abstraction layer configured to preprocess for preprocessing said EMG signals to form preprocessed EMG signals; and a classifier configured to receive said preprocessed EMG signals and for classifying the facial expression according to said preprocessed EMG signals; and a training system for training said classifier, said training system configured to: receive a plurality of sets of preprocessed EMG signals from a plurality of training users, wherein each set comprises a plurality of groups of preprocessed EMG signals from each training user, each group of preprocessed EMG signals corresponding to a previously classified facial expression of said training user; compute a similarity score for said previously classified facial expressions of said training users, fuse each plurality of said previously classified facial expressions having said similarity score above a threshold indicating excessive similarity, so as to reduce a number of said previously classified facial expressions; and train said classifier on said reduced number of said previously classified facial expressions.

According to at least some embodiments, there is provided a facial expression determination method for determining a facial expression on a face of a user, the method operated by a computational device, the method comprising: receiving a plurality of EMG (electromyography) electrode signals from EMG electrodes in contact with the face of the user; preprocessing said EMG signals to form preprocessed EMG signals, preprocessing comprising determining roughness of said EMG signals according to a predefined window; and determining if the facial expression is a neutral expression or a non-neutral expression; and classifying said non-neutral expression according to said roughness to determine the facial expression, when the facial expression is a non-neutral expression.

Optionally said preprocessing said EMG signals to form preprocessed EMG signals further comprises removing noise from said EMG signals before said determining said roughness, and further comprises normalizing said EMG signals after said determining said roughness.

Optionally said electrodes comprise unipolar electrodes and wherein said removing noise comprises removing common mode interference of said unipolar electrodes.

Optionally said predefined window is of 100 ms.

Optionally said normalizing said EMG signals further comprises calculating a log normal of said EMG signals and normalizing a variance for each electrode.

Optionally said normalizing said EMG signals further comprises calculating covariance across a plurality of users.

Optionally the method further comprises:

before classifying the facial expression, the method includes training said classifier on a plurality of sets of preprocessed EMG signals from a plurality of training users, wherein:

each set comprising a plurality of groups of preprocessed EMG signals from each training user,

each group of preprocessed EMG signals corresponding to a previously classified facial expression of said training user;

said training said classifier comprises determining a pattern of covariances for each of said groups of preprocessed EMG signals across said plurality of training users corresponding to each classified facial expression; and

said classifying comprises comparing said normalized EMG signals of the user to said patterns of covariance to adjust said classification of the facial expression of the user.

Optionally said classifier classifies said preprocessed EMG signals of the user according to a classifier selected from the group consisting of discriminant analysis; Riemannian geometry; Naïve Bayes, k-nearest neighbor classifier, RBF (radial basis function) classifier, Bagging classifier, SVM (support vector machine) classifier, NC (node classifier), NCS (neural classifier system), SCRLDA (Shrunken Centroid Regularized Linear Discriminate and Analysis), Random Forest, or a combination thereof.

Optionally said discriminant analysis classifier is selected from the group consisting of LDA (linear discriminant analysis), QDA (quadratic discriminant analysis) and sQDA.

Optionally said classifier is selected from the group consisting of Riemannian geometry, QDA and sQDA.

Optionally said classifying further comprises receiving at least one predetermined facial expression of the user before said determining if the facial expression is a neutral expression or a non-neutral expression.

Optionally said at least one predetermined facial expression is a neutral expression.

Optionally said at least one predetermined facial expression is a non-neutral expression.

Optionally said classifying further comprises retraining said classifier on said preprocessed EMG signals of the user to form a retrained classifier; and classifying said expression according to said preprocessed EMG signals by said retrained classifier to determine the facial expression.

Optionally the method further comprises:

training said classifier, before said classifying the facial expression, on a plurality of sets of preprocessed EMG signals from a plurality of training users, wherein:

each set comprising a plurality of groups of preprocessed EMG signals from each training user, and

each group of preprocessed EMG signals corresponding to a previously classified facial expression of said training user; and

determining a pattern of variance of for each of said groups of preprocessed EMG signals across said plurality of training users corresponding to each classified facial expression, wherein said classifying comprises comparing said preprocessed EMG signals of the user to said patterns of variance to classify the facial expression of the user.

Optionally the method further comprises:

training said classifier, before said classifying the facial expression, on a plurality of sets of preprocessed EMG signals from a plurality of training users,

wherein:

each set comprising a plurality of groups of preprocessed EMG signals from each training user,

each group of preprocessed EMG signals corresponding to a previously classified facial expression of said training user;

said training further comprises:

assessing a similarity score for said previously classified facial expressions of said training users, and

fusing together each plurality of said previously classified facial expressions having said similarity score above a threshold indicating excessive similarity, to form a reduced number of said previously classified facial expressions wherein said training said classifier comprises training on said reduced number of said previously classified facial expressions.

Optionally said training further comprises:

determining a pattern of variance for each of said groups of preprocessed EMG signals across said plurality of training users corresponding to each classified facial expression,

wherein said classifying comprises comparing said preprocessed EMG signals of the user to said patterns of variance to adjust said classification of the facial expression of the user.

According to at least some embodiments, there is provided a facial expression determination apparatus for determining a facial expression on a face of a user, comprising:

a plurality of unipolar or bipolar EMG (electromyography) electrodes in contact with the face of the user and

a computational device in communication with said electrodes, the device configured with instructions operating thereon to cause the device to:

receive a plurality of EMG signals from said EMG electrodes;

preprocess said EMG signals to form preprocessed EMG signals by removing common mode effects,

normalize said preprocessed EMG signals to form normalized EMG signals, and

classify said normalized EMG signals to detect the facial expression.

Optionally the apparatus further comprises:

an electrode interface; and

a mask which contacts an upper portion of the face of the user, said mask including an electrode plate attached to eight EMG electrodes and one reference electrode such that said EMG electrodes contact said upper portion of the face of the user, wherein said electrode interface being operatively coupled to said EMG electrodes and said computational device for providing said EMG signals from said EMG electrodes to said computational device.

According to at least some embodiments, there is provided a facial expression determination system for determining a facial expression on a face of a user comprising:

an apparatus comprising a plurality of EMG (electromyography) electrodes configured for contact with the face of the user; and

a computational device configured for receiving a plurality of EMG signals from said EMG electrodes, said computational device configured with instructions operating thereon to cause the computational device to:preprocess said EMG signals to form preprocessed EMG signals;determining a plurality of features according to said preprocessed EMG using a classifier, wherein said features include roughness and wherein said preprocessing preprocesses said EMG signals to determine a roughness of said EMG signals according to a predefined window; anddetermine the facial expression according to said features.

Optionally the instructions are further configured to cause the computational device to determine a level of said facial expression according to a standard deviation of said roughness, wherein said features further comprise said level of said facial expression.

Optionally said determining said roughness further comprises calculating an EMG-dipole, and determining said roughness for said EMG-dipole, wherein said features further comprise said roughness of said EMG-dipole.

Optionally said determining said roughness further comprises a movement of said signals according to said EMG-dipole, wherein said features further comprise said movement of said signals.

Optionally the system further comprises a weight prediction module configured for performing weight prediction of said features; and an avatar modeler for modeling said avatar according to a blend-shape, wherein said blend-shape is determined according to said weight prediction.

Optionally said electrodes comprise bi-polar electrodes.

Optionally the system, method or apparatus of any of the above claims further comprises detecting voice sounds made by the user; and animating the mouth of an avatar of the user in response thereto.

Optionally upon voice sounds being detected from the user, further comprising animating only an upper portion of the face of the user.

Optionally the system, method or apparatus of any of the above claims further comprises upon no facial expression being detected, animating a blink or an eye movement of the user.

Optionally said system and/or said apparatus comprises a computational device and a memory, wherein:

said computational device is configured to perform a predefined set of basic operations in response to receiving a corresponding basic instruction selected from a predefined native instruction set of codes, set instruction comprising:

a first set of machine codes selected from the native instruction set for receiving said EMG data,

a second set of machine codes selected from the native instruction set for preprocessing said EMG data to determine at least one feature of said EMG data and

a third set of machine codes selected from the native instruction set for determining a facial expression according to said at least one feature of said EMG data; wherein each of the first, second and third sets of machine code is stored in the memory.

As used herein, the term “EMG” refers to “electromyography,” which measures the electrical impulses of muscles.

As used herein, the term “muscle capabilities” refers to the capability of a user to operate a plurality of muscles in coordination for some type of activity. A non-limiting example of such an activity is a facial expression. It should be noted that a deficit in a muscle capability is not necessarily due to damage to the muscles involved, but may be due to damage in any part of the physiological system required for muscles to be operated in coordination, including but not limited to, central or peripheral nervous system damage, or a combination thereof.

US Patent Application No. 20070179396 describes a method for detecting facial muscle movements. The facial muscle movements are described as being detectable by using one or more of electroencephalograph (EEG) signals, electrooculograph (EOG) signals and electromyography (EMG) signals.

U.S. Pat. No. 7,554,549 describes a system and method for analyzing EMG (electromyography) signals from muscles on the face to determine a user's facial expression, but by using bipolar electrodes. Such expression determination is then used for computer animation.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which user matter of this disclosure belongs. The materials, methods, and examples provided herein are illustrative only and not intended to be limiting.

Implementation of the apparatuses, methods and systems of the present disclosure involves performing or completing certain selected tasks or steps manually, automatically, or a combination thereof. Specifically, several selected steps can be implemented by hardware or by software on an operating system, of a firmware, and/or a combination thereof. For example, as hardware, selected steps of the invention can be implemented as a chip or a circuit. As software, selected steps of the invention can be implemented as a number of software instructions being executed by a computer (e.g., a processor of the computer) using an operating system. In any case, selected steps of the method and system of the invention could be described as being performed by a data processor, such as a computing platform for executing a plurality of instructions.

Although the present invention is described with regard to a “computer” on a “computer network,” it should be noted that any device featuring a data processor and the ability to execute one or more instructions may be described as a computer or as a computational device, including but not limited to a personal computer (PC), a processor, a server, a cellular telephone, an IP telephone, a smart phone, a PDA (personal digital assistant), a thin client, a mobile communication device, a smart watch, head mounted display or other wearable that is able to communicate externally, a virtual or cloud based processor, a pager, and/or a similar device. Two or more of such devices in communication with each other may be a “computer network.”

