FIELD OF THE INVENTION

The invention relates to a ML model arrangement configured for evaluating motion patterns in a sequence of image data structures and to a mobile device comprising a ML model arrangement. The invention further relates to a method for evaluating a motion pattern in a sequence of image data structures. The invention also relates to a method for configuring a second ML model for evaluating a specific motion pattern.

Background of the Invention

U.S. Pat. No. 9,154,739 B1 describes a computer-implemented method, a system and a computer-readable medium that provide useful feedback for a user involved in exercise. A camera is used to track user motion by using image processing techniques to identify key points of a user's body and track their motion. The tracked points are compared to proper form for an exercise, and an embodiment gives feedback based on the relationship between the actual movement of the user and the proper form.

OBJECT OF THE INVENTION

The object of the invention is to provide a method and a system capable of analysing and evaluating motion patterns in a sequence of image data structures.

SUMMARY OF THE INVENTION

According to the invention, a ML model arrangement configured for evaluating motion patterns in a sequence of image data structures is provided. The ML model arrangement comprises a first ML model configured for predicting a set of key data elements for each image data structure of the sequence of image data structures, a key data element indicating a respective position of a landmark in the image data structure. The ML model arrangement further comprises at least one second ML model, each second ML model being a ML model configured for evaluating a corresponding specific motion pattern. Each second ML model is configured for determining, based on input data comprising at least one of the key data elements predicted for at least one image data structure or data derived therefrom, class labels for each image data structure, said class labels identifying at least one of: at least one motion phase of the specific motion pattern, at least one evaluation point of the specific motion pattern.

According to embodiments of the present invention, an arrangement of a first ML model and at least one second ML model is used for evaluating motion patterns in a sequence of image data structures. In the present application, the term “ML” stands for “machine learning”. The first ML model is configured for recognising landmarks in the image data structure and for predicting a set of key data elements, wherein a key data element indicates a respective position of a landmark in the image data structure. The second ML model is configured for classifying, based on key data elements provided by the first ML model, each image data set. For each image data set, a class label is determined, wherein the class label indicates at least one of: at least one motion phase, at least one evaluation point. For example, the motion phases of the motion pattern can be identified. Even complex motion patterns can be segmented into a number of motion phases. Furthermore, the second ML model may for example classify an image data structure as being an evaluation point. In case an image data structure is classified as an evaluation point, a further evaluation of the image data structure may for example be performed.

The arrangement of a first ML model and at least one second ML model has proven to be suitable for analysing and evaluating motion patterns in a sequence of image data structures. Both the first and the at least one second ML model are chosen to fulfill their respective tasks in an optimised manner. The first ML model is configured for predicting key data elements, and the second ML model is trained for performing the classification task based on the key data elements provided by the first ML model. For each of these tasks, a different type of ML model can be used. For example, the first ML model may be a convolutional neural network, whereas the second ML model may for example be a decision tree or a random forest. These are only examples, other types of machine learning models may be used as well.

Further according to the invention, a method for evaluating a motion pattern in a sequence of image data structures is provided. The method comprises predicting, using a first ML model, a set of key data elements for each image data structure of the sequence of image data structures, wherein a key data element indicates a respective position of a landmark in the image data structure. The method further comprises determining, using a second ML model, class labels for each image data structure based on input data comprising at least one of the key data elements predicted for at least one image data structure or data derived therefrom, said class labels identifying at least one of: at least one motion phase of the specific motion pattern, at least one evaluation point of the specific motion pattern.

Further according to the invention, a mobile device is provided, the mobile device comprising a ML model arrangement as described above. Preferably, the mobile device further comprises a camera configured for acquiring the sequence of image data structures.

Yet further according to the invention, a method for configuring a second ML model for evaluating a specific motion pattern is provided. The method comprises training the second ML model for classifying at least one of: at least one motion phase of the specific motion pattern, at least one evaluation point of the specific motion pattern. Said training is based on a plurality of sequences of image data structures showing different variants of the specific motion pattern, wherein for each image data structure, a set of key data elements is provided, a key data element indicating a respective position of a landmark in the image data structure. Said training is further based on class labels provided for each image data structure.

The method according to embodiments of the present invention provides a second learning model that is capable of classifying at least one of: at least one motion phase of a specific motion pattern, at least one evaluation point of a specific motion pattern. For this purpose, a training step of the second ML model is performed. The training is based on a plurality of sequences of image data structures showing different variants of the specific motion pattern. Together with each image data structure, a class label is specified. Training of the second ML model is performed based on sequences of image data structures and the corresponding class labels. After the training step has been performed, the second ML model is capable of classifying image data structures in a sequence of image data structures and for providing appropriate class labels indicating at least one of: at least one motion phase of the specific motion pattern, at least one evaluation point of the specific motion pattern.

PREFERRED EMBODIMENTS OF THE INVENTION

Preferred features of the invention which may be applied alone or in combination are discussed below and in the dependent claims.

ML Model Arrangement

Preferably, each second ML model is configured for determining, for each image data structure of the sequence of image data structures, a corresponding class label. By providing a classification for each image data structure, a complete description of the motion pattern is obtained, which may serve as a basis for further evaluation.

Preferably, each second ML model is of a different type than the first ML model. The first ML model's task is to predict key data elements for each image data structure. The task of the second ML model is to perform a classification of the image data structures and to provide a class label for an image data structure. Preferably, different ML models are used for these different tasks, each ML model being chosen and trained such that the respective task can be fulfilled in an optimised manner.

In a preferred embodiment, each second ML model is configured for determining class labels based on input data comprising key data elements predicted for the image data structure under consideration and for at least one of: at least one preceding image data structure, at least one future image data structure. By considering key data elements predicted for a number of preceding image data structures, the history of the key data element's motion is taken into account when determining a classification. In particular, when determining a class label indicating a motion phase or an evaluation point, the previous motion of the key data elements is taken into account. Additionally or alternatively, future image data structures may be taken into account. As a consequence, a more accurate classification is obtained.

Preferably, the input data for each second ML model comprises a set of parameters including the key data elements predicted for the image data structure under consideration.

Preferably, the input data for each second ML model comprises additional attributes for classification, said additional attributes being derived from key data elements predicted for the image data structure under consideration and for at least one of: at least one preceding image data structure, at least one future image data structure. These additional attributes may for example comprise parameters such as a minimum coordinate value of a particular key data element's position within a certain set of image data structures, or any other additional attribute derived from the key data elements' positions. In particular, the additional attributes may be chosen such that the classification task performed by the second ML model is promoted and simplified.

Preferably, the first ML model is a neural network. A neural network is well-suited for identifying landmarks in an image data structure and for predicting the positions of respective key data elements. Further preferably, the first ML model is a convolutional neural network.

Preferably, at least one of the second ML models is a classification model. A classification model is an ML model that is configured for determining, for each image data structure, a corresponding class label, the class label indicating for example a certain motion phase and/or an evaluation point.

In a preferred embodiment, at least one of the second ML models is a decision tree or a random forest comprising at least one decision tree. A decision tree may be descended starting at a root node, wherein each node of the tree acts as a test case for some attribute, and wherein the leaf node indicates a class label. In this regard, a decision tree is well-suited for performing a classification task. A random forest comprises a plurality of separate decision trees, wherein each decision tree of the random forest determines a class label and wherein the classification with the most votes decides the final classification.

Preferably, the decision tree or the random forest is configured for providing a class label for each image data structure of the sequence of image data structures.

According to a preferred embodiment, at least one second ML model is configured for determining class labels for at least one motion phase and for at least one evaluation point, wherein there is a predefined correlation between the at least one motion phase and the at least one evaluation point, with an evaluation point being a specific point of time within a motion phase or between consecutive motion phases.

In a preferred embodiment, the ML model arrangement further comprises at least one geometric evaluation unit, wherein the geometric evaluation unit is configured for performing a geometric evaluation of at least one of the key data elements predicted for an image data structure or for performing a geometric evaluation of at least one motion phase of a specific motion pattern. The geometric evaluation unit may for example be configured for geometrically evaluating a configuration of key data elements predicted for an image data structure or of a subset of these key data elements. In particular, the geometric evaluation unit may for example be configured for evaluating at least one of distances between key data elements, angles between key data elements, proportions between key data elements or sets of key data elements, relations between key data elements or sets of key data elements, etc. Additionally or alternatively, the geometric evaluation unit may for example be configured for evaluating at least one motion phase determined by at least one of the second ML models.

Preferably, the ML model arrangement is configured such that whenever a second ML model identifies that a certain image data structure is an evaluation point, a geometric evaluation of the configuration of key data elements predicted for the respective image data structure is initiated. For each evaluation point identified by a second ML model, a further geometric evaluation of the configuration of key data elements of the image data structure that corresponds to the evaluation point is performed.

Further preferably, evaluating a motion pattern comprises evaluating at least one motion phase determined by at least one of the second ML models. Preferably, the ML model arrangement is configured for evaluating at least one motion phase determined by at least one of the second ML models.

In a preferred embodiment, the ML model arrangement is configured for evaluating a sequence of image data structures showing a person doing physical exercising. The ML model arrangement can for example be used for evaluating a single physical exercise or several repetitions of a physical exercise. For each physical exercise, a corresponding second ML model that has been trained on this physical exercise may for example be used for identifying at least one motion phase and/or at least one evaluation point of the respective physical exercise.

Preferably, the first ML model is configured for predicting a representation of the person's body for each image data structure of the sequence of image data structures. For example, the person's body may be represented by a 3D mesh.

Preferably, the first ML model is configured for predicting a set of body key points for each image data structure of the sequence of image data structures. Body key points may for example be assigned to joints of the body and to body features like for example the forehead, the chin, the breastbone, the hip, etc. Preferably, the body key points identified for an image data structure form a skeleton of the human body and allow for detecting the human body's pose in the respective image data structure.

Preferably, a specific motion pattern corresponds to a particular physical exercise. Preferably, for each physical exercise, a dedicated second ML model is provided. Further preferably, the ML model arrangement comprises a first ML model configured for predicting body key points for each image data structure and a plurality of second ML models corresponding to a variety of different physical exercises to be analysed.

Preferably, at least one of the second ML models is an exercise specific second ML model configured for evaluating a specific physical exercise, the exercise specific second ML model being configured for determining class labels for at least one of: at least one motion phase of the specific physical exercise, at least one evaluation point of the specific physical exercise.

In a preferred embodiment, the ML model arrangement is configured such that whenever a second ML model corresponding to the physical exercise identifies that a certain image data structure is an evaluation point, a geometric evaluation of a particular pose at the respective evaluation point is initiated. In this regard, each evaluation point identified by the second ML model may for example relate to a particular pose that is to be evaluated in more detail.

Preferably, evaluating a particular pose comprises comparing one or more of the body key points related to the particular pose and/or one or more parameters derived therefrom with one or more predefined constraints related to the particular pose at the respective evaluation point. The one or more constraints may for example define how the pose at the particular evaluation point should look. In this regard, the one or more constraints may for example comprise constraints related to distances between body key points, constraints related to angles between body key points, constraints related to proportions between body key points or sets of body key points, etc.

Preferably, the ML model arrangement comprises a feedback unit configured for providing a feedback to the user in dependence on the result of the evaluation of the particular pose. By providing feedback, the user may for example be informed whether or not the exercise is performed correctly. Furthermore, the user may be informed on how to improve the way the exercise is performed. Preferably, the feedback is based on determining a deviation between the user's pose and the ideal pose at a certain evaluation point.

Method for Evaluating a Motion Pattern

Preferably, for each image data structure of the sequence of image data structures, a corresponding class label is determined.

Preferably, class labels are determined based on input data comprising key data elements predicted for the image data structure under consideration and for at least one of: at least one preceding image data structure, at least one future image data structure.

According to a preferred embodiment, using the second ML model, class labels are determined for at least one motion phase and for at least one evaluation point, wherein there is a predefined correlation between the at least one motion phase and the at least one evaluation point, with an evaluation point being a specific point of time within a motion phase or between consecutive motion phases.

In a preferred embodiment, the method further comprises initiating, whenever a certain image data structure is identified as an evaluation point, a geometric evaluation of the configuration of key data elements predicted for the respective image data structure.

Further preferably, the method comprises evaluating at least one motion phase determined by at least one of the second ML models.

According to a preferred embodiment, the method comprises evaluating a sequence of image data structures showing a person doing physical exercising.

Preferably, for each image data structure of the sequence of image data structures, a representation of the person's body is predicted.

Preferably, for each image data structure of the sequence of image data structures, a set of body key points is predicted.

Preferably, a specific motion pattern corresponds to a particular physical exercise.

Preferably, the method further comprises initiating, whenever a second ML model corresponding to the physical exercise identifies that a certain image data structure is an evaluation point, a geometric evaluation of a specific pose at the respective evaluation point.

In a preferred embodiment, evaluating a particular pose comprises comparing one or more of the body key points related to the particular pose and/or one or more parameters derived therefrom with one or more predefined constraints related to the particular pose at the respective evaluation point.

Preferably, the method comprises providing a feedback to the user in dependence on the result of the evaluation of the particular pose.

Preferably, the first ML model is a neural network.

In a preferred embodiment, each second ML model is a ML model configured for evaluating a specific motion pattern.

In a preferred embodiment, at least one of the second ML models is an exercise specific second ML model configured for evaluating a specific physical exercise, the exercise specific second ML model being configured for determining class labels for at least one of: at least one motion phase of the specific physical exercise, at least one evaluation point of the specific physical exercise and at least one phase without physical exercising.

Preferably, at least one of the second ML models is a decision tree or a random forest comprising at least one decision tree.

Method for Training a Second ML Model

Preferably, the second ML model is a decision tree or a random forest comprising at least one decision tree.

Preferably, after the training step, the second ML model is configured for evaluating a specific motion pattern. Training of a second ML model may for example be based on a large number of sequences of image data structures showing a motion pattern, together with class labels provided for each image data structure. After the training step has been performed, the trained second ML model can be used for evaluating a specific motion pattern.

In a preferred embodiment, after the training step, the second ML model is configured for determining class labels for the image data structures of a sequence of image data structures showing the specific motion pattern, said class labels identifying at least one of: at least one motion phase of the specific motion pattern, at least one evaluation point of the specific motion pattern.

