BACKGROUND

Technological Field

The disclosed embodiments generally relate to machine learning. More particularly, the disclosed embodiments relate to systems and methods for personalized quality assurance of machine learning models.

Background Information

Computerized devices are now prevalent, and data produced and maintained by those devices is increasing.

Audio sensors are now part of numerous devices, and the availability of audio data produced by those devices is increasing.

Image sensors are now part of numerous devices, from security systems to mobile phones, and the availability of images and videos produced by those devices is increasing.

Machine learning algorithms, that use data to generate insights, rules and algorithms, are widely used.

SUMMARY

In some embodiments, systems and methods for the creation, maintenance, and usage of datasets and annotations are provided.

In some embodiments, examples may be used selectively. For example, information associated with external devices may be obtained, weights may be assigned to examples using the obtained information, and inference models may be generated using the assignment of weights. In some examples, update information associated with the external devices may be obtained, the assignment of weights may be updated using the update information, and updated inference models may be generated using the updated assignment of weights.

In some embodiments, actions may be triggered automatically in a dataset management system. For example, a change to data maintained by the dataset management system may be detected, and a rule associated with the data and with an action may be obtained. Further, it may be determined if the change is a trigger according to the rule. In some examples, when the change is determined to be a trigger according to the rule, the action associated with the rule may be performed, for example with the data associated with the change. In some examples, when the change is determined not to be a trigger according to the rule, the system may forgo the performance of the action, for example for the data associated with the change.

In some embodiments, inference models may be employed based on available processing resources. For example, available processing resources information may be received, inference model may be selected (for example, based on the received information), and the selected inference model may be utilized. In some cases, an update to the available processing resources information may be received, the selected inference model may be updated (for example, based on the received update), and the updated inference model may be utilized.

In some embodiments, personalized quality assurance of inference models may be performed. For example, data items associated with a group of devices may be obtained, results of applying the data items to inference models may be obtained, the results of applying the data items to a first inference model may be compared with the results of applying the data items to a second inference model, and the compatibility of the second inference model to the group of devices may be assessed, for example based on the comparison results. In some examples, when the second inference model is found compatible, the second inference model may be utilized in tasks associated with the group of devices. In some examples, when the second inference model is found incompatible, the system may forgo the usage of the second inference model in one or more tasks associated with the group of devices.

In some embodiments, descriptors of artificial neural networks may be generated and/or used. For example, an artificial neural network may be obtained, the artificial neural network may be obtained, descriptors of the segments may be calculated, and a descriptor of the artificial neural network may be compiled. In some examples, a match score for a pair of artificial neural networks may be calculated (for example using the descriptors compiled for the two artificial neural networks), and actions may be selected based on the matching score.

In some embodiments, required processing resources for machine learning tasks may be estimated. For example, properties of a machine learning training task may be obtained, properties of external devices may be obtained, and the processing resources requirements of the machine learning training task may be estimated (for example using the properties of the external devices). In some examples, the estimation may be provided to a user. In some examples, a group of devices may be selected, and the selected group of devices may be triggered to perform the machine learning training task.

In some embodiments, datasets may be enriched while learning. For example, intermediate results of training machine learning algorithms may be obtained. Additional training examples may be selected based on the intermediate results. In some cases, synthetic examples may be generated based on the intermediate results. The machine learning algorithms may be further trained using the selected additional training examples and/or the generated synthetic examples.

In some embodiments, datasets may be incrementally annotated. For example, a group of labeled examples and a group of unlabeled examples may be obtained, a first inference model may be generated using the group of labeled examples, labels may be assigned to at least part of the group of unlabeled examples using the first inference model, confidence levels may be assigned to the assigned labels, a subset of the group of unlabeled examples may be selected using the confidence levels, and in some cases a second inference model may be generated using the selected subset and/or the corresponding assigned labels.

In some embodiments, annotations of datasets may be merged. For example, assignments of labels to data-points may be obtained, confidence levels associated with the assignments of labels may be obtained. Further, the assignments of labels may be merged, for example based on the confidence levels. In some cases, inference models may be generated using the merged assignment of labels. In some examples, an update to the assignments of labels to data-points and/or the confidence levels may be obtained, and the merged assignment of labels may be updated.

In some embodiments, access to a dataset management system may be controlled using permission records. For example, a request to access information in a dataset management system may be obtained from an entity, and a permission record associated with the entity may be selected. Further, it may be determined if the entity has permission to access the information. In some examples, when the entity has permission to access the information, the access to the information may be allowed. In some examples, when the entity has no permission to access the information, the access to the information may be denied.

In some embodiments, access to a dataset management system may be controlled using quotas. For example, a request to perform an action in a dataset management system may be obtained from an entity, and a quota record associated with the entity may be selected. Further, it may be determined if the entity has sufficient quota to perform the action. In some examples, when the entity has sufficient quota to perform the action, the request may be allowed. In some examples, when the entity has insufficient quota to perform the action, the request may be denied.

In some embodiments, a project schedule may be maintained in a dataset management system. For example, a progress update and a project schedule record may be obtained. The project schedule record may be updated, for example according to the progress update. In some examples, a delay may be predicted, and a corresponding indication may be provided.

