For a machine learning method X (Deep Neural Nets variant), which performs classification tasks.In the output layer, for every label method, X also emits the uncertainty U associated with the respective prediction.So, in case the testing dataset is very different then the predictions made by X should have a high U. (Mean of U across all labels should be high)In the case of testing, the dataset is indeed from the same source from which training dataset was generated. For true positives, the mean of uncertainty for the correct label would be low. In the case of false positives, the mean of U would be high (ideally). What is the principled approach to measuring the effectiveness of method X in outputting the uncertainty? In other words, what would be a good metric to understand the performance of the method in computing the uncertainty for each label? Metric similar to Root mean squared error or others.
Scoring metric for machine learning method
machine learning;neural networks;classification