I'm training a neural network that, for each of six classes, tries to predict the probability that a sample belongs to it. After that, I want to use these probabilities as fractions of the sample belonging to that class. My network gives softmax output and is trained using cross-entropy costs (in fact linear output which is then transformed to softmax by tf.nn.softmax_cross_entropy_with_logits)Is this the right cost function to use when I want to train the network in getting all 6 probabilities correct instead of just classifying each sample as one of the 6 classes? I started hesitating because the tensorflow documentation says about tf.nn.softmax_cross_entropy_with_logits: Measures the probability error in discrete classification tasks in which the classes are mutually exclusive (each entry is in exactly one class).UPDATEEven though it sound strange, the cross-entropy cost function seems to work best here. In my understanding this is because the cross-entropy for discrete probability distributions (wikipedia) is$H(p, q) = -\sum_x p(x)\, \log q(x).$This function is minimal when $p(x) = q(x)$ for all $x$. This explains why minimizing the cross-entropy error forces the output distribution $q$ to the target distribution $p$, even though I'm abusing tf.nn.softmax_cross_entropy_with_logits by instead of feeding it labels I feed it a discrete probability distribution.Why it works better than MSE probably is because the magnitudes of the 6 class probabilities are very different. Class 1 may have a probability of 80% while class 2 has a probability of 0.5%, so MSE pays more attention to getting the 80% class right. Does this mean that CE is the best option still, or is there a way to scale the outputs or weigh them such that the network pays attention to getting each class correct?
Is cross-entropy a good cost function if I'm interested in the probabilities of a sample belonging to a certain class?
machine learning;neural network;classification;tensorflow;cost function