We propose a method, called Label Embedding Network, which can learn label representation (label embedding) during the training process of deep networks. With the proposed method, the label embedding is adaptively and automatically learned through back propagation. The original one-hot represented loss function is converted into a new loss function with soft distributions, such that the originally unrelated labels have continuous interactions with each other during the training process. As a result, the trained model can achieve substantially higher accuracy and with faster convergence speed. Experimental results based on competitive tasks demonstrate the effectiveness of the proposed method, and the learned label embedding is reasonable and interpretable. The proposed method achieves comparable or even better results than the state-of-the-art systems. Most of the existing methods of neural networks use one-hot vector representations for labels. The one-hot vector has two main restrictions. The first restriction is the "discrete distribution", where each label is distributed at a completely different dimension from the others. The second restriction is the "extreme value" based representation, where the value at each dimension is either 1 or 0, and there is no "soft value" allowed. Those deficiencies may cause the following two potential problems.First, it is not easy to measure the correlation among the labels due to the "discrete distribution". Not being able to measure the label correlation is potentially harmful to the learned models, e.g., causing the data sparseness problem. Given an image recognition task, the image of the shark is often similar to the image of the dolphin. Naturally, we expect the two labels to be "similar". Suppose that we have a lot of training examples for shark, and very few training examples for dolphin. If the label shark and the label dolphin have similar representations, the prediction for the label dolphin will suffer less from the data sparsity problem.Second, the 0/1 value encoding is easy to cause the overfitting problem. Suppose A and B are labels of two similar types of fishes. One-hot label representation prefers the ultimate separation of those two labels. For example, if currently the system output probability for A is 0.8 and the probability for B is 0.2, it is good enough to make a correct prediction of A. However, with the one-hot label representation, it suggests that further modification to the parameters is still required, until the probability of A becomes 1 and the probability of B becomes 0. Because the fish A and the fish B are very similar in appearance, it is probably more reasonable to have the probability 0.8 for A and 0.2 for B, rather than completely 1 for A and 0 for B, which could lead to the overfitting problem.We aim to address those problems. We propose a method that can automatically learn label representation for deep neural networks. As the training proceeds, the label embedding is iteratively learned and optimized based on the proposed label embedding network through back propagation. The original one-hot represented loss function is softly converted to a new loss function with soft distributions, such that those originally unrelated labels have continuous interactions with each other during the training process. As a result, the trained model can achieve substantially higher accuracy, faster convergence speed, and more stable performance. The related prior studies include the traditional label representation methods BID7 BID10 BID1 , the "soft label" methods BID22 , and the model distillation methods BID9 ).Our method is substantially different from those existing work, and the detailed comparisons are summarized in Appendix E. The contributions of this work are as follows:• Learning label embedding and compressed embedding: We propose the Label Embedding Network that can learn label representation for soft training of deep networks. Furthermore , some large-scale tasks have a massive number of labels, and a naive version of label embedding network will suffer from intractable memory cost problem. We propose a solution to automatically learn compressed label embedding, such that the memory cost is substantially reduced.• Interpretable and reusable: The learned label embeddings are reasonable and interpretable, such that we can find meaningful similarities among the labels. The proposed method can learn interpretable label embeddings on both image processing tasks and natural language processing tasks. In addition, the learned label embeddings can be directly adapted for training a new model with improved accuracy and convergence speed.• General-purpose solution and competitive results: The proposed method can be widely applied to various models, including CNN, ResNet, and Seq-to-Seq models. We conducted experiments on computer vision tasks including CIFAR-100, CIFAR-10, and MNIST, and on natural language processing tasks including LCSTS text summarization task and IWSLT2015 machine translation task. Results suggest that the proposed method achieves significantly better accuracy than the existing methods (CNN, ResNet, and Seq-to-Seq). We achieve results comparable or even better than the state-of-the-art systems on those tasks. We propose a method that can learn label representation during the training process of deep neural networks. Furthermore, we propose a solution to automatically learn compressed label embedding, such that the memory cost is substantially reduced. The proposed method can be widely applied to different models. We conducted experiments on CV tasks including CIFAR-100, CIFAR-10, and MNIST, and also on natural language processing tasks including LCSTS and IWSLT2015. Results suggest that the proposed method achieves significant better accuracies than the existing methods (CNN, ResNet, and Seq-to-Seq). Moreover, the learned label embeddings are reasonable and interpretable, which provides meaningful semantics of the labels. We achieve comparable or even better results with the state-of-the-art systems on those tasks.
Learning Label Representation for Deep Networks
{"src": "scitldr-AIC_train", "paper_id": "BkVf1AeAZ"}