The ability of detecting out-of-distribution (OOD) samples is important to secure reliability of deep neural networks in real-world applications. Considering the nature of OOD samples, detection methods should not have hyperparameters whose optimal values vary sensitively depending on incoming OOD samples. This requirement is not met by many previous methods. In this paper, we propose a simple, hyperparameter-free method that is based on softmax of scaled cosine similarity. It resembles the approach employed by recent metric learning methods, but it differs in details; the differences are essential to achieve high detection performance. As compared with the current state-of-the-art methods, which needs hyperparameter tuning that could compromise real-world performance, the proposed method attains at least competitive detection accuracy even without (tuning of) a hyperparameter; furthermore, it is computationally more efficient, since it needs only a single forward pass unlike previous methods that need backpropagation for each input.
These metric learning methods are identical in that they use cosine similarity to improve a few issues with the ordinary softmax. They differ in i) if and how the weight @math or the feature @math of the last layer are normalized and ii) if and how margins are used with the cosine similarity that (further) encourage maximization of inter-class variance and minimization of intra-class variance. According to this categorization, our method is the most similar to NormFace @cite_36 , in which both @math and @math are normalized and no margin is utilized. However, our method still differs from NormFace that it uses only a single fully-connected layer to compute the cosine similarity and it predicts scale @math multiplied with the inputs to softmax, as will be described below.
@cite_36
2609575245
Thanks to the recent developments of Convolutional Neural Networks, the performance of face verification methods has increased rapidly. In a typical face verification method, feature normalization is a critical step for boosting performance. This motivates us to introduce and study the effect of normalization during training. But we find this is non-trivial, despite normalization being differentiable. We identify and study four issues related to normalization through mathematical analysis, which yields understanding and helps with parameter settings. Based on this analysis we propose two strategies for training using normalized features. The first is a modification of softmax loss, which optimizes cosine similarity instead of inner-product. The second is a reformulation of metric learning by introducing an agent vector for each class. We show that both strategies, and small variants, consistently improve performance by between 0.2 to 0.4 on the LFW dataset based on two models. This is significant because the performance of the two models on LFW dataset is close to saturation at over 98 .